Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

BigQuery for Rust

Library provides a simple API for Google BigQuery using gRPC for every call:

  • Fluent high-level and strongly typed API;
  • Read and write rows as Rust structures using Serde, with own codecs for every BigQuery type, including NUMERIC/BIGNUMERIC, INTERVAL, RANGE, nested STRUCT and ARRAY, and JSON columns mapped by the field type (String, serde_json::Value or your own structure);
  • Support for:
    • Table reads through the Storage Read API, in parallel streams, with column projection and typed filters;
    • Queries with named and positional parameters, including ARRAY and STRUCT parameters, always bound on the server side;
    • Short query mode, so BigQuery can answer short queries without creating a job;
    • Large query results read through the Storage Read API automatically;
    • Query results written into a destination table of your own and read back the same way;
    • Raw Arrow record batches for table reads and query results;
    • DML statements with affected row counts, and dry runs;
    • Writes through the Storage Write API with built-in batching and backpressure, of your structures or of raw Arrow record batches: at least once, exactly once, atomic (all rows or none), or buffered (rows readable once you flush them);
    • Updates and deletes by the primary key through change data capture (CDC), with fluent update() and delete() or a lower-level CDC writer;
    • Declarative table schemas, inferred from your structures or declared by hand, planned and synced with one call: new columns, renames, drops, widening, partitioning, clustering, primary key, recreating an empty table;
    • Datasets, tables and jobs management;
    • Bytes processed and billed, slot milliseconds and cache hits for every query, as span fields and as results;
  • Retries with jitter for the errors where retrying makes sense, including BigQuery rate limits;
  • Full async based on Tokio runtime;
  • Macros that help you use your structure fields as column paths;
  • Google client based on gcloud-sdk library that automatically detects GCE environment or application default accounts for local development;

To start using the library, see Getting started.

Why this crate

Google has its own Rust crate for BigQuery, google-cloud-bigquery (0.18.0 at the time of writing, October 2026). Both crates work with the same APIs in different ways, so below is what each of them provides that the other does not.

What this crate provides

  • High-level typed API. Fluent builders for table reads, queries, writes, updates and deletes, and table schemas. Rows are your own structures in both directions, through Serde, for table reads, query results, writes and CDC. Dataset and table IDs are checked types, column paths come from your structure fields with path!/paths!, and table schemas are declared once and planned or synced with .plan()/.sync().
  • gRPC throughout. Queries, jobs, datasets and tables go through the BigQuery v2 API over gRPC, as well as Storage Read and Write. The official crate sends queries over REST and reads their rows as JSON pages.
  • Performance. Measured on 2026-10-05 against the official crate and Python on one home connection in Sweden to europe-north2, full details in the benchmarks:
    • small queries take 0.117 s for SELECT 1, about the same as the official crate, which also uses BigQuery’s short query mode, and faster than Python’s 0.270 s, since Python creates a job;
    • a 200,000-row query result as typed rows takes 1.16 s against 4.39 s, because the library reads a large result through Storage Read and the official crate pages it as JSON;
    • a 1M-row table scan as typed rows takes 9.6 s. The official crate has no typed Storage Read, so its typed path is a SELECT * query, which took 95.2 s and billed the whole table every run;
    • a 1M-row Storage Write from structures takes 34.0 s against 35.4 s, with 18% fewer bytes sent as protobuf than the official crate’s Arrow.
  • Observability. Every query, read and write span carries what BigQuery reports: bytes processed and billed, slot milliseconds, cache hits, rows and bytes read, rows appended and bytes sent, retries. query_with_stats() returns a query’s figures together with its rows.
  • Safety. Query parameter values are always bound on the server side and never written into the SQL text. The typed filter and the generated DDL write values as escaped literals and names as quoted identifiers, tested against a corpus of hostile values and on BigQuery itself. The calls that lose data say so in their names: dangerously_delete_with_contents(), dangerously_recreate_with_data_loss(). No unsafe code.
  • Testing. A fake BigQuery for your tests, in bigquery::testing. The code under test runs its real client against a loopback gRPC server, which answers queries, serves table rows and fails calls as the test scripts it with your own structures. No credentials or network.

What the official crate provides and this one does not

Checked against google-cloud-bigquery 0.18.0 and google-cloud-bigquery-v2 1.0.0:

  • It is maintained by Google as part of google-cloud-rust, and its v2 API crate is already 1.0;
  • It uses the REST API for v2, which is GA. The v2 API over gRPC this crate uses works for every call the library makes, but Google does not document it and it is pre-GA, so be aware it can change without notice;
  • It has stub traits to mock its clients in your tests. This crate has no mocks of its clients and provides a fake BigQuery server instead, see Testing with the fake.

Getting started

Cargo.toml:

[dependencies]
bigquery = "0.10"

The default feature tls-roots uses the native TLS roots of your system. Use tls-webpki-roots instead if you want the bundled Mozilla roots. It needs default-features = false, which also turns off the default feature auth-default-crypto, so enable it again, see Crypto provider:

[dependencies]
bigquery = { version = "0.10", default-features = false, features = ["tls-webpki-roots", "auth-default-crypto"] }

Crypto provider

Service account keys sign a JWT, and google-cloud-auth signs it with a rustls crypto provider. The default feature auth-default-crypto gives it the aws-lc-rs provider, which is built from C sources.

tls-webpki-roots builds aws-lc-rs for rustls already, so auth-default-crypto adds no C build with it. tls-roots without auth-default-crypto builds no aws-lc-rs.

Without auth-default-crypto, install a rustls provider before creating a client, with the same rustls version as google-cloud-auth (0.23):

[dependencies]
bigquery = { version = "0.10", default-features = false, features = ["tls-roots"] }
rustls = { version = "0.23", default-features = false, features = ["ring"] }
rustls::crypto::ring::default_provider().install_default().expect("Failed to install rustls crypto provider");

Without a provider:

  • BigQueryDb::new, with_options and for_default_project_id fail with a SystemError whose code is CryptoProviderMissing, when the application default credentials are a service account key, used directly or as the source of an impersonation;
  • with_options_service_account_key_file fails with the same error;
  • credentials passed to with_options_auth are not checked, and the first call that signs with a key panics.

Crypto provider error

Depends on your other dependencies you may see the error like:

no process-level CryptoProvider available -- call CryptoProvider::install_default() before this point

It appears when rustls cannot pick a provider by itself, because none or more than one is compiled into your application. Install one with CryptoProvider::install_default() before creating the client, as above.

Creating a client

BigQueryDb is the client. It opens two authenticated gRPC channels, one to the BigQuery v2 API for queries, jobs, datasets and tables, and one to the Storage API for reads and writes. Clones are cheap and share both channels, so create it once and clone it where you need it.

use bigquery::*;

async fn example() -> BigQueryResult<()> {
// Application default credentials, the project given explicitly
let db = BigQueryDb::new("my-gcp-project-id").await?;

// The project detected from GCP_PROJECT, PROJECT_ID or GCP_PROJECT_ID,
// the quota project of the local credentials or the metadata server
let db = BigQueryDb::for_default_project_id().await?;

// With options
let db = BigQueryDb::with_options(
    BigQueryDbOptions::new("my-gcp-project-id".to_string()).with_max_retries(5),
)
.await?;

// Everything else starts from the fluent API
let outcome = db.fluent().query("SELECT 1 AS x").execute().await?;
let _ = outcome;
Ok(())
}

for_default_project_id() fails with InvalidParametersError if no project can be detected.

The project ID is checked only for what would break a resource path: it must not be empty and must not contain / or a control character. Everything else is up to BigQuery.

Client options

BigQueryDbOptions has:

  • google_project_id: the project that runs the jobs and owns the datasets by default;
  • location: the location to send with jobs and queries, unset by default, see Locations;
  • max_retries: how many times a failed retryable request is sent again, 3 by default;
  • bigquery_api_url: overrides the v2 API endpoint, https://bigquery.googleapis.com;
  • bigquery_storage_api_url: overrides the Storage API endpoint, https://bigquerystorage.googleapis.com.

Each has a with_... builder method, as in the example above. db.options() returns the options a client was created with.

Google authentication

BigQueryDb::new, with_options and for_default_project_id use the application default credentials for the cloud-platform scope. They look for credentials in the following places, preferring the first location found:

  • A JSON file whose path is specified by the GOOGLE_APPLICATION_CREDENTIALS environment variable;
  • A JSON file in a location known to the gcloud command-line tool using gcloud auth application-default login;
  • On Google Compute Engine, Cloud Run, GKE, etc., the metadata server.

The JSON file can be a service account key, user credentials, impersonated service account credentials or a workload identity federation configuration.

For local development don’t confuse gcloud auth login with gcloud auth application-default login, since the first one authorizes only the gcloud tool to access the Cloud Platform.

Service account key file

use bigquery::*;

async fn example() -> BigQueryResult<()> {
let db = BigQueryDb::with_options_service_account_key_file(
    BigQueryDbOptions::new("my-gcp-project-id".to_string()),
    "/path/to/service-account.json".into(),
)
.await?;
let _ = db;
Ok(())
}

The key is used for the cloud-platform scope.

Other credentials

with_options_auth takes anything that converts into GoogleAuthHeaders of gcloud-sdk:

  • Credentials built with google-cloud-auth: impersonation, the metadata server, workload identity federation, a service account key with settings of your own, or a CredentialsProvider of your own;
  • IDTokenCredentials, for a server that takes Google ID tokens;
  • a GoogleAuthHeaders, for example the application default credentials or a service account key for other scopes.

The library re-exports the gcloud-sdk version it is built against as bigquery::gcloud_sdk, and google-cloud-auth as bigquery::gcloud_sdk::google_cloud_auth.

The application default credentials for other OAuth2 scopes:

use bigquery::gcloud_sdk::GoogleAuthHeaders;
use bigquery::*;

async fn example() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let auth = GoogleAuthHeaders::from_adc_with_scopes(vec![
    "https://www.googleapis.com/auth/bigquery".to_string(),
])
.await?;
let db = BigQueryDb::with_options_auth(
    BigQueryDbOptions::new("my-gcp-project-id".to_string()),
    auth,
)
.await?;
let _ = db;
Ok(())
}

A service account key built with google-cloud-auth, here to bill another project for quota:

use bigquery::gcloud_sdk::google_cloud_auth::credentials::service_account;
use bigquery::*;

async fn example() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let key = std::fs::read("/path/to/service-account.json")?;
let credentials = service_account::Builder::new(serde_json::from_slice(&key)?)
    .with_quota_project_id("my-quota-project-id")
    .build()?;
let db = BigQueryDb::with_options_auth(
    BigQueryDbOptions::new("my-gcp-project-id".to_string()),
    credentials,
)
.await?;
let _ = db;
Ok(())
}

Be aware that with_options_auth does not check for a rustls crypto provider, see Crypto provider.

Credentials that mint tokens refresh them on the Tokio runtime they are built in, so build them in the runtime that runs the client.

Both channels share one set of credentials, so a token is minted and refreshed once for both.

The token never reaches the library’s logs or spans. The one info! line at client creation logs the project and the two endpoints.

Endpoints

The library uses Google’s two global endpoints by default. You can change them, for example for a Private Service Connect endpoint or a proxy:

use bigquery::*;

async fn example() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let options = BigQueryDbOptions::new("my-gcp-project-id".to_string())
    .with_bigquery_api_url("https://bigquery-myendpoint.p.googleapis.com".parse()?)
    .with_bigquery_storage_api_url("https://bigquerystorage-myendpoint.p.googleapis.com".parse()?);
let db = BigQueryDb::with_options(options).await?;
let _ = db;
Ok(())
}

The URLs are url::Url (re-exported as bigquery::url). Creating the client fails with InvalidParametersError for a scheme other than http or https, or a URL without a host.

There is no emulator support. The BigQuery emulators speak only the REST API, and this library uses gRPC for everything.

The v2 API over gRPC is pre-GA and not documented by Google. It works for every call the library makes, and the library’s live tests run against it, but be aware Google can change it without notice.

Locations

Leave the location unset unless you need it. BigQuery finds the location of a dataset from the dataset itself and the location of a query from the tables it reads. A wrong location fails with DataNotFoundError, so setting one only adds a way to fail.

You need it when BigQuery cannot find it by itself, for example for a query that reads no table and should run in a particular region. You can set it for every query and job of a client, or for one query:

use bigquery::*;

const STOCKHOLM: BigQueryLocation = BigQueryLocation::from_static("europe-north2");

async fn example() -> BigQueryResult<()> {
// For every query and job of this client
let db = BigQueryDb::with_options(
    BigQueryDbOptions::new("my-gcp-project-id".to_string()).with_location(STOCKHOLM),
)
.await?;

// For one query, overriding the client's location
let outcome = db
    .fluent()
    .query("SELECT 1")
    .location(BigQueryLocation::new("EU")?)
    .execute()
    .await?;
let _ = outcome;
Ok(())
}

BigQueryLocation is just a name, since Google adds regions all the time. It accepts regions such as europe-north2 and multi-regions such as US and EU, and checks only that the name is not empty. Whether BigQuery knows it is checked by BigQuery. from_static checks it at compile time in a const.

The location BigQuery reports for a job is kept in its BigQueryJobRef, and the job calls (get_job, cancel_job, delete_job) send it back, so they find jobs in any region without any setting.

Retries

Requests that fail with a retryable error are sent again, up to max_retries times. The retryable errors are UNAVAILABLE, RESOURCE_EXHAUSTED, ABORTED, INTERNAL, BigQuery’s rate limit errors and transport errors. Before retry number n the client waits a random delay of up to 2^(n-1) seconds. Each retry is logged at warn! inside the call’s span.

A retried query does not run a DML statement twice, since every attempt carries the same request_id. The writer resends batches with its own rules, see Writing data.

Raw gRPC clients

For the calls the library has no API for, BigQueryDb gives you the raw gRPC clients on its shared channels: job_client(), dataset_client(), table_client(), model_client(), routine_client(), row_access_policy_client(), project_client(), read_client() and write_client().

use bigquery::*;

async fn example(db: BigQueryDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
use bigquery::gcloud_sdk::google::cloud::bigquery::v2::GetServiceAccountRequest;

let response = db
    .project_client()
    .get_service_account(GetServiceAccountRequest {
        project_id: db.options().google_project_id.clone(),
    })
    .await?;
println!("{}", response.into_inner().email);
Ok(())
}

Their types come from gcloud-sdk, re-exported as bigquery::gcloud_sdk, so they change with its version.

Running the examples

All examples available in the examples directory. Each one creates its own scratch dataset and deletes it at the end.

To run an example:

PROJECT_ID=<your-google-project-id> cargo run --example query

Reading data

The library reads rows in two ways, and decodes both with the same Arrow decoder, into your structures with serde or as raw Arrow record batches:

  • Reading tables: a table read through the Storage Read API, with column projection, typed filters and parallel streams. It runs no query and no job;
  • Queries: GoogleSQL with parameters, through the v2 Query call. A large result is read through the Storage Read API as well.

Table reads or queries compares the two, with billing hints and which one to use when.

Reading tables

The library reads tables through the BigQuery Storage Read API. A read runs no query and no job: BigQuery streams the table’s columns as Arrow, and the library decodes them into your structures with serde. When to read a table and when to query it, see Table reads or queries.

use bigquery::*;
use futures::TryStreamExt;
use futures::stream::BoxStream;
use serde::Deserialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const PEOPLE: BigQueryTableId = BigQueryTableId::from_static("people");
#[derive(Debug, Deserialize)]
struct Person {
    name: String,
    city: String,
    year: i64,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let people: BoxStream<BigQueryResult<Person>> = db
    .fluent()
    .select()
    .fields(paths!(Person::{name, city, year})) // Optionally select the columns needed
    .from(SHOP.table(PEOPLE))
    .filter(|filter| {
        filter.for_all([
            filter.field(path!(Person::city)).eq("Stockholm"),
            filter.field(path!(Person::year)).ge(2010),
        ])
    })
    .obj() // Reading rows as structures using serde
    .stream_query_with_errors()
    .await?;

let as_vec: Vec<Person> = people.try_collect().await?;
println!("{as_vec:?}");
Ok(())
}

A read needs the bigquery.readsessions.create permission on the project and read access to the table, the same as any Storage Read client.

Selecting columns

.fields(..) takes the column names to read. path! and paths! build them from the fields of your structure, so a renamed field is a compile error instead of a read that fails on a column the table does not have:

  • path!(Person::city) is "city";
  • paths!(Person::{name, city}) is vec!["name", "city"];
  • path!(Person::home.county) is "home.county", a subfield of a STRUCT column.

Without .fields(..), a typed read selects by itself the columns that the top-level fields of your structure name, so it does not read columns it would throw away. It costs one GetTable call to learn the table’s columns. A structure with #[serde(flatten)], a map, serde_json::Value or a tuple has no complete list of fields, so it reads every column. record_batches() without .fields(..) reads every column too.

The paths are Rust field names. A field under #[serde(rename = "...")] needs path_camel_case! for camelCase columns, or the column name as a plain string.

Be aware that BigQuery checks selected_fields against a schema that lags behind: a column added or renamed in the last 30 seconds or so is refused, and the read fails with SchemaMismatchError. The automatic projection falls back to reading every column in that case and logs a warning.

Filtering

.filter(..) takes a closure that receives a BigQueryFilterBuilder and returns an Option<BigQueryFilter>:

  • f.field(..) with eq, neq, lt, le, gt, ge, is_null, is_not_null, is_in and is_not_in;
  • f.for_all for AND conditions;
  • f.for_any for OR conditions;
  • f.not for NOT.

You can nest them, and a None entry is dropped, so optional conditions can be written inline:

use bigquery::*;
use serde::Deserialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const PEOPLE: BigQueryTableId = BigQueryTableId::from_static("people");
#[derive(Deserialize)]
struct Person {
    name: String,
    city: String,
    year: i64,
}
async fn example(db: BigQueryDb, min_year: Option<i64>) -> BigQueryResult<()> {
let people: Vec<Person> = db
    .fluent()
    .select()
    .from(SHOP.table(PEOPLE))
    .filter(|filter| {
        filter.for_all([
            filter.for_any([
                filter.field(path!(Person::city)).is_in(["Malmö", "Lund"]),
                filter.field(path!(Person::city)).is_null(),
            ]),
            min_year.and_then(|year| filter.field(path!(Person::year)).ge(year)),
        ])
    })
    .obj()
    .query()
    .await?;
let _ = people;
Ok(())
}

Storage Read has no query parameters: the filter is SQL text in the session’s row_restriction. The builder writes values into it as escaped GoogleSQL literals and column names as quoted identifiers, so a value cannot change the condition it is in, whatever it holds. Values are any Serialize, with the same mapping as query parameters: a string is a STRING, an integer an INT64, BigQueryTimestamp a TIMESTAMP, etc. A condition can name any column, selected or not.

A few things to know:

  • a NULL value is refused, since n = NULL matches no row; use is_null() instead;
  • GoogleSQL’s three-valued logic applies: a row where the column is NULL matches neither neq(..) nor not(eq(..));
  • a closure that returns None reads every row;
  • a value without a literal form fails the read before any request, with SerializeError;
  • the whole restriction is at most 1 MB, a limit BigQuery checks.

Full example available here.

Raw SQL filters

.filter_sql(..) sends condition text as it is, for example "year > 2010 AND city != ''".

Be aware not to build this text from user input. It is SQL, and a value spliced into it can rewrite the condition. Use .filter(..) for every condition that carries a value, and keep .filter_sql(..) for fixed text from your own code. .filter(..) and .filter_sql(..) set the same restriction, so the last call wins.

Snapshots and sampling

.snapshot_time(..) reads the table as it was at that time instead of now, using BigQuery’s time travel. It works only within the time travel window of the dataset, 7 days by default:

use bigquery::*;
use serde::Deserialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const PEOPLE: BigQueryTableId = BigQueryTableId::from_static("people");
#[derive(Deserialize)]
struct Person {
    name: String,
}
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let an_hour_ago = BigQueryInstant::now() - jiff::SignedDuration::from_hours(1);

let people: Vec<Person> = db
    .fluent()
    .select()
    .from(SHOP.table(PEOPLE))
    .snapshot_time(an_hour_ago)
    .obj()
    .query()
    .await?;
let _ = people;
Ok(())
}

.sample_percentage(..) reads a random sample of about that percentage of the table, above 0 and up to 100. BigQuery samples by storage blocks, so treat the percentage as approximate, especially on small tables.

Reading rows

.obj::<T>() decodes the rows into T with the library’s own Arrow decoder, which covers every BigQuery type. Then pick one of:

  • query() to collect every row into a Vec<T>, failing on the first error;
  • stream_query_with_errors() to stream the rows, with every failure as an Err item. A row that fails to decode is one Err(DeserializeError) and the stream goes on;
  • stream_query() to stream only the rows that decode. Failures are logged at error! and skipped.

Be aware that a stream from stream_query() that ends does not mean every row was read: a read stream that failed for good ends it too, only with a log line. Use stream_query_with_errors() when you need to know.

Rows are decoded on the task that reads their stream, so T is Send + 'static and cannot borrow from the batch.

Record batches

.record_batches() streams the Arrow RecordBatches as BigQuery sent them, after the IPC decode and decompression. It is a bit faster than typed rows and the way to hand the data to other Arrow tools. arrow_array and arrow_schema are re-exported, so you use the same Arrow version as the library.

BigQueryBatchRows decodes a batch into typed rows later, with the same mapping as .obj(). A row that fails is one Err item, and the rows after it still decode:

use bigquery::*;
use futures::TryStreamExt;
use serde::Deserialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const PEOPLE: BigQueryTableId = BigQueryTableId::from_static("people");
#[derive(Deserialize)]
struct Person {
    name: String,
    year: i64,
}
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let mut batches = db
    .fluent()
    .select()
    .fields(paths!(Person::{name, year}))
    .from(SHOP.table(PEOPLE))
    .record_batches()
    .await?;

while let Some(batch) = batches.try_next().await? {
    println!("{} rows, {} columns", batch.num_rows(), batch.num_columns());
    for person in BigQueryBatchRows::<Person>::new(&batch) {
        let person = person?;
        println!("{} {}", person.name, person.year);
    }
}
Ok(())
}

BigQueryBatchRows is not Send, so decode a batch where you hold it and do not keep the iterator across an .await.

Full example available here.

Parallel streams and resume

One read opens one read session, and BigQuery splits the table into several read streams. Each stream runs on its own task: ReadRows, the Arrow decode and, for typed reads, the row decode. The streams meet in one bounded channel, so a slow consumer slows the read down instead of buffering the table in memory, and rows from different streams arrive in no particular order.

BigQueryReadOptions sets how the session opens:

  • max_stream_count: the most streams to ask for, the machine’s available parallelism by default;
  • preferred_min_stream_count: the fewest streams BigQuery should aim for;
  • compression: Lz4 by default, Zstd or None.
use bigquery::*;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const PEOPLE: BigQueryTableId = BigQueryTableId::from_static("people");
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let batches = db
    .fluent()
    .select()
    .from(SHOP.table(PEOPLE))
    .options(
        BigQueryReadOptions::new()
            .with_max_stream_count(8)
            .with_compression(BigQueryReadCompression::Zstd),
    )
    .record_batches()
    .await?;
let _ = batches;
Ok(())
}

BigQuery decides the real count and often gives fewer streams than asked for. In the benchmarks it gave 4 streams of the 16 asked for on a 1M-row table, and 1 for a 200,000-row query result.

A stream that fails with a retryable error is resumed at its row offset after a backoff, so the rows it already sent are not read again. Consecutive failures are capped by the client’s max_retries. A stream that fails for good is one Err item on the read, and then the whole read ends, since a scan that lost a stream is incomplete.

One case is never retried: an INTERVAL whose time part does not fit Arrow’s nanoseconds fails the stream on BigQuery’s side, and every resume would fail the same way. Read such a column through a query with CAST(.. AS STRING) instead.

The read session, its stream count, BigQuery’s estimate of the bytes scanned and the rows and bytes read are recorded on the BigQuery Read tracing span.

Queries

The library runs GoogleSQL queries through the v2 Query call over gRPC and reads the results as your own structures with serde, or as Arrow record batches. When to query and when to read the table instead, see Table reads or queries.

use bigquery::*;
use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct WordCount {
    word: String,
    word_count: i64,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let words: Vec<WordCount> = db
    .fluent()
    .query(
        "SELECT word, word_count FROM `bigquery-public-data.samples.shakespeare` \
         WHERE corpus = @corpus AND word_count >= @min_count \
         ORDER BY word_count DESC LIMIT 10",
    )
    .param("corpus", "hamlet")
    .param("min_count", 100)
    .obj::<WordCount>()
    .query()
    .await?;
println!("{words:?}");
Ok(())
}

Columns map to the fields of your structure by name, with the types described in Type mapping. The target type is DeserializeOwned + Send + 'static, since the rows of a large result are decoded on each read stream’s task.

Reading the results

.obj::<T>() has these terminals:

  • query(): every row in a Vec, failing on the first error;
  • stream_query_with_errors(): a stream of BigQueryResult<T>. A row that fails to decode is one Err item and the stream goes on; a read stream that fails for good is one Err and then the stream ends;
  • stream_query(): a stream of T. Failures are logged at error! and skipped, so a stream that ended does not mean every row was read. Use stream_query_with_errors() if you need to tell the two apart;
  • query_with_stats() and stream_query_with_stats(): the same as query() and stream_query_with_errors(), with what the job used, see Job stats.
use bigquery::*;
use futures::TryStreamExt;
use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct WordCount {
    word: String,
    word_count: i64,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let mut words = db
    .fluent()
    .query("SELECT word, word_count FROM `bigquery-public-data.samples.shakespeare`")
    .obj::<WordCount>()
    .stream_query_with_errors()
    .await?;

while let Some(word) = words.try_next().await? {
    println!("{}: {}", word.word, word.word_count);
}
Ok(())
}

The query terminal waits for the job to finish before the first row streams. Dropping the stream stops the reading, and does not cancel the job, see Cancellation.

Rows of a large result come from several read streams at once, so they arrive in no particular order, even with ORDER BY. A result that comes inline keeps its order. If the order matters for a large result, sort the rows on your side, or ask for one read stream with .read_options(BigQueryReadOptions::new().with_max_stream_count(1)). Google’s Python client asks for one stream for ORDER BY queries. The library does not test that one stream keeps the order of the result.

To get the result as Arrow, without serde, use record_batches() directly on the query:

use bigquery::*;
use futures::TryStreamExt;

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let mut batches = db
    .fluent()
    .query("SELECT corpus, COUNT(*) AS words FROM `bigquery-public-data.samples.shakespeare` GROUP BY corpus")
    .record_batches()
    .await?;

while let Some(batch) = batches.try_next().await? {
    println!("{} rows, schema {:?}", batch.num_rows(), batch.schema());
}
Ok(())
}

RecordBatch is arrow_array::RecordBatch, re-exported as bigquery::arrow_array.

Parameters

Values go into the query only as query parameters, which BigQuery binds on its side. The library never writes a value into the SQL text, so a value with quotes, backticks, comments or @x in it is just a value. A corpus of such values is tested on all the parameter forms below, as STRING, ARRAY, STRUCT and JSON, against a fake server. Against BigQuery itself a few injection payloads are tested as STRING parameters.

Be aware this protects only the values: SQL you build with format! from untrusted input is still your SQL.

Named parameters

.param(name, value) adds @name, with the type inferred from the value’s serde form:

  • integers are INT64, floats FLOAT64, bool BOOL;
  • strings and char are STRING, bytes (serde_bytes) BYTES, unit enum variants STRING;
  • sequences are ARRAY of the first element’s type;
  • structures and string-keyed maps are STRUCT, with the fields in order;
  • the library’s wrappers are their own types: BigQueryTimestamp, BigQueryDate, BigQueryTime, BigQueryDateTime, BigQueryJson, BigQueryInterval, BigQueryRange, and BigQueryDecimal as NUMERIC, or BIGNUMERIC when the value does not fit NUMERIC.
use bigquery::*;
use serde::{Deserialize, Serialize};

#[derive(Serialize)]
struct Window {
    earliest: BigQueryTimestamp,
    latest: BigQueryTimestamp,
}

#[derive(Debug, Deserialize)]
struct Order {
    id: i64,
    city: String,
}

async fn example(db: BigQueryDb) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let window = Window {
    earliest: BigQueryTimestamp("2026-10-01T00:00:00Z".parse()?),
    latest: BigQueryTimestamp("2026-10-02T00:00:00Z".parse()?),
};

let orders: Vec<Order> = db
    .fluent()
    .query(
        "SELECT id, city FROM shop.orders \
         WHERE city IN UNNEST(@cities) AND placed_at BETWEEN @window.earliest AND @window.latest \
         AND total >= @min_total",
    )
    .param("cities", vec!["Malmö", "Lund"])
    .param("window", window)
    .param("min_total", BigQueryDecimal("99.90"))
    .obj::<Order>()
    .query()
    .await?;
let _ = orders;
Ok(())
}

Some values have no type to infer: None, an empty sequence, elements of different types. And a plain jiff value serializes as text, so .param sends it as STRING. For these use .param_as(name, type, value), which takes the type and accepts the value in any form the library writes for that type. None is a NULL of that type.

use bigquery::*;

async fn example(db: BigQueryDb, since: Option<jiff::Timestamp>) -> BigQueryResult<()> {
let outcome = db
    .fluent()
    .query(
        "SELECT COUNT(*) FROM shop.orders \
         WHERE (@since IS NULL OR placed_at >= @since) \
         AND id NOT IN UNNEST(@excluded_ids) \
         AND ST_DWITHIN(location, @store, 10000)",
    )
    .param_as("since", BigQueryFieldType::Timestamp, since)
    .param_as("excluded_ids", BigQueryParamType::array_of(BigQueryFieldType::Int64), Vec::<i64>::new())
    .param_as("store", BigQueryFieldType::Geography, "POINT(13.0 55.6)")
    .execute()
    .await?;
let _ = outcome;
Ok(())
}

.params(&value) adds every top-level field of a structure or string-keyed map as a named parameter, inferred as by .param:

use bigquery::*;
use serde::Serialize;

#[derive(Serialize)]
struct Filter {
    city: String,
    min_total: f64,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let filter = Filter {
    city: "Malmö".to_string(),
    min_total: 100.0,
};
let outcome = db
    .fluent()
    .query("DELETE FROM shop.orders WHERE city = @city AND total < @min_total")
    .params(&filter)
    .execute()
    .await?;
let _ = outcome;
Ok(())
}

Parameter names must be GoogleSQL identifiers: ASCII letters, digits and _, not starting with a digit.

SQL files

A longer statement can live in its own .sql file. sql_file!(path, names...) embeds the file into the binary at compile time and lists the names of its named parameters. The path is relative to the Rust file that calls the macro, the same as include_str!, and Cargo rebuilds the crate when the file changes.

With query/sql/top_words.sql:

SELECT word, word_count
FROM `bigquery-public-data.samples.shakespeare`
WHERE corpus = @corpus
  AND word_count >= @min_count
ORDER BY word_count DESC
LIMIT 10

the query takes the file in place of the text, and the values are bound with .param as usual:

use bigquery::*;
use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct WordCount {
    word: String,
    word_count: i64,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let top_words: Vec<WordCount> = db
    .fluent()
    .query(bigquery::sql_file!("query/sql/top_words.sql", corpus, min_count))
    .param("corpus", "hamlet")
    .param("min_count", 100)
    .obj()
    .query()
    .await?;
let _ = top_words;
Ok(())
}

The names are checked twice:

  • at compile time, against the file: a parameter the file uses and the list leaves out, or a listed name the file never uses, fails the build;
  • at run time, against the .param and .params calls, before anything is sent: a listed name with no value, or a value bound to a name the list does not have, fails the terminal with InvalidParametersError naming the parameter.

The check reads the file with GoogleSQL’s lexical rules, so an @ inside a string, a quoted identifier or a comment is not a parameter, and neither is a @@ system variable. The rest follows BigQuery’s parser:

  • whitespace and comments may stand between @ and the name;
  • a backtick-quoted name such as @`corpus` is listed as corpus;
  • a STRUCT parameter read as @window.earliest is listed as window;
  • names are compared ignoring case, so @Corpus is listed as corpus and bound with .param("corpus", ...).

Nothing is sent to BigQuery to check the statement itself; use a dry run for that.

Positional parameters

.positional_param(value) and .positional_param_as(type, value) add the next ?, inferred or typed the same way as the named ones:

use bigquery::*;

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let outcome = db
    .fluent()
    .query("UPDATE shop.orders SET status = ? WHERE id = ?")
    .positional_param("shipped")
    .positional_param(42)
    .execute()
    .await?;
let _ = outcome;
Ok(())
}

Named and positional parameters cannot be mixed in one query.

Full example available here.

Errors

The builder methods never fail. A parameter that cannot be encoded is kept in the builder, and the terminal returns its error, InvalidParametersError or SerializeError, without sending anything.

Job settings

The query builder also has:

  • .location(..): where the job runs, see Locations;
  • .default_dataset(..): the dataset unqualified table names resolve in, a BigQueryDatasetId in the client’s project or a BigQueryDatasetRef for another project;
  • .label(key, value) and .labels(..): labels on the job;
  • .maximum_bytes_billed(bytes): the job fails without running if it would bill more;
  • .use_query_cache(false): BigQuery answers from its query cache by default;
  • .timeout(..): how long the first Query call waits for the job, 10 seconds by default. A job still running then is polled until it completes;
  • .job_timeout(..): how long BigQuery lets the job run before it cancels the job itself;
  • .request_id(..): the idempotency key of the Query call, see DML;
  • .inline_rows_limit(..) and .read_options(..): how the rows come back, see Where the rows come from.
  • .destination_table(..): a table of your own for the result, see Destination tables.
use bigquery::*;
use serde::Deserialize;
use std::time::Duration;

const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");

#[derive(Debug, Deserialize)]
struct Order {
    id: i64,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let orders: Vec<Order> = db
    .fluent()
    .query("SELECT id FROM orders WHERE status = 'new'")
    .default_dataset(SHOP)
    .labels([("team", "shop"), ("report", "new-orders")])
    .maximum_bytes_billed(1_000_000_000)
    .job_timeout(Duration::from_secs(60))
    .obj::<Order>()
    .query()
    .await?;
let _ = orders;
Ok(())
}

Short query mode

By default the library sends every query with BigQuery’s short query mode (JOB_CREATION_OPTIONAL), as Google’s own Rust crate does. BigQuery then answers a short query whose result fits in the first response without creating a job. In the benchmarks creating the job cost 80-165 ms.

A query that ran without a job:

  • reports a query_id and no job in its stats and outcome;
  • is still listed in the INFORMATION_SCHEMA.JOBS views, with its query_id as the job_id and with its labels;
  • has nothing for get_job to read or cancel_job to cancel. GetJob refuses its query_id, since it is not a job ID.

BigQuery still creates a job for a query that runs long, a result too large for the response, and DML and DDL statements (every one of them got a job in the library’s live tests).

.job_creation_required() makes every query create a job. Use it when the query needs a job resource: for job history read through the job calls, for a job ID it is sure to get, or to cancel it by its job.

use bigquery::*;

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let outcome = db
    .fluent()
    .query("SELECT 1")
    .job_creation_required()
    .execute()
    .await?;

let job = outcome.job.expect("a required job is always reported");
println!("{} in {:?}", job.job_id, job.location);
Ok(())
}

A retry of a failed Query call always requires a job, whatever the first attempt asked for. BigQuery recognises a repeated request_id and returns the job of the first attempt only in that mode, and answers it with AlreadyExists in the optional one.

DML and DDL

execute() runs a statement, waits for it, and returns a BigQueryQueryOutcome without reading any rows:

use bigquery::*;

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let outcome = db
    .fluent()
    .query("UPDATE shop.orders SET status = 'shipped' WHERE id = @id")
    .param("id", 42)
    .execute()
    .await?;

println!(
    "{:?} changed {:?} rows: {:?}",
    outcome.statement_type, outcome.num_dml_affected_rows, outcome.dml_stats
);
Ok(())
}

BigQueryQueryOutcome is the same BigQueryJobStats the queries return, see Job stats. For DML it has:

  • statement_type: Some(BigQueryStatementType::Update), Insert, Delete, Merge, etc.;
  • num_dml_affected_rows: the rows the statement changed;
  • dml_stats: the rows inserted, updated and deleted, as BigQueryDmlStats.

DDL (CREATE TABLE, ALTER TABLE, etc.) goes through execute() the same way. .obj::<T>() on a DML or DDL statement is just an empty result.

Every terminal call sends a fresh random request_id unless you set one, and every retry of that call repeats it, so a retried DML statement is not run twice. BigQuery keeps the keys for a limited time window. Set your own BigQueryRequestId with .request_id(..) if your code may send the same statement again on its own, for example after a restart:

use bigquery::*;

async fn example(db: BigQueryDb, order_id: i64) -> BigQueryResult<()> {
let request_id = BigQueryRequestId::new(format!("ship-order-{order_id}"))?;
let outcome = db
    .fluent()
    .query("UPDATE shop.orders SET status = 'shipped' WHERE id = @id")
    .param("id", order_id)
    .request_id(request_id)
    .execute()
    .await?;
let _ = outcome;
Ok(())
}

A statement that fails is an error from the terminal: a syntax error, a missing table or ERROR() in the SQL come back from the Query call itself, and a job that finished with an error is a JobError with BigQuery’s reason and messages.

Dry run

dry_run() validates the statement and returns the bytes it would process and the schema of its result, without running it. It never creates a job and bills nothing.

With a destination table the dry run is sent as a dry-run job with the same destination, so BigQuery checks that part too. It still creates no job and writes nothing.

use bigquery::*;

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let estimate = db
    .fluent()
    .query("SELECT * FROM `bigquery-public-data.samples.shakespeare`")
    .dry_run()
    .await?;

println!("Would process {:?} bytes", estimate.total_bytes_processed);
if let Some(schema) = estimate.schema {
    for field in schema.fields {
        println!("{}: {}", field.name, field.field_type);
    }
}
Ok(())
}

Full example available here.

Job stats

query_with_stats() returns the rows together with what the query used, as BigQueryJobStats:

  • job: the job that ran the query, None when BigQuery ran it without one;
  • query_id: the ID BigQuery gave the query, with or without a job;
  • statement_type: the kind of statement;
  • total_rows: rows in the result;
  • total_bytes_processed and total_bytes_billed: bytes processed, and bytes billed after BigQuery’s rounding and minimums;
  • total_slot_ms: slot milliseconds the job used;
  • cache_hit: whether the query cache answered;
  • num_dml_affected_rows and dml_stats: for DML.
use bigquery::*;
use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct Total {
    words: i64,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let (rows, stats) = db
    .fluent()
    .query("SELECT SUM(word_count) AS words FROM `bigquery-public-data.samples.shakespeare`")
    .obj::<Total>()
    .query_with_stats()
    .await?;

println!(
    "{rows:?}: {:?} bytes billed, {:?} slot ms, cache hit {:?}",
    stats.total_bytes_billed, stats.total_slot_ms, stats.cache_hit
);
Ok(())
}

A figure BigQuery did not report is None, never 0. The library makes no extra call for the stats: every figure comes from a response the query receives anyway. A result answered in the first Query response has only that response’s figures. A larger one also has the figures of the job’s statistics, which the query reads to find its destination table.

stream_query_with_stats() returns the stats together with the stream, before any row. The query waits for its job to finish before the first row streams, so every job figure is already known then. What reading the rows cost is on the read’s span, see Observability.

Even a query that reads no table uses slots. On 1,000 generated rows BigQuery reported 0 bytes processed and billed, and 25 slot ms answered inline, 152 slot ms read through Storage Read.

Full example available here.

Cancellation

Dropping a query’s stream or future does not cancel its job, BigQuery runs it to the end. To limit how long a job can run, set .job_timeout(..), and BigQuery cancels it itself.

To cancel a job from your code use cancel_job with its BigQueryJobRef. It returns once BigQuery accepted the request, which is before the job stops, and a job that already finished stays finished.

A query terminal returns only when its job has finished, so to cancel a long query find it while it runs, for example by a label:

use bigquery::*;
use futures::StreamExt;

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
// The long query was started elsewhere with .label("report", "monthly-totals")
let mut running = db
    .stream_jobs(BigQueryListJobsParams::new().with_states(vec![BigQueryJobState::Running]))
    .await?;

while let Some(job) = running.next().await {
    if job.labels.get("report") == Some("monthly-totals") {
        db.cancel_job(&job.reference).await?;
    }
}
Ok(())
}

A query that runs long always has a job, even in short query mode, since BigQuery creates one for a query that outlives the first Query call.

Where the rows come from

Every query asks BigQuery for an Arrow result. Then:

  • a result that comes complete in the first Query response is decoded from the inline Arrow in that response;
  • a larger result, or one whose job outlived the first call, is read from the job’s destination table through the Storage Read API, with several streams in parallel;
  • a query with a destination table is always read from that table through the Storage Read API;
  • a statement without rows, such as DML or DDL, reads nothing.

Both paths use the same Arrow decoder as table reads, so a type maps the same way in a query and in a table read. The query’s span records which path it took, as /bigquery/route.

BigQuery decides how much of a result goes inline. A small result reaches its last row sooner inline, since a Storage Read session costs a call to open before the first row. A large result is much faster through Storage Read: in the benchmarks 200,000 rows took 1.16 s, against 4.39 s for the official crate and 4.55 s for Python reading the same result as REST pages.

.inline_rows_limit(rows) caps the rows of the first response, so a result with more goes to Storage Read. .read_options(..) sets how that read opens its session, the same BigQueryReadOptions as for table reads:

use bigquery::*;

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let batches = db
    .fluent()
    .query("SELECT * FROM `bigquery-public-data.samples.shakespeare`")
    .inline_rows_limit(10_000)
    .read_options(
        BigQueryReadOptions::new()
            .with_max_stream_count(4)
            .with_compression(BigQueryReadCompression::Zstd),
    )
    .record_batches()
    .await?;
let _ = batches;
Ok(())
}

By default the read asks for as many streams as the machine has parallelism, with LZ4 compression, and BigQuery decides how many it actually gives.

Destination tables

BigQuery writes every query result into a temporary table, and you can name a table of your own instead. The table is a setting of the job, so the query stays a plain SELECT:

  • .destination_table(table): the job writes the result into table, and fails if the table already holds rows. This is BigQuery’s own default, WRITE_EMPTY, so nothing is overwritten unless you ask for it;
  • .append_to_destination_table(table): the result is added after the rows the table holds (WRITE_APPEND);
  • .dangerously_overwrite_destination_table(table): the result replaces every row of the table and its schema (WRITE_TRUNCATE).

BigQuery creates the table when it does not exist, with the schema of the result. Each write applies only when the job succeeds, as one update of the table.

use bigquery::*;
use serde::Deserialize;

const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const CITY_TOTALS: BigQueryTableId = BigQueryTableId::from_static("city_totals");

#[derive(Debug, Deserialize)]
struct CityTotal {
    city: String,
    total: f64,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let totals: Vec<CityTotal> = db
    .fluent()
    .query(
        "SELECT c.city, SUM(o.total) AS total FROM shop.orders o \
         JOIN shop.customers c ON c.id = o.customer_id GROUP BY c.city",
    )
    .destination_table(SHOP.table(CITY_TOTALS))
    .obj::<CityTotal>()
    .query()
    .await?;
let _ = totals;
Ok(())
}

Every terminal works with a destination. The rows are read back from the table through the Storage Read API, so after an append they are the whole table’s, the rows it held before included. execute() writes the table and reads nothing back.

Be aware a table keeps no row order, so the rows come back in no particular order, even with ORDER BY in the query and even for a small result, which never comes inline here. Sort them on your side if the order matters.

A write into a table that already holds rows, with .destination_table(..), fails with a DataConflictError, BigQuery’s AlreadyExists for that table.

The query always runs as a job, inserted with InsertJob, since the Query call has no destination field. So .job_creation_required(), .inline_rows_limit(..) and .request_id(..) do not apply: the job’s own ID, new for each terminal call, makes a retried InsertJob safe. The query cache does not answer a query with a destination table either.

Be aware the table is not temporary: it stays until you delete it or it expires, and its storage and the Storage Read of it are billed, see Billing hints.

Full example available here.

Table reads or queries

The library reads data in two ways:

  • a table read, select().from(..) with .filter(..), goes through the Storage Read API. It runs no query and no job;
  • a query, query(..) with named parameters, runs GoogleSQL through the v2 Query call.

Both decode rows with the same Arrow decoder, so a type maps the same way in each, see Type mapping.

The same rows both ways:

use bigquery::*;
use serde::Deserialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const PEOPLE: BigQueryTableId = BigQueryTableId::from_static("people");
#[derive(Debug, Deserialize)]
struct Person {
    name: String,
    city: String,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let read: Vec<Person> = db
    .fluent()
    .select()
    .from(SHOP.table(PEOPLE))
    .filter(|filter| filter.field(path!(Person::city)).eq("Malmö"))
    .obj()
    .query()
    .await?;

let queried: Vec<Person> = db
    .fluent()
    .query("SELECT name, city FROM shop.people WHERE city = @city")
    .param("city", "Malmö")
    .obj::<Person>()
    .query()
    .await?;
let _ = (read, queried);
Ok(())
}

Comparison

Table readQuery
What it readsOne table: the columns you select, the rows the filter keeps, at a snapshot time or as a sampleAny GoogleSQL: joins, aggregates, ORDER BY, LIMIT, views, computed columns
WritesNoDML and DDL
Values in conditionsEscaped literals in the session’s row_restriction, Storage Read has no parametersQuery parameters, the value never becomes SQL text
JobNone, so no job stats, dry run or cancellationShort query mode answers small ones without a job, others get one
Columns readThe ones your structure names, by itselfThe ones the SELECT names
First row of a small resultAfter opening a read sessionInline in the first response, about 0.1 s in the benchmarks
Large resultsSeveral streams in parallel, each resumed at its row offset after a retryable errorRead from the job’s destination table, through the same Storage Read path
BillingStorage Read: bytes readQuery: bytes processed, then nothing for reading the result

Be aware that Storage Read cannot read views, logical or materialized, and external tables, as the Storage Read API limitations say. Query them instead.

Billing hints

Prices here are the on-demand list prices in USD from BigQuery pricing in October 2026, so check them for your region and contract:

  • a table read is billed $1.10 per TiB read, and the first 300 TiB a month for each billing account are free. Reads within the same location are free of data transfer;
  • a query is billed $6.25 per TiB processed, with the first 1 TiB a month free. It bills every column the query touches over the whole table, unless partitioning or clustering prune it, even with LIMIT, and at least 10 MB per table referenced;
  • reading a query result is free: the result is a temporary table, and reads of those are not billed. Neither is a query answered from the query cache;
  • a table read bills the columns it reads, so select only the ones you need. The automatic projection from your structure does it for you, see Selecting columns.

So for a scan of a large table a table read is much cheaper: a bit more than a sixth of the query price per byte, and free under the monthly allowance. In the benchmarks the 215 MB table scans went through the Storage Read free tier, while each SELECT * scan of the official crate billed the table. Row filters were not measured for bytes billed, so do not count on a filter making a table read cheaper.

.maximum_bytes_billed(..) makes a query fail without running if it would bill more, and a dry run tells you the bytes before you run it.

Which one to use

  • Rows of one table, filtered by values: a table read. Exports, syncs, full scans, “the orders of this customer”. It needs no job, reads in parallel and resumes by itself;
  • Joins, aggregates, ordering, LIMIT, views or computed columns: a query, with named parameters for every value;
  • A small lookup, a few rows by key: a query. Short query mode returns it in one call, while a table read opens a session first;
  • DML, DDL, or when you need job stats, labels on the job or a dry run: a query;
  • Arrow for your own processing: either, both have record_batches().

Many rows from a complex query

A query with joins or aggregates that returns a lot of rows needs no table of your own and no second step. BigQuery writes every query result to a temporary table, and the library reads a large one from there through the Storage Read API, in parallel streams, see Where the rows come from. Stream the rows instead of collecting them all:

use bigquery::*;
use futures::StreamExt;
use serde::Deserialize;
#[derive(Debug, Deserialize)]
struct OrderWithCity {
    id: i64,
    total: f64,
    city: String,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let mut orders = db
    .fluent()
    .query(
        "SELECT o.id, o.total, c.city FROM shop.orders o \
         JOIN shop.customers c ON c.id = o.customer_id \
         WHERE o.total > @minimum",
    )
    .param("minimum", 100.0)
    .obj::<OrderWithCity>()
    .stream_query()
    .await?;
while let Some(order) = orders.next().await {
    println!("{order:?}");
}
Ok(())
}

Reading that temporary table is free, so you pay only for the query. .read_options(..) sets the number of read streams and the compression, as for a table read.

BigQuery limits how large a query result can be in its temporary table, see the maximum response size in Quotas and limits. For a result above it, write the result into a table of your own with .destination_table(..), see Destination tables. The query is the same, and the rows are read back from that table through the same Storage Read path:

use bigquery::*;
use futures::StreamExt;
use serde::Deserialize;
#[derive(Debug, Deserialize)]
struct OrderWithCity {
    id: i64,
    total: f64,
    city: String,
}
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS_EXPORT: BigQueryTableId = BigQueryTableId::from_static("orders_export");

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let mut orders = db
    .fluent()
    .query(
        "SELECT o.id, o.total, c.city FROM shop.orders o \
         JOIN shop.customers c ON c.id = o.customer_id",
    )
    .destination_table(SHOP.table(ORDERS_EXPORT))
    .obj::<OrderWithCity>()
    .stream_query()
    .await?;
while let Some(order) = orders.next().await {
    println!("{order:?}");
}

db.fluent().schema().table(SHOP.table(ORDERS_EXPORT)).delete().await?;
Ok(())
}

The default refuses to write into a table that already holds rows, so a second run fails until the table is deleted. Use .dangerously_overwrite_destination_table(..) to replace it instead. Give the dataset a default table expiration, so the table does not stay behind if your process stops before the delete.

That table is a table of your own, so its storage is billed while it exists, and reading it back is billed as Storage Read, see Billing hints.

Be aware not to splice values into SQL text in either path. Use .filter(..) instead of .filter_sql(..), and .param(..) instead of format! in the query text.

Writing data

The library writes rows through the BigQuery Storage Write API. Rows are your structures, serialized with serde straight into protobuf against the table’s schema, so there is no JSON and no schema to declare on the client. If your data is in Arrow already, the library writes raw record batches too, see Arrow record batches.

There are two ways to write:

  • db.fluent().insert() for rows you already have: it opens a writer, writes every row, finishes and returns a summary;
  • db.create_streaming_writer() for a long-running producer: one writer you keep open and write rows to as they come.

Both batch the rows by themselves, so you never need to batch before writing.

Inserts

use bigquery::*;
use serde::Serialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[derive(Serialize)]
struct Order {
    id: i64,
    customer: String,
    placed_at: jiff::Timestamp,
}

async fn example(db: BigQueryDb, orders: Vec<Order>) -> BigQueryResult<()> {
// One row
db.fluent()
    .insert()
    .into(SHOP.table(ORDERS))
    .object(&orders[0])
    .execute()
    .await?;

// Many rows
let summary = db
    .fluent()
    .insert()
    .into(SHOP.table(ORDERS))
    .objects(&orders)
    .execute()
    .await?;
println!("{} rows written", summary.rows_written);
Ok(())
}

objects(..) takes anything iterable whose items are Serialize, so a Vec, a slice or an iterator over rows built on the fly all work. execute() returns the first failed batch’s error, or a BigQueryWriteSummary with the rows written, the batches and the bytes sent. A row that does not serialize stops it with SerializeError. The rows before it may or may not be written: only the requests already sent can land, and the rows still waiting in a batch are dropped. If you need all rows or none, use .atomic() or check the rows before the insert.

Every execute() opens a write stream first, which is a round trip of about 300 ms. That is fine for a load of many rows, but for many small writes keep one streaming writer instead.

Full example available here.

Streaming writer

use bigquery::*;
use futures::StreamExt;
use serde::Serialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[derive(Serialize)]
struct Order {
    id: i64,
    customer: String,
}
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let (mut writer, mut responses) = db
    .create_streaming_writer::<Order>(SHOP.table(ORDERS))
    .await?;

// Reading the responses is optional, one item per batch
let responses_task = tokio::spawn(async move {
    while let Some(response) = responses.next().await {
        match response {
            Ok(written) => println!("batch {} written", written.batch_index),
            Err(err) => eprintln!("batch failed: {err}"),
        }
    }
});

for id in 0..10_000 {
    let order = Order {
        id,
        customer: format!("customer-{id}"),
    };
    writer.write(&order).await?;
}

let summary = writer.finish().await?;
println!("{} written, {} failed", summary.rows_written, summary.rows_failed);
let _ = responses_task.await;
Ok(())
}

The writer owns one AppendRows connection, run by a background task. It is Send but not Clone, so open more writers if you need more connections. write_all(..) writes several rows in order.

The response stream yields one BigQueryWriteResponse or one error per batch, in batch order, and ends when the writer finishes. Reading it is optional, finish() reports the failed rows either way.

Be aware not to just drop the writer instead of calling finish(). That logs a warning, drops the rows not acknowledged yet and leaves a pending stream uncommitted.

db.create_streaming_writer_with_options(..) takes BigQueryStreamingWriteOptions, the same options as .options(..) on an insert.

Full example available here.

Built-in batching

Rows are encoded as you write them and collected into a batch, one AppendRows request. A batch is sent when one of these comes first:

  • size: the next row would take it over max_request_bytes, 19,000,000 bytes by default;
  • time: max_batch_delay has passed since its first row, 100 ms by default, so the rows of a producer that went quiet still go out;
  • rows: it holds max_batch_rows rows, if you set it, no limit by default;
  • flush() or finish() is called.

max_request_bytes can be lowered, never raised. BigQuery’s limit is 20 MiB per request, and a request over it is not rejected on its own: it ends the whole connection. So the library counts every request’s exact size before sending it. A single row too large for a request alone fails with SerializeError of kind RowTooLarge and is never sent.

use bigquery::*;
use serde::Serialize;
use std::time::Duration;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[derive(Serialize)]
struct Order {
    id: i64,
}
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let (writer, _responses) = db
    .create_streaming_writer_with_options::<Order>(
        SHOP.table(ORDERS),
        BigQueryStreamingWriteOptions::new()
            .with_max_batch_delay(Duration::from_millis(500))
            .with_max_batch_rows(10_000),
    )
    .await?;
let _ = writer;
Ok(())
}

Backpressure

The writer does not wait for one batch to be acknowledged before sending the next, it pipelines them. Two limits bound what is sent and not acknowledged yet:

  • max_inflight_requests: 8 by default;
  • max_inflight_bytes: 64 MiB by default.

When either is reached, write().await waits until BigQuery acknowledges a batch. So a producer faster than the network slows down to its speed instead of buffering rows in memory.

Flush and finish

  • flush() sends the open batch and waits until every batch written so far has an outcome. It fails only when the writer itself failed for good; a failed batch is reported on the response stream and in the summary.
  • finish() flushes, waits for every acknowledgement and closes the writer, then returns the BigQueryWriteSummary. On the default stream that is all; a committed stream is finalized, a buffered one is flushed up to its last row and finalized, and a pending one is finalized and committed.

Write modes

The mode decides which write stream the rows go through, and with it the delivery guarantee:

ModeInsertWriter optionRows visibleGuarantee
Default.objects(..)BigQueryWriteMode::Defaultas soon as each batch is acknowledgedat least once: a batch resent after a reconnect can be stored twice
Exactly once.exactly_once()BigQueryWriteMode::Committedas soon as each batch is acknowledgedexactly once: every request has an offset, and BigQuery recognises a resent one
Atomic.atomic()BigQueryWriteMode::Pendingall together, at the commitall rows or none
Buffered.buffered()BigQueryWriteMode::Bufferedup to the offset you flushexactly once, as committed, see buffered streams
CDC.changes(..) or .upsert()default streamafter BigQuery applies the changesupserts and deletes by primary key, see change data capture
use bigquery::*;
use serde::Serialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[derive(Serialize)]
struct Order {
    id: i64,
}
async fn example(db: BigQueryDb, orders: Vec<Order>) -> BigQueryResult<()> {
// Every row exactly once, even when a request is resent
db.fluent()
    .insert()
    .into(SHOP.table(ORDERS))
    .objects(&orders)
    .exactly_once()
    .execute()
    .await?;

// Every row becomes visible at one commit, or none does
let summary = db
    .fluent()
    .insert()
    .into(SHOP.table(ORDERS))
    .objects(&orders)
    .atomic()
    .execute()
    .await?;
println!("committed at {:?}", summary.commit_time);
Ok(())
}

.options(..) replaces the options and the mode with them, so call it before .exactly_once() or .atomic().

The atomic mode is what a “batch write” in the transactional sense is. If any batch fails, finish() returns WriteStreamError with the code NOT_COMMITTED and the table gets nothing.

Several pending writers on one table can commit together. finalize() instead of finish() leaves the commit to db.commit_write_streams(..):

use bigquery::*;
use serde::Serialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[derive(Serialize)]
struct Order {
    id: i64,
}
async fn example(db: BigQueryDb, first: Vec<Order>, second: Vec<Order>) -> BigQueryResult<()> {
let options = BigQueryStreamingWriteOptions::new().with_mode(BigQueryWriteMode::Pending);

let (mut first_writer, _) = db
    .create_streaming_writer_with_options::<Order>(SHOP.table(ORDERS), options.clone())
    .await?;
let (mut second_writer, _) = db
    .create_streaming_writer_with_options::<Order>(SHOP.table(ORDERS), options)
    .await?;

first_writer.write_all(&first).await?;
second_writer.write_all(&second).await?;

let streams = vec![first_writer.finalize().await?, second_writer.finalize().await?];
let commit_time = db.commit_write_streams(streams).await?;
println!("committed at {commit_time}");
Ok(())
}

The commit is retried like any other call. When a retry finds every stream already committed at one time, commit_write_streams returns that commit time. Any other stream error, an already committed stream on the first attempt included, is a WriteStreamError with BigQuery’s code.

Errors

  • A row that does not fit the table’s schema fails a streaming writer’s write() with SerializeError, naming the row by its index in write order. Nothing of it is sent, and the writer goes on.
  • insert().execute() stops at the first row that does not serialize and returns SerializeError. On the default and committed streams the earlier rows may or may not be written: the requests already sent stay written, and the rows still in the open or queued batches are dropped. With the default batch size a short insert usually has all of its rows in one open batch, so nothing is written. A pending stream (.atomic()) commits nothing.
  • A batch BigQuery rejects is RowErrors with every row BigQuery named, by its index in write order, so you can find and resend them. None of the batch’s rows are written, and the other batches are not affected.
  • A retryable connection failure makes the writer reconnect and resend every batch not acknowledged yet, up to the client’s max_retries. In the default mode that is where a row can be stored twice.
  • Once the writer failed for good, every write() returns that error.

Schema changes

The writer follows the table’s schema while it runs. A column added to the table is picked up for the next batches, from BigQuery’s own notice or when a row has a field the writer did not know.

Be aware not to drop a column while a writer still sends it. BigQuery keeps accepting the values for about 9 seconds and drops them silently, before it starts rejecting the rows. Stop every writer from sending the column first, then drop it.

A field a row leaves out is stored as NULL by default. missing_value with BigQueryMissingValue::DefaultValue asks BigQuery to use the column’s default value expression instead. The library sends it as BigQuery’s default_missing_value_interpretation and has no live test for it yet.

Caveats

  • A large .atomic() load stays invisible, and holds its rows in a pending stream, until it is committed. It suits a single load with a clear end. For an endless stream use the default or the exactly once mode.
  • Rows from two writers on one table can interleave, nothing orders rows between writers.
  • Only a CDC sequence number orders changes to the same key, see change data capture.
  • Every insert pays the round trip to open its write stream; share a streaming writer for many small writes.

The benchmarks have Storage Write runs of 1M rows, from structures and from Arrow record batches, against the official crate.

Buffered streams

A buffered stream keeps the rows it acknowledged invisible until you flush them. A flush makes every row up to an offset readable, and a later flush moves that offset further. It suits a producer that has to decide when its rows count, for example only after it saved its own checkpoint:

  • flush_rows() sends the open batch, waits for every acknowledgement and flushes the stream up to its last row. It returns the flushed offset, or None while the stream has no rows;
  • flush_rows_to(offset) flushes up to and including offset, the stream offset of a row. The last row of an acknowledged batch is at offset + row_count - 1 of its BigQueryWriteResponse. It does not wait for the batches in flight, and BigQuery checks the offset.
use bigquery::*;
use serde::Serialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[derive(Serialize)]
struct Order {
    id: i64,
}
async fn save_checkpoint(offset: i64) {}
async fn example(db: BigQueryDb, orders: Vec<Order>) -> BigQueryResult<()> {
let (mut writer, _responses) = db
    .create_streaming_writer_with_options::<Order>(
        SHOP.table(ORDERS),
        BigQueryStreamingWriteOptions::new().with_mode(BigQueryWriteMode::Buffered),
    )
    .await?;

for chunk in orders.chunks(1_000) {
    writer.write_all(chunk).await?;
    // The rows of this chunk become readable here, not before
    if let Some(offset) = writer.flush_rows().await? {
        save_checkpoint(offset).await;
    }
}

let summary = writer.finish().await?;
println!("{} rows written", summary.rows_written);
Ok(())
}

finish() flushes the rest before it finalizes the stream, so every row you wrote becomes readable. Finalizing alone does not flush, and only flushed rows are readable. If you need the unflushed rows to stay unread, drop the writer instead of finishing it; it logs a warning and the stream is never finalized.

After the writer failed for good, flush_rows_to(..) still works, since it does not need the connection. Flush to the last acknowledged row to make every acknowledged row readable.

.buffered() on an insert writes through a buffered stream and flushes once at the end. Unlike .atomic(), a failed batch does not hold back the others.

Google itself calls the buffered type an advanced one, for the Apache Beam connector mostly. If you only need a few rows to appear together, the exactly once mode with all of them in one batch does that too.

Arrow record batches

The library writes Arrow record batches as they are, the way reads return them with .record_batches(). Every write mode works the same as for structures:

  • .record_batches(..) on an insert, with .exactly_once(), .atomic(), .buffered() and .options(..);
  • db.create_record_batch_writer(..) for a long-running producer, with write_batch(..) and the same flush(), finish(), finalize(), flush_rows() and flush_rows_to(..).
use bigquery::*;
use bigquery::arrow_array::{Int64Array, RecordBatch, StringArray};
use bigquery::arrow_schema::{DataType, Field, Schema};
use std::sync::Arc;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
async fn example(db: BigQueryDb) -> Result<(), Box<dyn std::error::Error>> {
let schema = Arc::new(Schema::new(vec![
    Field::new("id", DataType::Int64, false),
    Field::new("customer", DataType::Utf8, true),
]));
let batch = RecordBatch::try_new(
    schema,
    vec![
        Arc::new(Int64Array::from(vec![1, 2])),
        Arc::new(StringArray::from(vec!["customer-1", "customer-2"])),
    ],
)?;

// An insert, here exactly once
db.fluent()
    .insert()
    .into(SHOP.table(ORDERS))
    .record_batches([batch.clone()])
    .exactly_once()
    .execute()
    .await?;

// A streaming writer, here buffered
let (mut writer, _responses) = db
    .create_record_batch_writer_with_options(
        SHOP.table(ORDERS),
        BigQueryStreamingWriteOptions::new().with_mode(BigQueryWriteMode::Buffered),
    )
    .await?;
writer.write_batch(&batch).await?;
writer.flush_rows().await?;
let summary = writer.finish().await?;
println!("{} rows written", summary.rows_written);
Ok(())
}

Each batch goes with its own schema, and the library does not check it against the table, since BigQuery does. A batch that does not fit the table fails as a batch, on the response stream and in the summary. Google lists how Arrow types map to BigQuery types in supported data types. For example, a TIMESTAMP column takes Timestamp(Microsecond, "UTC"), a DATETIME column the same without a time zone, and a NUMERIC column Decimal128. BigQuery refuses dictionary-encoded columns, so cast them to their value type first, with arrow::compute::cast for instance.

The batching is a bit different from structures:

  • a record batch is sent as it is written, it never shares a request with another one, so max_batch_delay does not apply. Many tiny batches are as many requests;
  • a batch larger than max_request_bytes is sent as slices of it, one request each. The library serializes every slice to measure it, so no request goes over the limit. A single row too large for a request alone fails with SerializeError of kind RowTooLarge;
  • max_batch_rows caps the rows of every slice;
  • a batch with another schema than the one before it opens a new connection, since BigQuery reads an Arrow schema from the first request of a connection only.

CDC writes take structures only, since the change columns have to go as protobuf.

Full example available here.

Updating and deleting

BigQuery tables are made for appending rows, but the library supports updates and deletes by primary key too:

  • db.fluent().update() writes whole rows: each replaces the row with its primary key, or is inserted if there is none;
  • db.fluent().delete() deletes the rows with the given primary keys.

Both are written through BigQuery change data capture (CDC), as upserts and deletes on the table’s default write stream. There is no UPDATE or MERGE statement to run, BigQuery applies the changes itself.

CDC or DML queries

BigQuery has its own UPDATE, DELETE and MERGE statements too, and the library runs them as queries, see DML and DDL. The two do not mix on one table: BigQuery does not support mutating DML on a table while CDC changes are streamed to it. So the choice is one for the whole table.

Use update() and delete() when:

  • you change rows by primary key, and have the whole row for an update;
  • changes come often, one by one or as a steady stream, like mirroring an OLTP database. A CDC change is a Storage Write API write, so it does not wait in the DML queue below;
  • a change can arrive late or twice, sequence numbers put them in order;
  • queries can read data up to max_staleness old, or pay for merging the changes at query time.

Use a DML query when:

  • the rows to change are given by a condition, for example DELETE FROM shop.orders WHERE created < @cutoff;
  • you change some columns and keep the rest, or compute the new value from the old one or from another table: SET total = total * 2, a MERGE from a staging table, etc.;
  • the table has no primary key, or its key is not unique;
  • a change must apply to all its rows or none of them. Each DML statement is an implicit transaction, committed when it succeeds. CDC writes go through the default stream, which has no atomic commit, see the limits of CDC tables;
  • you need to know how many rows matched. A DML statement returns num_dml_affected_rows, a CDC write returns rows_written, the rows sent, whether a row with the key existed or not;
  • the change is a rare bulk rewrite or cleanup. Google’s best practices describe UPDATE and DELETE as “oriented towards periodic rewrites of your data”.

Be aware not to run DML for every single change. From BigQuery’s DML documentation:

  • up to 2 mutating statements (UPDATE, DELETE, MERGE) run on a table at once, up to 20 more wait as PENDING, and the next one fails with Too many DML statements outstanding against table;
  • two statements running at once that change the same partition conflict, and BigQuery reruns the failed one up to three times.

On demand, an UPDATE or DELETE is billed for the bytes the statement reads plus the size of the table before the change, or of the partitions it changes for a partitioned table, as the on-demand query size calculation describes. Every statement pays that size, even one that changes one row. CDC has its own costs, the background applying and the merge at query time, see the limits of CDC tables.

On a table that takes CDC changes, a change by condition goes through CDC as well: query the keys that match, then delete them by key:

use bigquery::*;
use serde::Deserialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[derive(Deserialize)]
struct OrderId {
    id: i64,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let cancelled: Vec<OrderId> = db
    .fluent()
    .query("SELECT id FROM shop.orders WHERE status = @status")
    .param("status", "cancelled")
    .obj::<OrderId>()
    .query()
    .await?;

db.fluent()
    .delete()
    .from(SHOP.table(ORDERS))
    .keys(cancelled.iter().map(|order| order.id))
    .execute()
    .await?;
Ok(())
}

The table

The table needs a primary key, declared with schema():

use bigquery::*;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
struct Order {
    id: i64,
    customer: String,
    status: String,
    note: Option<String>,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
db.fluent()
    .schema()
    .table(SHOP.table(ORDERS))
    .columns(|columns| {
        columns.fields([
            columns.field(path!(Order::id)).int64().required(),
            columns.field(path!(Order::customer)).string(),
            columns.field(path!(Order::status)).string(),
            columns.field(path!(Order::note)).string(),
        ])
    })
    .primary_key([path!(Order::id)])
    .sync()
    .await?;
Ok(())
}

BigQuery never enforces the key, it only uses it to find the row a change is for. So it is up to you that the key is unique.

Updates

use bigquery::*;
use serde::Serialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[derive(Serialize)]
struct Order {
    id: i64,
    customer: String,
    status: String,
    note: Option<String>,
}

async fn example(db: BigQueryDb, orders: Vec<Order>) -> BigQueryResult<()> {
let order = Order {
    id: 42,
    customer: "Ada".to_string(),
    status: "shipped".to_string(),
    note: None,
};
db.fluent()
    .update()
    .in_table(SHOP.table(ORDERS))
    .object(&order)
    .execute()
    .await?;

// Many rows through one writer
db.fluent()
    .update()
    .in_table(SHOP.table(ORDERS))
    .objects(&orders)
    .execute()
    .await?;
Ok(())
}

An update replaces the whole row. There is no field mask, so a column you leave out or set to None becomes NULL. To change one column, write the whole row with the new value. Be aware that BigQuery does not support UPDATE, DELETE or MERGE statements on a table while CDC changes are streamed to it, so a DML query is not a way around this.

Without sequence numbers, of two rows with one key the row BigQuery ingested last wins, see ordering changes.

Deletes

A delete needs only the primary key. .key(..) takes the key value itself: a plain value for a key of one column, a tuple in the key’s column order for a key of several:

use bigquery::*;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
const ORDER_LINES: BigQueryTableId = BigQueryTableId::from_static("order_lines");
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
db.fluent()
    .delete()
    .from(SHOP.table(ORDERS))
    .key(42)
    .execute()
    .await?;

// The primary key of order_lines is (order_id, line)
db.fluent()
    .delete()
    .from(SHOP.table(ORDER_LINES))
    .keys([(42, "line-1"), (42, "line-2")])
    .execute()
    .await?;
Ok(())
}

The library reads the key’s columns from the table’s metadata, one GetTable call per execute(), and writes rows with only those columns. A table without a primary key, or a tuple with another number of values than the key has columns, fails with InvalidParametersError before anything is written. A value for a key of one column is written as it is, so a Vec<u8> works for a BYTES key, and a value of the wrong type fails at the write with SerializeError.

A row works as well, .object(..) and .objects(..) take a struct of just the key, or the whole row:

use bigquery::*;
use serde::Serialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[derive(Serialize)]
struct OrderKey {
    id: i64,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
db.fluent()
    .delete()
    .from(SHOP.table(ORDERS))
    .object(&OrderKey { id: 42 })
    .execute()
    .await?;

db.fluent()
    .delete()
    .from(SHOP.table(ORDERS))
    .objects([OrderKey { id: 1 }, OrderKey { id: 2 }])
    .execute()
    .await?;
Ok(())
}

The columns the key struct or .key(..) leaves out must be NULLABLE. A row without a REQUIRED column does not serialize, so for a table with other REQUIRED columns pass the whole row instead.

Ordering changes

Without sequence numbers, of the changes to one key the one BigQuery ingested last wins. That is fine for one producer at a time. If a change can arrive late or twice, give it a sequence number, the highest one wins:

use bigquery::*;
use serde::Serialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[derive(Serialize)]
struct Order {
    id: i64,
    customer: String,
    status: String,
    note: Option<String>,
}
async fn example(db: BigQueryDb, order: Order, orders: Vec<Order>) -> BigQueryResult<()> {
db.fluent()
    .update()
    .in_table(SHOP.table(ORDERS))
    .object(&order)
    .sequence_number(BigQueryChangeSequenceNumber::from(1041))
    .execute()
    .await?;

// One number for the rows of one source transaction
db.fluent()
    .update()
    .in_table(SHOP.table(ORDERS))
    .objects(&orders)
    .sequence_number(BigQueryChangeSequenceNumber::from(1042))
    .execute()
    .await?;

db.fluent()
    .delete()
    .from(SHOP.table(ORDERS))
    .key(42)
    .sequence_number(BigQueryChangeSequenceNumber::from(1043))
    .execute()
    .await?;
Ok(())
}

.sequence_number(..) gives the same number to every row of the call. Between changes to one key with the same number, the one BigQuery ingested last wins. For changes each with its own number, use .changes(..) on an insert, see change data capture.

Be aware that sequence numbers are a choice for the whole table: once a table takes changes with sequence numbers, send one with every change to it. Mixing changes with and without them gives an unpredictable order.

When the changes show up

BigQuery applies the changes in the background, and the table’s max_staleness option says how old the applied data may be:

  • without max_staleness, a query merges the changes not applied yet at query time, so it sees every change as soon as execute() returns, and pays for that merge;
  • with max_staleness, say 10 minutes, a query may read data up to 10 minutes old, so a read right after an update may not show it yet.

More on max_staleness, its costs and the limits of CDC tables in change data capture.

Each execute() opens a CDC writer and finishes it, which costs a round trip or so. For a steady stream of changes keep one writer open with db.create_cdc_writer(..) instead.

Full example available here.

Change data capture

BigQuery tables are made for appending rows. Change data capture (CDC) is BigQuery’s way to keep a table in sync with a changing source instead: each row you write is a change to the row with the same primary key, an upsert or a delete, and BigQuery applies the changes for you. There is no MERGE or UPDATE statement to run and no staging table to clean up.

The typical use is mirroring an OLTP database into BigQuery as a stream. You read the changes of a PostgreSQL or MySQL table from its log, and write each one as it comes. The BigQuery table then looks like the source table, with a delay you choose.

This chapter is about the CDC API itself. To update or delete a few rows by primary key, db.fluent().update() and db.fluent().delete() are simpler, see updating and deleting. When to use CDC and when to run UPDATE and DELETE statements instead, see CDC or DML queries.

How it works

A CDC table needs a primary key. BigQuery’s keys are NOT ENFORCED: BigQuery never checks them, so it is up to you that the key is unique in the source. A key can have up to 16 columns.

Every row written through CDC carries two pseudo-columns besides the table’s own columns:

  • _CHANGE_TYPE: UPSERT inserts the row, or replaces the row with the same key; DELETE deletes the row with the same key, and only its key columns matter;
  • _CHANGE_SEQUENCE_NUMBER, optional: orders the changes to one key.

Without a sequence number BigQuery orders the changes to one key by the time it received them, the latest wins. That is fine for one producer writing in order, but rows can be resent after a reconnect, and two producers can race. With a sequence number the highest one wins, whenever it arrived, so an old change that arrives late or twice does not overwrite a newer one.

A sequence number is up to four sections of hexadecimal digits separated by /, each up to 16 digits, from 0 to FFFFFFFFFFFFFFFF/FFFFFFFFFFFFFFFF/FFFFFFFFFFFFFFFF/FFFFFFFFFFFFFFFF. BigQuery compares the sections as numbers, from left to right. A PostgreSQL LSN such as 16/B374D848 is already in this form. Between two changes with the same number, the one BigQuery ingested last wins. Sequence numbers are a choice for the whole table: once a table gets changes with sequence numbers, send one with every change to it, mixing changes with and without them gives an unpredictable order.

BigQuery does not rewrite the table on every change. It keeps the recent changes beside the table and applies them in the background. The table’s max_staleness option says how old the applied data may be:

  • without max_staleness, a query merges the changes not applied yet at query time, so it always sees the latest data and pays for that merge;
  • with max_staleness = INTERVAL 10 MINUTE, BigQuery applies the changes at least once every 10 minutes with background jobs, and a query reads the applied table, which can be up to 10 minutes old. If the background jobs fall behind the interval, queries merge at query time again.

Either way a reader sees the merged result, never the raw changes. The pseudo-columns cannot be queried.

Creating the table

Declare the primary key with schema(), and set max_staleness with a DDL statement:

use bigquery::*;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const CUSTOMERS: BigQueryTableId = BigQueryTableId::from_static("customers");
struct Customer {
    id: i64,
    name: String,
    city: Option<String>,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
db.fluent()
    .schema()
    .table(SHOP.table(CUSTOMERS))
    .columns(|columns| {
        columns.fields([
            columns.field(path!(Customer::id)).int64().required(),
            columns.field(path!(Customer::name)).string(),
            columns.field(path!(Customer::city)).string(),
        ])
    })
    .primary_key([path!(Customer::id)])
    .cluster_by([path!(Customer::id)])
    .sync()
    .await?;

db.fluent()
    .query("ALTER TABLE shop.customers SET OPTIONS (max_staleness = INTERVAL 10 MINUTE)")
    .execute()
    .await?;
Ok(())
}

Clustering by the key is what Google’s own CDC example does; it is not required.

Writing changes

.changes(..) on an insert takes BigQueryChanges, each with a BigQueryChangeType and an optional BigQueryChangeSequenceNumber:

use bigquery::*;
use serde::Serialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const CUSTOMERS: BigQueryTableId = BigQueryTableId::from_static("customers");
#[derive(Serialize)]
struct Customer {
    id: i64,
    name: String,
    city: Option<String>,
}

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let changes = vec![
    BigQueryChange {
        change_type: BigQueryChangeType::Upsert,
        sequence_number: Some(BigQueryChangeSequenceNumber::from(1)),
        row: Customer {
            id: 1,
            name: "Ada".to_string(),
            city: Some("Stockholm".to_string()),
        },
    },
    BigQueryChange {
        change_type: BigQueryChangeType::Delete,
        sequence_number: Some("16/B374D848".parse()?),
        row: Customer {
            id: 2,
            name: String::new(),
            city: None,
        },
    },
];

db.fluent()
    .insert()
    .into(SHOP.table(CUSTOMERS))
    .changes(changes)
    .execute()
    .await?;

// Or plain rows as upserts, for a table that does not use sequence numbers
let customer = Customer {
    id: 3,
    name: "Grace".to_string(),
    city: None,
};
db.fluent()
    .insert()
    .into(SHOP.table(CUSTOMERS))
    .object(&customer)
    .upsert()
    .execute()
    .await?;
Ok(())
}

BigQueryChangeSequenceNumber::from(u64) writes the number in hex, as one section. Parsing a string takes the text as it is and BigQuery checks its form, so a source that has its own sequence text, like the LSN above, can pass it through.

For a long-running stream of changes, db.create_cdc_writer(..) opens a CDC writer, the same as the streaming writer with its batching, backpressure and response stream:

use bigquery::*;
use serde::Serialize;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const CUSTOMERS: BigQueryTableId = BigQueryTableId::from_static("customers");
#[derive(Serialize)]
struct Customer {
    id: i64,
    name: String,
    city: Option<String>,
}
async fn example(db: BigQueryDb, source_changes: Vec<BigQueryChange<Customer>>) -> BigQueryResult<()> {
let (mut writer, _responses) = db
    .create_cdc_writer::<Customer>(
        SHOP.table(CUSTOMERS),
        BigQueryStreamingWriteOptions::new(),
    )
    .await?;

for change in &source_changes {
    writer.write_change(change).await?;
}

let summary = writer.finish().await?;
println!("{} changes written", summary.rows_written);
Ok(())
}

writer.upsert(&row) and writer.delete(&row) write a change without a sequence number.

Full example available here.

Why protobuf

BigQuery takes CDC changes only as protobuf rows on the default write stream, Apache Arrow rows are not supported for CDC. This is the main reason the library writes every row as protobuf: one encoder serves plain inserts and CDC alike. Protobuf requests were also smaller than Arrow for the same rows in the benchmarks, 18% fewer bytes for 1M rows.

Limits

From BigQuery’s side, as its CDC limitations list them:

  • CDC goes only through the default stream, so .exactly_once() and .atomic() cannot be combined with it; the library refuses them with InvalidParametersError before any request;
  • the primary key is not enforced and has at most 16 columns;
  • the table has at most 2,000 top-level columns;
  • mutating DML (UPDATE, DELETE, MERGE), wildcard table queries and search indexes are not supported on the table;
  • while queries merge at query time because max_staleness is too low or not set, the table cannot be copied, cloned or snapshotted, and cannot be read through the Storage Read API, so table reads need a max_staleness the background jobs keep up with;
  • a query that merges at query time scans the whole table, whatever its partition filter;
  • exports do not include changes not applied yet;
  • the applying is BigQuery compute and is billed as such, on demand unless you have a BACKGROUND reservation, which Standard edition does not have.

From the library’s side:

  • a CDC writer always writes _CHANGE_TYPE, so write plain inserts to the same table with a separate writer. BigQuery’s CDC documentation says rows with and without a change type on one connection are not supported;
  • a delete serializes a whole row, but only its key columns matter. A struct of just the key works when every column it leaves out is NULLABLE;
  • delivery is at least once, as on every default stream write. With sequence numbers a change sent twice is harmless.

Schema management

Tables are usually created with DDL in the console or a migration script, and changed by hand when the code needs a new column. The library supports declaring a table’s schema and settings in Rust instead, next to the structs that read and write it, and making the table match with one explicit call.

It has three parts: a declaration, a read-only .plan() and a .sync() that applies it.

Declaring a table

Everything starts from db.fluent().schema().table(..), then a chain of declarations ending in .plan() or .sync():

use bigquery::*;

struct Address {
    city: String,
}

struct Order {
    id: i64,
    customer: String,
    total: String,
    shipping: Address,
    tags: Vec<String>,
    placed_at: jiff::Timestamp,
}

const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let report = db
    .fluent()
    .schema()
    .table(SHOP.table(ORDERS))
    .columns(|columns| {
        columns.fields([
            columns.field(path!(Order::id)).int64().required(),
            columns.field(path!(Order::customer)).string_with_max_length(64),
            columns.field(path!(Order::total)).numeric_with(10, 2).default_value("0"),
            columns.field(path!(Order::shipping)).record(|address| {
                address.fields([address.field(path!(Address::city)).string()])
            }),
            columns.field(path!(Order::tags)).string().repeated(),
            columns.field(path!(Order::placed_at))
                .timestamp()
                .description("When the customer placed the order"),
        ])
    })
    .primary_key([path!(Order::id)])
    .partition_by_day(path!(Order::placed_at))
    .cluster_by([path!(Order::customer)])
    .description("Orders")
    .labels([("team", "shop")])
    .sync()
    .await?;
println!("{report}");
Ok(())
}

A missing table is created with one InsertTable. An existing one is changed in place where BigQuery allows it.

.columns() declares the columns in table order. Each column needs one type:

  • int64(), float64(), bool();
  • numeric(), numeric_with(precision, scale), bignumeric(), bignumeric_with(..);
  • string(), string_with_max_length(n), bytes(), bytes_with_max_length(n);
  • date(), time(), datetime(), timestamp(), interval(), range(element);
  • geography(), json();
  • record(|address| ..) for a STRUCT, with its fields declared the same way;
  • of_type(BigQueryFieldType) for any of the above as a value.

A column is NULLABLE until required() or repeated() says otherwise. description(..) and default_value(..) are optional, and renamed_from(..) is described below.

The table settings are primary_key(..) (always NOT ENFORCED, as every BigQuery key), partition_by_hour/day/month/year(column) or partition_by(BigQueryPartitioning), partition_expiration(Duration), cluster_by(..), description(..), labels(..) and expiration(BigQueryInstant).

Be aware a default value is trusted SQL. It is sent to BigQuery as it is, and written into generated DDL as one parenthesized expression, so it must never carry text from your users. A string default is written as its own quoted literal, default_value("'none'"). Descriptions, labels and option values are always sent as values or escaped literals.

The library checks only what it needs itself to build the requests: a column without a type, an empty column name or one with . or a control character, two columns that differ only in case, renamed_from on a nested field, a partitioning column that is not declared, cluster_by or primary_key with no columns, partition_expiration without partitioning and snapshot_first() without a recreate opt-in. These fail .plan() and .sync() with InvalidParametersError before any request. Lengths, naming rules and label rules are left to BigQuery.

Inferring the columns from a structure

The library can also infer the columns from the structure you read and write the table with, so the declaration does not repeat every field. from_type::<T>() goes through the structure’s Deserialize, the same serde path the library’s codecs use, and gives one column per field, in declaration order:

use bigquery::*;
use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
struct Address {
    city: String,
    street: Option<String>,
}

#[derive(Debug, Serialize, Deserialize)]
struct Order {
    id: i64,
    customer: Option<String>,
    total: String,
    shipping: Option<Address>,
    tags: Vec<String>,
    placed_at: jiff::Timestamp,
}

const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let report = db
    .fluent()
    .schema()
    .table(SHOP.table(ORDERS))
    .columns(|columns| {
        columns
            .from_type::<Order>()
            .with(path!(Order::total), |total| total.numeric_with(10, 2))
            .with(path!(Order::shipping~city), |city| city.string_with_max_length(64))
            .with(path!(Order::placed_at), |placed_at| {
                placed_at.description("When the customer placed the order")
            })
    })
    .primary_key([path!(Order::id)])
    .partition_by_day(path!(Order::placed_at))
    .sync()
    .await?;
println!("{report}");
Ok(())
}

Each field maps by the type mapping:

  • Option<T> is NULLABLE, a bare T is REQUIRED, Vec<T> and Option<Vec<T>> are REPEATED;
  • integers are INT64, floats FLOAT64, bool BOOL, Vec<u8> BYTES;
  • jiff::Timestamp is TIMESTAMP and jiff::civil::Time is TIME;
  • the temporal wrappers, BigQueryDecimal<T> (NUMERIC), BigQueryJson<T>, BigQueryInterval and BigQueryRange<T> are their own types;
  • String, unit enums and any other type that reads from a string are STRING;
  • a structure is a RECORD, with its fields inferred the same way.

Serde attributes are followed: rename and rename_all name the column, skip leaves the field out, an alias adds no column, and default does not change the mode.

An aliased field infers as it would without the alias. Inference tells an alias from the field’s own name by building a sample value of the field, and a type that reads only its own text form, such as uuid::Uuid, has no sample value. An alias on a REQUIRED field of such a type is told apart only when that field is the one field of the structure without a sample value, and is not #[serde(default)]. Otherwise each name that cannot be told apart is refused at its own path with UnresolvedAlias, and the structure is declared with fields([..]).

Be aware of the modes. An inferred bare T is REQUIRED, while a column declared by hand with field(..) is NULLABLE until required(). So moving customer: String from columns.field("customer").string() to from_type makes the column REQUIRED, and BigQuery cannot make an existing NULLABLE column REQUIRED in place. Use Option<String> in the structure, or .with(path!(Order::customer), |customer| customer.nullable()).

.with(path, |column| ..) changes one inferred column. The closure gets the column as it was inferred:

  • description(..), default_value(..) and renamed_from(..) keep its type and mode;
  • a type method such as numeric() replaces the type and keeps the mode;
  • nullable(), required() and repeated() change the mode;
  • a dotted path reaches a field of a RECORD, path!(Order::shipping~city) steps through the Option;
  • on a RECORD column a type method such as json() drops its fields, and record(..) replaces them.

Some types do not tell the column type, and need .with:

  • NUMERIC, BIGNUMERIC, GEOGRAPHY and JSON held in a String are inferred as STRING;
  • a plain jiff::civil::Date and jiff::civil::DateTime read the same text forms, so the library cannot tell DATE from DATETIME and leaves the type to you. The BigQueryDate and BigQueryDateTime wrappers, and serialize_as_date/serialize_as_datetime, are inferred;
  • serde_json::Value, maps, enums with data and arrays of arrays get no type.

A column without an inferred type, or a .with path that names no column, fails .plan() and .sync() with SchemaInferenceError before any request, and the error carries the column path. A structure with #[serde(flatten)] does not list its fields at all, so declare its table with fields([..]).

Full example available here.

plan() versus sync()

.plan() is read-only. It sends one GetTable (and ListRowAccessPolicies when a recreate is planned) and reports what .sync() would do, writing nothing.

.sync() plans the same way and then applies the plan.

Both return a value with a Display impl, so it can be printed or logged directly:

use bigquery::*;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let plan = db
    .fluent()
    .schema()
    .table(SHOP.table(ORDERS))
    .columns(|columns| {
        columns.fields([
            columns.field("id").int64().required(),
            columns.field("customer").string(),
            columns.field("note").string(),
        ])
    })
    .plan()
    .await?;

if !plan.is_empty() {
    println!("{plan}");
}
Ok(())
}

BigQueryTablePlan has:

  • create: the table to create, when it does not exist;
  • changes: the in-place changes, in the order .sync() sends them;
  • withheld: changes left out until the declaration opts in to them, each with its BigQueryWithheldReason (PruneUndeclared or AllowWidening);
  • recreate: the recreate .sync() runs instead of changes, when a change is impossible in place and a recreate opt-in applies;
  • impossible and refusal: the changes that are impossible in place, and why .sync() refuses them.

BigQueryTableSyncReport from .sync() has the same shape for what was done: created, applied, withheld, recreated, snapshot and dropped_data, the last one listing every column or row a sync deleted.

One statement owns one table

A chain names exactly one table, and that table is the unit of ownership: .prune_undeclared() never reaches anything outside it. Several tables need several statements, for example one per table in a startup function.

What a declaration leaves out

What the chain declares is what .sync() makes the table hold. What it leaves undeclared is kept as the table has it:

  • an undeclared column, label, clustering, primary key or partitioning is kept and listed under withheld with PruneUndeclared;
  • an undeclared description, default value or expiration is kept and not reported.

So a declaration can start small, with only the columns your code needs, and the rest of the table stays untouched.

Before comparing, both sides are normalised the way BigQuery itself compares them. GetTable returns the legacy type names, so INTEGER equals INT64, FLOAT equals FLOAT64, RECORD equals STRUCT, etc. A column created by DDL comes back with an empty mode, which equals NULLABLE. Type parameters are part of the type, so STRING(10) and STRING(20) differ, and NUMERIC(10) equals NUMERIC(10, 0). Column names compare ignoring case.

Change classes

Every difference between the declaration and the table is one BigQuerySchemaChange, and BigQuery applies each kind in its own way:

ChangeApplied byNeeds
add a NULLABLE or REPEATED column, a field inside an existing RECORD, a RECORD columnPatchTable
add a column with a default valuePatchTable, then a second PatchTable for the default
relax REQUIRED to NULLABLEPatchTable
set a column default, a column or table descriptionPatchTable
add or change a label, the expiration, the partition expirationPatchTable
add or change clustering, add or change the primary keyPatchTable
remove the primary keyPatchTableprune_undeclared()
remove a label or the clusteringUpdateTableprune_undeclared()
rename a columnDDL RENAME COLUMNrenamed_from(..)
widen a column typeDDL ALTER COLUMN SET DATA TYPEallow_widening()
drop a column, with every value in itDDL DROP COLUMNprune_undeclared()
anything elseimpossible in placea recreate opt-in

BigQuery cannot add a column with a default in one step, so the column is added first and the default set right after. Rows already in the table stay NULL in that column; only new rows get the default. The same holds for a default set on an existing column.

Impossible in place means BigQuery has no call or statement that does it without recreating the table:

  • a type change that is not a widening, INT64 to FLOAT64 and every narrowing included;
  • a change between a RECORD and another type, or a widening of a nested field;
  • NULLABLE to REQUIRED, and any change to or from REPEATED;
  • a new REQUIRED column or nested field;
  • dropping a nested field;
  • adding or changing the partitioning, or removing it with prune_undeclared().

Without a recreate opt-in .sync() refuses such a plan and writes nothing, not even the changes that are possible in place. The error is SchemaChangeRefused, carrying the plan:

use bigquery::*;
use bigquery::errors::BigQueryError;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let synced = db
    .fluent()
    .schema()
    .table(SHOP.table(ORDERS))
    .columns(|columns| columns.fields([columns.field("id").string().required()]))
    .sync()
    .await;

match synced {
    Ok(report) => println!("{report}"),
    Err(BigQueryError::SchemaChangeRefused(refused)) => {
        for change in &refused.plan.impossible {
            eprintln!("impossible in place: {change}");
        }
    }
    Err(err) => return Err(err),
}
Ok(())
}

The order sync() writes changes in

.sync() reads the table once and writes in a fixed order:

  1. one PatchTable with every change that can go into a patch;
  2. a second PatchTable with the default values of the columns the first one added;
  3. one UpdateTable for removed labels and clustering, since a patch cannot remove them;
  4. DDL through a query, one statement at a time: renames, then widenings, then drops.

Patch and Update carry the etag of the table the sync read, as an if-match precondition. When the table changed in between, the write fails with DataConflictError and nothing after it is sent. Run the sync again: it reads the table as it is now. Changes sent before a failure stay applied, and the sync logs at warn what it applied so far.

BigQuery limits how often one table can be updated: about 5 DDL statements or 7 to 8 patches in a quick burst, then it starts rejecting them. So a sync sends its first five writes back to back and spaces the rest about 2 seconds apart, and the rate-limit errors are retried.

Rolling out a change

BigQuery applies a schema change at once, but running writers and readers see it later. During a rolling update old and new replicas run side by side, so a sync that removes a column before the old replicas stop is the same as removing it under them.

So run .sync() once per release, in two steps:

  • before the deploy, the sync without .prune_undeclared(): it adds columns, relaxes REQUIRED, sets descriptions, labels, clustering, etc., all of which keep running writers working;
  • after the rollout has finished, the same sync with .prune_undeclared(), which drops what the release no longer declares.
use bigquery::*;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
fn orders(db: &BigQueryDb) -> BigQueryTableSchemaBuilder<'_> {
    db.fluent()
        .schema()
        .table(SHOP.table(ORDERS))
        .columns(|columns| {
            columns.fields([
                columns.field("id").int64().required(),
                columns.field("customer").string(),
                columns.field("channel").string(),
            ])
        })
}

async fn example(db: BigQueryDb, rollout_finished: bool) -> BigQueryResult<()> {
let report = if rollout_finished {
    orders(&db).prune_undeclared().sync().await?
} else {
    orders(&db).sync().await?
};
println!("{report}");
Ok(())
}

From a Kubernetes Job or a CI/CD step, run the first one before the deploy step and the second one once the new version is serving.

What writers and readers see after a sync:

  • New columns. New writer connections accept rows with the new column about a second after the sync. Open connections get BigQuery’s updated_schema after about 7 seconds, and the library’s streaming writer encodes the next batches against it. Nothing is lost on the way.
  • Relaxed columns. An open connection can keep rejecting NULL for a relaxed column after the sync: for 2.1 s in one test and still after 5 minutes in another. A fresh connection accepted NULL after 0.4 s and 11.5 s. The library’s streaming writer reconnects by itself once it sees the relaxed column, so you only need to reconnect other writers before they send NULL.
  • Dropped columns. For about 9 seconds after a drop, an open writer’s values for the column are accepted and lost silently. Then the connection fails with Input schema has more fields than BigQuery schema. That is why a drop waits for the second step.
  • Readers. A read with selected fields that names a new or renamed column fails with The following selected fields do not exist in the table schema for about 30 seconds after the sync. A read without selected fields sees the change at once.

Renaming a column

renamed_from(old_name) declares that a column used to be called old_name:

use bigquery::*;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
db.fluent()
    .schema()
    .table(SHOP.table(ORDERS))
    .columns(|columns| {
        columns.fields([
            columns.field("id").int64().required(),
            columns.field("customer_name").string().renamed_from("customer"),
        ])
    })
    .sync()
    .await?;
Ok(())
}

When the table has customer and no customer_name, .sync() renames it with RENAME COLUMN and the values are kept. Once the table has customer_name, the declaration is a no-op, so it can stay in the code for a while.

For writers a rename is a drop of the old name: a writer still sending customer loses that value silently for a few seconds and then fails. renamed_from(..) is not held back by .prune_undeclared(), so the rename runs in whichever sync carries it. Be aware not to add renamed_from(..) to the declaration the first step runs: add it only to the one used after the rollout, once the writers have moved to the new name. Only top-level columns can be renamed.

Full example available here.

Widening a column

A declared type that is wider than the column’s is withheld with AllowWidening until .allow_widening() is set. Then .sync() widens it with ALTER COLUMN SET DATA TYPE, and the values are kept. The library takes these pairs as widenings:

  • INT64 to NUMERIC or BIGNUMERIC, and NUMERIC to BIGNUMERIC, without parameters;
  • NUMERIC(p, s) or BIGNUMERIC(p, s) to parameters that fit every value of the old ones, or to no parameters;
  • STRING(n) and BYTES(n) to a longer maximum length, or to no maximum.

INT64 to NUMERIC and a longer STRING(n) were checked against BigQuery; the other pairs follow GoogleSQL’s assignability rules. INT64 to FLOAT64 is not a widening for BigQuery, and every other type change is impossible in place. Only top-level columns can be widened.

Removing what you no longer declare

.prune_undeclared() makes .sync() also remove what the table has and the statement does not declare:

  • undeclared columns are dropped, with every value in them;
  • undeclared labels, clustering and the primary key are removed.

Be aware this is the one in-place change that loses data. Every dropped column is logged at warn and listed in the report’s dropped_data. An undeclared nested field or partitioning cannot be removed in place, so with .prune_undeclared() set they make the plan need a recreate.

Recreating a table

When a change is impossible in place, the only way is to drop the table and create it again, losing its rows. No data is migrated. A declaration opts in to that with one of:

  • recreate_if_empty(): recreate only when GetTable reports num_rows == 0, otherwise refuse with BigQueryRefusal::NotEmpty;
  • dangerously_recreate_with_data_loss(): recreate whatever the table holds.

The last of the two in a chain wins. snapshot_first() adds a CREATE SNAPSHOT TABLE before the recreate, as a cheap undo:

use bigquery::*;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const EVENTS: BigQueryTableId = BigQueryTableId::from_static("events");
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let report = db
    .fluent()
    .schema()
    .table(SHOP.table(EVENTS))
    .columns(|columns| {
        columns.fields([
            columns.field("id").string().required(),
            columns.field("happened_at").timestamp(),
        ])
    })
    .partition_by_month("happened_at")
    .dangerously_recreate_with_data_loss()
    .snapshot_first()
    .sync()
    .await?;

if let Some(snapshot) = &report.snapshot {
    println!("snapshot taken: {snapshot}");
}
Ok(())
}

The table is recreated as declared, plus whatever it has undeclared and not pruned. There are two ways, and the library picks one by what changes:

CREATE OR REPLACE TABLEDROP TABLE and CREATE TABLE in one script
used whenthe partitioning and clustering stay the samethe partitioning or clustering changes, which CREATE OR REPLACE cannot do
rowslostlost
columns, key, partitioning, clustering, description, labels, expirationsas declared or keptas declared or kept
table IAM bindingskeptlost
row access policieslostlost

The plan lists the row access policies that will be lost, and BigQueryRecreate names the method. The snapshot is named <table>_snapshot_<unix seconds> in the same dataset, and the library never deletes it.

These are the limits of a recreate, as measured against BigQuery, and the library does not hide any of them:

  • num_rows lags Storage Write. Rows written through committed or pending streams show in num_rows after 60 to 70 seconds, and rows on the default stream can take more than 5 minutes. So with recreate_if_empty() a table written seconds ago can read as empty and be recreated, its rows lost. The row and byte counts in the plan and in dropped_data have the same lag and can be lower than what is lost. Use recreate_if_empty() for tables that are new or being experimented on, with no writer running.
  • The snapshot misses the streaming buffer. A CREATE SNAPSHOT TABLE does not hold rows still in the streaming buffer, which is every row written through Storage Write in roughly the last minute or more.
  • Nothing guards a recreate. BigQuery takes no precondition on DROP TABLE or CREATE OR REPLACE, so a change made to the table between the plan and the recreate is lost too.
  • Writers do not see it. A writer connection opened before the recreate keeps getting successful acks for about 4 to 7 seconds, for rows that are lost. Then the next append waits about 120 seconds for DeadlineExceeded. New writers can get NotFound (is truncated, is re-created) for up to a few minutes, so the report does not promise that the table is writable yet. A writer still using the old column types gets InvalidArgument at once, and that one is not retryable.
  • Readers do not see it either. A read session opened before the recreate keeps returning the old rows and the old schema. Readers have to start a new session.

So stop writers and readers before a recreate. A dangerous recreate is logged at warn with the row count it read.

Full example available here.

Logs and spans

Every .plan() and .sync() runs in a BigQuery schema span with the table in the /bigquery/table field. A created table and every applied write are logged at info, and the data-losing steps at warn: a dropped column, a dangerous recreate and a sync that failed part way, with the report of what it had applied.

Datasets, tables and jobs

Besides the declarative table schemas, the library provides the plain admin calls over the v2 API:

  • create, read, update, delete and list datasets;
  • read, delete and list tables;
  • read, delete, cancel and list jobs.

Every call returns the library’s own types, such as BigQueryDataset, BigQueryTable and BigQueryJob, and a value BigQuery leaves out is None.

Dataset and table IDs

Every call that names a dataset or a table takes the validated ID types, BigQueryDatasetId and BigQueryTableId. Declare the ones your application knows up front as constants, and build references from them:

use bigquery::*;

const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");

// The table in the client's project
let orders = SHOP.table(ORDERS);
assert_eq!(orders.to_string(), "shop.orders");

// The same table in another project
let other = BigQueryDatasetRef::new("acme-prod", SHOP)?.table(ORDERS);
assert_eq!(other.to_string(), "acme-prod.shop.orders");
Ok::<(), bigquery::errors::BigQueryError>(())

from_static checks the literal at compile time, so an invalid one in a const fails the build. An ID that arrives at run time goes through new, parse() or try_into(), and deserializing an ID checks it the same way:

use bigquery::*;
let dataset = BigQueryDatasetId::new("shop_eu")?;
let table: BigQueryTableId = "orders_2026".parse()?;
let orders = dataset.table(table);
let _ = orders;

assert!(BigQueryDatasetId::new("shop.eu").is_err());
Ok::<(), bigquery::errors::BigQueryError>(())

The check is deliberately narrow. An ID is rejected when it is empty, longer than 1,024 bytes, or holds a control character or any of `, ', ", \, ., /, $ and @, since those would change a resource path, a dotted reference or SQL text built from the ID. BigQuery’s full naming rules are left to BigQuery, so an ID that passes here can still be refused by the call that sends it.

BigQueryTableRef and BigQueryDatasetRef also parse from text, dataset.table or project.dataset.table. Be aware not to parse text you do not trust this way, since the text picks the project. Build the reference from the IDs instead, or check that project() is None after parsing.

The project ID stays a plain String, because it is shared by every Google Cloud product. It is checked only for being non-empty and free of / and control characters. A location is a BigQueryLocation, such as BigQueryLocation::from_static("EU"), checked only for being non-empty.

Datasets

Datasets are reached through db.fluent().schema().dataset(..), which takes a BigQueryDatasetId for the client’s project or a BigQueryDatasetRef for another one:

use bigquery::*;
use std::time::Duration;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const EU: BigQueryLocation = BigQueryLocation::from_static("EU");

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let shop = db
    .fluent()
    .schema()
    .dataset(SHOP)
    .create()
    .location(EU)
    .description("Shop data")
    .default_table_expiration(Duration::from_secs(90 * 24 * 60 * 60))
    .labels([("team", "shop")])
    .execute()
    .await?;
println!("created {} in {:?}", shop.reference, shop.location);

let shop = db.fluent().schema().dataset(SHOP).get().await?;
println!("{:?}", shop.labels.get("team"));
Ok(())
}

The location cannot change after the dataset is created. Unset, BigQuery picks US. Creating a dataset that already exists fails with DataConflictError.

update() changes the description and the labels, and keeps everything else as the dataset has it:

use bigquery::*;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let shop = db
    .fluent()
    .schema()
    .dataset(SHOP)
    .update()
    .label("stage", "prod")
    .remove_label("tmp")
    .clear_description()
    .execute()
    .await?;
let _ = shop;
Ok(())
}
  • description(..) sets the description, clear_description() removes it;
  • label(key, value) adds or changes one label, remove_label(key) removes one, and labels(..) replaces all of them. Label changes apply in the order they are made.

The update reads the dataset and writes it back with the etag it read as an if-match precondition, so a dataset that changed in between fails with DataConflictError and nothing is written. Run it again. Only the metadata is written, the access list stays as it is.

There are two ways to delete a dataset:

  • delete() deletes an empty dataset. BigQuery refuses one that holds any table, view, model or routine, and nothing is deleted;
  • dangerously_delete_with_contents() deletes every table in the dataset with every row in it, and its views, models and routines, then the dataset. Nothing is kept for an undo.
use bigquery::*;
const SCRATCH: BigQueryDatasetId = BigQueryDatasetId::from_static("scratch");
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
db.fluent()
    .schema()
    .dataset(SCRATCH)
    .dangerously_delete_with_contents()
    .await?;
Ok(())
}

Listing datasets and tables

Listings are streams that fetch the next page when the stream reaches it:

use bigquery::*;
use futures::StreamExt;

const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let mut datasets = db.fluent().schema().datasets().stream_all().await?;
while let Some(dataset) = datasets.next().await {
    println!("{} in {:?}", dataset.reference, dataset.location);
}

let tables: Vec<BigQueryTableSummary> = db
    .fluent()
    .schema()
    .dataset(SHOP)
    .tables()
    .page_size(100)
    .stream_all()
    .await?
    .collect()
    .await;
let _ = tables;
Ok(())
}

stream_all() logs a failed page at error and ends there, since the listing cannot go on without the token that page would have returned. stream_all_with_errors() yields the failure as the stream’s last item instead. page_size(..) sets how many items one call returns, and datasets().project(..) lists another project’s datasets.

The listings return summaries, BigQueryDatasetSummary and BigQueryTableSummary, with what BigQuery’s list calls return: the reference, the location or table type, the labels, etc. Read the full value with get().

Full example available here.

Tables

Tables are created and changed through the declarative schemas, see Schema management. The same builder has two plain calls, which act on the table and ignore anything declared before them:

use bigquery::*;
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let orders = db.fluent().schema().table(SHOP.table(ORDERS)).get().await?;
for field in &orders.schema.fields {
    println!("{} {} {}", field.name, field.field_type, field.mode);
}
println!("{:?} rows, partitioning {:?}", orders.num_rows, orders.partitioning);

db.fluent().schema().table(SHOP.table(ORDERS)).delete().await?;
Ok(())
}

get() returns a BigQueryTable: the reference, the table_type, the schema as BigQueryTableSchema, the description, labels, partitioning, clustering, num_rows, num_bytes, the location, and the creation, last modified and expiration times. It reads views, materialized views, external tables and snapshots as well. The schema uses the same types as the type mapping, so INTEGER from the v2 API reads as BigQueryFieldType::Int64.

Be aware num_rows and num_bytes lag Storage Write: rows written through committed or pending streams show after about a minute, and rows on the default stream can take more than 5 minutes. A SELECT COUNT(*) gives the current count.

delete() deletes the table with every row in it. Nothing is kept for an undo, and open writers to it fail.

Jobs

Every query that runs as a job names it in its outcome, as a BigQueryJobRef with the project, the job ID and the location. The job calls are methods on BigQueryDb:

use bigquery::*;
async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let outcome = db
    .fluent()
    .query("SELECT 1")
    .job_creation_required()
    .execute()
    .await?;

if let Some(job_ref) = &outcome.job {
    let job = db.get_job(job_ref).await?;
    println!(
        "{:?} {:?}, billed {:?} bytes",
        job.statement_type, job.state, job.total_bytes_billed
    );
    db.delete_job(job_ref).await?;
}
Ok(())
}
  • get_job(&job_ref) reads a job: its type, state, error, user, labels, statement type, the creation, start and end times, and the bytes processed and billed;
  • cancel_job(&job_ref) asks BigQuery to cancel a running job and returns once the request is accepted, before the job stops. Dropping a query’s stream or future does not cancel its job;
  • delete_job(&job_ref) deletes the metadata of a finished job, for example a failed query’s SQL that should not stay in the job history. It does not cancel a running job.

A short query that BigQuery answered without a job names none. .job_creation_required() makes the query always run as a job.

stream_jobs(..) lists jobs, newest first, with the filters of BigQueryListJobsParams:

use bigquery::*;
use futures::StreamExt;

async fn example(db: BigQueryDb) -> BigQueryResult<()> {
let since: BigQueryInstant = "2026-10-01T00:00:00Z".parse().expect("valid instant");
let mut jobs = db
    .stream_jobs(
        BigQueryListJobsParams::new()
            .with_min_creation_time(since)
            .with_states(vec![BigQueryJobState::Done]),
    )
    .await?;
while let Some(job) = jobs.next().await {
    println!("{} {:?}", job.reference.job_id, job.total_bytes_billed);
}
Ok(())
}

The filters are the project, all_users (which needs the Owner role on the project), the minimum and maximum creation time, the states, the parent job of a script and the page size. stream_jobs_with_errors(..) yields a failed page as the last item, the same as for the other listings.

The job state, the job type, the statement type and the table type are enums with an Other(String) case for names the library does not know yet, and as_str() gives back the name BigQuery sent for every value.

Write streams

There is no listing of write streams. The Storage Write API has no list call, so a stream is reachable only by the name its creator got back.

Errors and retries

  • A missing dataset, table or job is DataNotFoundError;
  • creating a dataset that exists, and an update that lost the if-match precondition, are DataConflictError;
  • every call goes through the client’s retries.

A retry after a lost response repeats a call that may already have succeeded. So a create can report DataConflictError for the dataset it created, and a delete DataNotFoundError for the dataset, table or job it deleted.

Type mapping

The library reads and writes rows with serde, through its own codecs:

  • reads, from Storage Read and from inline query results, go through one Arrow decoder;
  • writes, through Storage Write, go through one protobuf encoder.

Both codecs share one mapping, so every BigQuery type has one set of Rust forms, the same in both directions. Whatever you write in a form, you can read back in the same form.

The library re-exports jiff, so the temporal types need no extra dependency.

Modes

A column’s mode decides the Rust shape around its type:

NULLABLEREQUIREDREPEATED
Rust fieldOption<T>TVec<T>
also on reada bare T, which fails a NULL row with NullForNonOptionOption<T>, always SomeOption<Vec<T>>, always Some
None on writestored as NULLNullForRequiredan empty array
field missing from the Rust row on writestored as NULLMissingRequiredFieldan empty array
NULL element on writeNullArrayElement

BigQuery never stores a NULL array: an empty REPEATED column reads as an empty Vec. A REPEATED column can be any serde sequence on both sides, such as VecDeque<T>, BTreeSet<T> or [T; N] when the array has exactly N elements.

An ARRAY column is the REPEATED mode of its element type, so there is no ARRAY type of its own. BigQuery has no array of arrays as a column type, and the encoder refuses one with UnsupportedType.

Every type at a glance

“Default” works with plain #[derive(Serialize, Deserialize)] and no attributes. The other forms work in both directions too.

BigQuery typeDefault Rust typeOther formsWrapper
INT64i64other integer types, range-checked
FLOAT64f64f32
NUMERIC, BIGNUMERICStringintegers, f64BigQueryDecimal<T>
BOOLbool
STRINGStringBox<str>, char, unit enums, any type serialized as a string
BYTESVec<u8>serde_bytes::ByteBuf, [u8; N], serde_json::Value (an array of byte numbers)
DATEjiff::civil::DateString, i32 daysBigQueryDate
TIMEjiff::civil::TimeString, i64 microseconds of the dayBigQueryTime
DATETIMEjiff::civil::DateTimeString, i64 civil microsecondsBigQueryDateTime
TIMESTAMPjiff::TimestampString, i64 microseconds since the epochBigQueryTimestamp
GEOGRAPHYString (WKT)
JSONby the Rust type, see JSONBigQueryJson<T>
INTERVALBigQueryIntervalString
RANGEBigQueryRange<T>
STRUCTa struct with derived serdeHashMap<String, V>, BTreeMap<String, V>, serde_json::Value

A form not in this table is not part of the mapping, even if it happens to work. A form the column does not take fails that row with TypeMismatch, in either direction.

A typical row looks like this:

use bigquery::*;
use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
struct Order {
    id: i64,
    customer: Option<String>,
    total: String,
    paid: bool,
    tags: Vec<String>,
    placed_on: jiff::civil::Date,
    placed_at: Option<jiff::Timestamp>,
    shipping: Option<Address>,
    attributes: serde_json::Value,
}

#[derive(Debug, Serialize, Deserialize)]
struct Address {
    city: String,
    street: Option<String>,
}

Integers and floats

INT64 reads into any Rust integer type, i8 to i128 and u8 to u128, and a value that does not fit fails that row with OutOfRange. On write a value above i64::MAX is OutOfRange.

FLOAT64 is f64, or f32, which is cast on read with no range check. NaN, both infinities and -0.0 round-trip exactly. Integers are not taken for FLOAT64 and floats are not taken for INT64, in either direction.

NUMERIC and BIGNUMERIC

NUMERIC holds 38 digits with 9 after the point, BIGNUMERIC 76 digits with 38 after the point. A NUMERIC(P, S) or BIGNUMERIC(P, S) column narrows them further.

  • String is the default and loses nothing. On read it is the canonical text with trailing fractional zeros removed, so 0.00 comes back as "0". On write it is a plain decimal with an optional sign and no exponent.
  • Integers are exact both ways. A value with a fractional part read into an integer fails that row with OutOfRange.
  • f64 works both ways and is lossy, rounded to 9 fractional digits (38 for BIGNUMERIC) on write and parsed into the nearest f64 on read.
  • BigQueryDecimal<T> holds any decimal type that round-trips through its Display and FromStr, such as bigdecimal::BigDecimal or rust_decimal::Decimal. The text form is used whatever the type’s own serde does, since some decimal types serialize as f64. For a bare field there are serialize_as_decimal and serialize_as_optional_decimal.
mod bigdecimal { pub type BigDecimal = String; }
use bigquery::*;
use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
struct Invoice {
    total: BigQueryDecimal<bigdecimal::BigDecimal>,
    #[serde(with = "bigquery::serialize_as_optional_decimal")]
    discount: Option<bigdecimal::BigDecimal>,
}

On write, a NUMERIC with more than 29 digits before the point or more than 9 after it, a BIGNUMERIC outside its range or with more than 38 digits after the point, NaN and the infinities fail that row with OutOfRange. A text that T::from_str rejects on read, for example rust_decimal::Decimal above its 28 digits, fails with Custom.

Be aware of what BigQuery does on a NUMERIC(P, S) column. The library sends every value at the full scale of the type, and BigQuery rounds the digits beyond the column’s scale half away from zero, with no error. So 1.245 and -1.245 written to a NUMERIC(10, 2) column are stored as 1.25 and -1.25. A value over the column’s precision is checked by BigQuery only.

BOOL, STRING and BYTES

BOOL is bool only. Integers and strings are not taken for it.

STRING is String, Box<str>, a char for one character, a unit-variant enum, or any type whose serde form is a string, such as uuid::Uuid. The maximum length of a STRING(n) column is checked by BigQuery only.

BYTES is Vec<u8>, serde_bytes::ByteBuf, or [u8; N] when the value has exactly N bytes. A string is not taken for BYTES on write, and bytes are not taken for STRING: BigQuery stores a string sent to BYTES as its raw UTF-8 bytes and does not decode base64, so a base64 text would be stored as text by accident. A serde_json::Value reads BYTES as an array of byte numbers, such as [0, 255], and the writer takes that array back for BYTES.

GEOGRAPHY

GEOGRAPHY is a String. BigQuery returns WKT and takes both WKT and GeoJSON on write, storing either as a geography. Invalid text is rejected by BigQuery.

Dates and times

The four temporal types map to jiff by default, with no attribute:

BigQuery typejiff typeRangeString formInteger form
DATEjiff::civil::Date0001-01-01 to 9999-12-31YYYY-MM-DDi32 days since 1970-01-01
TIMEjiff::civil::Time00:00:00 to 23:59:59.999999HH:MM:SS[.ffffff]i64 microseconds of the day
DATETIMEjiff::civil::DateTime0001-01-01T00:00:00 to 9999-12-31T23:59:59.999999YYYY-MM-DDTHH:MM:SS[.ffffff]i64 civil microseconds since 1970-01-01T00:00:00
TIMESTAMPjiff::Timestamp0001-01-01 to 9999-12-31T23:59:59.999999 UTCRFC 3339i64 microseconds since the epoch

Some details for each:

  • BigQuery keeps microseconds, so sub-microsecond digits are dropped on write, floored for TIMESTAMP;
  • a DATE or DATETIME before year 1 on write is OutOfRange, since jiff allows years down to -9999;
  • a DATETIME has no offset, so jiff::Timestamp is not taken for it, and a TIMESTAMP is not taken by jiff::civil::DateTime or jiff::ZonedDateTime;
  • a DATETIME String uses T on read and takes T or a space on write;
  • a TIMESTAMP String is always ...Z on read, and on write needs Z or ±HH:MM, with T or a space between date and time. Other shapes, such as +00 or UTC, fail with InvalidText.

Be aware of the TIMESTAMP range. jiff::Timestamp ends at 9999-12-30T22:00:00.999999999Z, a day before BigQuery’s maximum. A TIMESTAMP above it read into a jiff type, the wrapper included, fails that row with OutOfRange. String and i64 hold the full range, so read into one of those if your data can have such values, for example 9999-12-31 used as “never”. The same holds for the TIMESTAMP ends of a RANGE<TIMESTAMP>.

TIMESTAMP columns with picosecond precision (timestamp_precision = 12) are not supported yet, and fail with UnsupportedType.

Temporal wrappers

jiff’s serde speaks text only, so a plain jiff field is printed as text by the codec and parsed by jiff, on both read and write. The wrappers keep the jiff types and skip the text:

  • BigQueryTimestamp(pub jiff::Timestamp);
  • BigQueryDate(pub jiff::civil::Date);
  • BigQueryTime(pub jiff::civil::Time);
  • BigQueryDateTime(pub jiff::civil::DateTime).

With the library’s codecs they read and write BigQuery’s integers directly. In any other serde format they are the same as the plain jiff type, so a row of wrappers serializes to the same JSON as a row of plain jiff fields. For a field you do not want to change the type of, the with modules do the same:

use bigquery::*;
use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
struct Event {
    happened_at: BigQueryTimestamp,
    day: Option<BigQueryDate>,
    #[serde(with = "bigquery::serialize_as_timestamp")]
    received_at: jiff::Timestamp,
    #[serde(with = "bigquery::serialize_as_optional_datetime")]
    local_time: Option<jiff::civil::DateTime>,
}

The modules are serialize_as_{timestamp,date,time,datetime} and their serialize_as_optional_.. forms. A wrapper is checked against its column, so a BigQueryDate on a TIMESTAMP column fails with TypeMismatch, since its integer form would read microseconds as days.

Use them for wide tables or hot paths. benches/read_codec.rs and benches/write_codec.rs compare plain jiff fields, the wrappers and integers.

JSON

A JSON column maps by the Rust type, with no wrapper:

  • a String field is the JSON text as it is, in both directions;
  • any other serde shape, such as serde_json::Value, a typed struct, a map or a sequence, is parsed on read and printed as JSON on write.

The same holds for a JSON field inside a STRUCT and for the elements of an ARRAY<JSON>:

use bigquery::*;
use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
struct Payload {
    kind: String,
    amount: i64,
}

#[derive(Debug, Serialize, Deserialize)]
struct Message {
    // JSON column, as text
    raw: String,
    // JSON column, as any JSON value
    document: serde_json::Value,
    // JSON column, parsed into a struct
    payload: Option<Payload>,
    // ARRAY<JSON> column
    events: Vec<serde_json::Value>,
}

JSON has two kinds of null, SQL NULL and the JSON value null, and the rules for them are:

  • Option<serde_json::Value> keeps them apart: SQL NULL is None, JSON null is Some(Value::Null);
  • Option<String> gets SQL NULL as None and JSON null as the text "null";
  • any other Option<T>, such as Option<Payload> or Option<i64>, reads both as None;
  • a bare typed field, such as Payload, fails a JSON null with Custom.

Text that does not parse into the Rust type on read fails that row with Custom. On write, invalid JSON text from a String field is checked by BigQuery only: the append fails with RowErrors, with a FIELDS_ERROR for that row, and no row of that batch is stored.

BigQueryJson<T> is for the two cases the rule above leaves out:

  • BigQueryJson<String> is a JSON string value parsed into a String, where a plain String field would be the raw text with its quotes;
  • a top-level Value::String is written the way a String is, as the JSON text itself. A document that can be a bare JSON string needs BigQueryJson<serde_json::Value>.

In other serde formats BigQueryJson<T> is the JSON text as a string. For a bare field there are serialize_as_json and serialize_as_optional_json.

INTERVAL

An INTERVAL has three parts, each with its own sign, so -1 month +3 days is a valid value. That is what BigQueryInterval holds:

use bigquery::*;

let interval = BigQueryInterval {
    months: -1,
    days: 3,
    nanos: 4 * 3_600 * 1_000_000_000,
};

// jiff::Span has one sign for all its units, so the conversion can fail
let span: Result<jiff::Span, _> = interval.try_into();
assert!(span.is_err());

let interval = BigQueryInterval::try_from(jiff::Span::new().days(3).hours(4))?;
assert_eq!(interval.days, 3);
Ok::<(), bigquery::errors::BigQueryError>(())

A String works too, in BigQuery’s canonical form [-]Y-M [-]D [-]H:M:S[.ffffff], for example 1-2 -3 4:5:6.000789.

BigQuery keeps microseconds, so nanos that are not whole microseconds fail the row with OutOfRange on write. Be aware of one BigQuery limit: Storage Read fails the whole stream for an INTERVAL whose time part is beyond about 2,562,047 hours, and no Arrow client can work around it. So the library refuses such a value on write, where it could not be read back. A value like that written another way can still be read with CAST(iv AS STRING) in a query.

RANGE

BigQueryRange<T> holds a RANGE<DATE>, RANGE<DATETIME> or RANGE<TIMESTAMP>, with T any Rust form of the element type, the temporal wrappers included:

use bigquery::*;
use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
struct Promotion {
    valid: Option<BigQueryRange<jiff::civil::Date>>,
    window: BigQueryRange<BigQueryTimestamp>,
}

start is inclusive and end exclusive, and None is an unbounded end. A NULL range into a bare BigQueryRange<T> fails with NullForNonOption. Whether start comes before end is checked by BigQuery.

NULLABLE and REPEATED RANGE columns keep unbounded ends as None. A REQUIRED RANGE column has no NULL for its ends, so BigQuery sends an unbounded end of it as 1970-01-01 (or the epoch for DATETIME and TIMESTAMP). The library reads that epoch as None when it breaks start < end, which BigQuery keeps for every range:

  • an epoch end at or before the start is an unbounded end;
  • an epoch start at or after the end is an unbounded start;
  • [UNBOUNDED, UNBOUNDED) reads as both None.

A real epoch bound looks the same as an unbounded one, so four cases read wrong:

  • an unbounded start with an end after the epoch reads the start as the epoch;
  • an unbounded end with a start before the epoch reads the end as the epoch;
  • [1970-01-01, UNBOUNDED) reads as both None, the same as [UNBOUNDED, UNBOUNDED);
  • [UNBOUNDED, 1970-01-01) reads as both None too.

Use a NULLABLE RANGE column where these values matter.

STRUCT

A STRUCT column is a struct with derived serde, its fields matched by name, or a map with string keys, or a serde_json::Value object. Nested structs and ARRAY<STRUCT> work the same way.

  • a NULL STRUCT into a bare struct fails with NullForNonOption, also when every field of the struct is an Option, so use Option<Address> for a NULLABLE STRUCT;
  • on write a Rust field with no column fails with UnknownField naming the path, for example shipping.inner.zip, and a REQUIRED subfield the row never wrote with MissingRequiredField;
  • a tuple works as a whole row only, a STRUCT column into a tuple is TypeMismatch.

Full example available here.

Field names and serde attributes

Fields are matched to columns by name, so the order of the fields does not matter. The usual serde attributes work in both directions:

  • rename and rename_all, for example for camelCase columns;
  • alias, for a column that has another name in some tables;
  • skip, default, flatten and deny_unknown_fields;
  • a missing Option column reads as None, and a missing column with default takes its default.

When an aliased field finds both its own column and the alias column in one table, the row fails with serde’s duplicate field, the same as serde_json gives for that input. A hand-written Deserialize that answers struct fields by its own integer numbering is not supported; one that takes field names works.

path! builds column names from Rust fields. For a field renamed to camelCase use path_camel_case!, for any other #[serde(rename)] the column name as a string.

Dynamic rows

A row can be a serde_json::Value, a map with string keys, or an #[serde(untagged)] enum on read, when the shape is not known up front. For whole Arrow batches with no serde at all, the read and query builders have record_batches().

Query parameters

Query parameters take the same Rust forms. .param(name, value) infers the type from the value’s serde form:

  • integers are INT64, floats FLOAT64, strings STRING, bool BOOL, serde_bytes values BYTES;
  • sequences are ARRAY, structs and string-keyed maps STRUCT;
  • the wrappers are their own types: BigQueryTimestamp is TIMESTAMP, BigQueryDate DATE, etc., BigQueryJson<T> JSON and BigQueryInterval INTERVAL;
  • BigQueryDecimal<T> is NUMERIC, or BIGNUMERIC for a value NUMERIC cannot hold;
  • BigQueryRange<T> is RANGE when its bounds are temporal wrappers.

A Vec<u8> serializes as a sequence of integers, so it is an ARRAY<INT64>; use serde_bytes::ByteBuf or param_as for BYTES.

A plain jiff value serializes as text, so it is inferred as a STRING. None, an empty sequence and a sequence of mixed types cannot be inferred either. For those, .param_as(name, type, value) sets the type and takes the value in any form the write path takes for it, and None there is a NULL of that type:

use bigquery::*;
async fn example(db: BigQueryDb, since: jiff::Timestamp) -> BigQueryResult<()> {
let outcome = db
    .fluent()
    .query(
        "SELECT COUNT(*) AS n FROM shop.orders \
         WHERE placed_at >= @since AND (@customer IS NULL OR customer = @customer)",
    )
    .param_as("since", BigQueryFieldType::Timestamp, since)
    .param_as(
        "customer",
        BigQueryFieldType::String { max_length: None },
        None::<String>,
    )
    .execute()
    .await?;
let _ = outcome;
Ok(())
}

BigQueryParamType::array_of(..) declares an ARRAY parameter.

Schema types

Table schemas use one vocabulary, whatever API they came from: BigQueryTableSchema with its BigQueryFieldSchema columns, each with a name, a BigQueryFieldType, a BigQueryFieldMode, a description and a default value expression.

The v2 API names types with the legacy names, the Storage API with an enum, and Arrow with physical types only, so the library normalises all of them. INTEGER is Int64, FLOAT is Float64, BOOLEAN is Bool, RECORD is Struct, DECIMAL is Numeric, BIGDECIMAL is BigNumeric, and an empty mode, as DDL creates it, is Nullable. Type parameters are part of the type: String { max_length }, Numeric(Some(BigQueryDecimalParams { precision, scale })), etc. So two schemas compare with ==.

Display prints GoogleSQL type syntax, such as STRING(10), NUMERIC(10, 2), RANGE<DATE> and STRUCT<a INT64, b ARRAY<STRING>>.

Codec errors

A value that does not fit fails as BigQueryError::SerializeError on write and BigQueryError::DeserializeError on read. Both carry a BigQuerySerializationError with a BigQueryCodecErrorKind, the field path, such as recs[1].v, and on read the row index. A read error fails only its own row: the other rows of the batch still decode.

KindWhen
TypeMismatchthe Rust form is not one the column’s type takes, or a temporal wrapper is on a column of another type
NullForNonOptionread: NULL into a target that is not an Option, a NULL STRUCT or RANGE included
NullForRequiredwrite: None for a REQUIRED field
NullArrayElementwrite: None inside a REPEATED field
OutOfRangethe value is outside the BigQuery type (a year before 1, NUMERIC digits, the INTERVAL time part) or outside the Rust target (a narrower integer, jiff’s TIMESTAMP maximum)
InvalidTexta text form that does not parse, such as a malformed DATE, TIMESTAMP, NUMERIC or INTERVAL
UnknownFieldwrite: a field or map key with no column
MissingRequiredFieldwrite: a REQUIRED field the row never wrote
UnsupportedTypea column type the library does not handle, such as TIMESTAMP with picosecond precision
RowTooLargewrite: one encoded row is larger than the request budget
Customanything the target type’s own serde impl raises, such as missing field or a FromStr failure

Observability

The library uses tracing. Every call opens one span at the DEBUG level, and the spans carry what a call used besides how long it took: bytes processed and billed, slot milliseconds, rows and bytes read or sent, etc.

The spans are:

  • BigQuery Query: one query terminal call, query(), execute(), dry_run() etc.;
  • BigQuery Read: one table read through the Storage Read API, including the read of a large query result;
  • BigQuery streaming write: one writer, from insert(), create_streaming_writer or create_cdc_writer;
  • BigQuery commit write streams: one commit_write_streams call;
  • BigQuery Cancel Job, and the admin spans BigQuery dataset, BigQuery datasets, BigQuery table, BigQuery tables, BigQuery job, BigQuery jobs and BigQuery schema, which carry only the resource they work on.

Retries are logged at WARN inside the span of the call that retried.

Showing the spans

Any tracing subscriber works. With tracing-subscriber, an EnvFilter that enables DEBUG for the library and span close events, you see every span with its fields when it ends:

use bigquery::*;
use tracing_subscriber::fmt::format::FmtSpan;

async fn example() -> BigQueryResult<()> {
tracing_subscriber::fmt()
    .with_env_filter("info,bigquery=debug")
    .with_span_events(FmtSpan::CLOSE)
    .init();

let db = BigQueryDb::new("my-gcp-project-id").await?;
let outcome = db
    .fluent()
    .query("SELECT word FROM `bigquery-public-data.samples.shakespeare` LIMIT 10")
    .execute()
    .await?;
let _ = outcome;
Ok(())
}

Full example available here.

OpenTelemetry

With tracing-opentelemetry every span field becomes a span attribute with the same name, such as /bigquery/bytes_billed, so you can see the cost of a request next to its latency in Cloud Trace, Jaeger, etc.

The fields are declared empty when the span opens and recorded once their value is known, which tracing-opentelemetry exports the same way as fields given at the start. Add its layer to your subscriber as usual:

use opentelemetry::trace::TracerProvider;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;

let provider = opentelemetry_sdk::trace::SdkTracerProvider::builder()
    .with_batch_exporter(exporter) // any exporter, e.g. opentelemetry-otlp
    .build();

tracing_subscriber::registry()
    .with(tracing_subscriber::EnvFilter::new("info,bigquery=debug"))
    .with(tracing_opentelemetry::layer().with_tracer(provider.tracer("my-service")))
    .init();

Be aware the library spans are DEBUG, so your filter has to enable that level for bigquery.

Common rules for the fields

  • A figure BigQuery did not report is left unrecorded, never recorded as 0. The two estimates of a read session are the exception: BigQuery sends them as plain numbers, with no way to tell 0 from unset.
  • The library makes no call only for the stats. Every figure comes from a response the call receives anyway, or is counted locally.
  • No span field carries the SQL text, a parameter value or a row filter. The query span has the length of the SQL only.
  • A row that stream_query() or a table read skips is logged with the kind of error, the row and the field path, without the message, which can contain the cell’s text. A retry is logged with BigQuery’s error message, as BigQuery wrote it.

Query span

BigQuery Query covers a query from the Query call until the library knows where the rows are. It does not cover streaming the rows: a result read through Storage Read has its own BigQuery Read span, a sibling of the query span under your current span.

FieldWhat it meansWhere it comes from
/bigquery/sql_lenthe length of the SQL text in bytesthe SQL, when the span opens
/bigquery/job_idthe job that ran the query; unset for a query BigQuery ran without a jobthe job reference of the Query response
/bigquery/query_idthe ID BigQuery gave the query, with or without a jobthe Query response
/bigquery/locationwhere the query ranthe job reference, otherwise the location of the Query response
/bigquery/statement_typeSELECT, INSERT, UPDATE, CREATE_TABLE, etc.the Query response, otherwise the job’s query statistics
/bigquery/bytes_processedbytes the query processedthe Query response, GetQueryResults, then the job’s statistics
/bigquery/bytes_billedbytes billed, after BigQuery’s rounding and minimumsthe Query response, then the job’s query statistics
/bigquery/slot_msslot milliseconds the query usedthe Query response, then the job’s statistics
/bigquery/cache_hitwhether the query cache answeredthe Query response, GetQueryResults, then the job’s query statistics
/bigquery/dml_rowsrows a DML statement changedthe same three as cache_hit
/bigquery/total_rowsrows in the resultthe Query response, then GetQueryResults
/bigquery/routewhere the rows came from: inline, storage_read or nonethe library, for the terminals that read rows

Each figure is taken from the first response that reports it, in the order of the table. Which of them a query can have depends on its route:

  • Inline, a result complete in the first response: only the Query response’s own figures, since there is no further call on this route. In the live tests BigQuery reported bytes processed, bytes billed, slot milliseconds and cache hit there;
  • Storage Read, a larger result: the Query response, and then the statistics of the job that the library reads anyway to find the destination table;
  • Polled, a job that was not complete in the first response: GetQueryResults and the job’s statistics. GetQueryResults has no bytes billed and no slot milliseconds, so those come from the job only;
  • execute(): the same as inline when the job completes in the first response. route is not recorded, since execute() reads no rows;
  • dry_run(): none of the figures. Its estimate is in BigQueryDryRunResult instead.

The same figures are in BigQueryJobStats from query_with_stats(), stream_query_with_stats() and execute(), see Job stats.

slot_ms is not 0 even for a query that reads no table. On 1,000 generated rows BigQuery reported 0 bytes processed and billed, and 25 slot ms inline, 152 slot ms through Storage Read.

Read span

BigQuery Read covers one read session, from opening it until the stream of rows or batches is dropped.

FieldWhat it meansWhere it comes from
/bigquery/tablethe table being read, project.dataset.table or dataset.tablethe read, when the span opens
/bigquery/streamsthe read streams BigQuery gave the sessionthe read session
/bigquery/estimated_bytes_scannedBigQuery’s estimate of the bytes the session scansthe read session, as BigQuery sends it
/bigquery/estimated_rowsBigQuery’s estimate of the rows the session returnsthe read session, as BigQuery sends it
/bigquery/rows_readrows received over every streamthe sum of the row counts of every ReadRows response
/bigquery/bytes_readuncompressed bytes receivedthe sum of the uncompressed sizes the ReadRows responses report, when any of them does
/bigquery/throttle_percenthow much BigQuery throttled the read, 0 to 100the highest throttle state any ReadRows response reported, when any of them does

The three running totals are recorded when the stream is dropped, which also happens when it ends or fails. A read you drop early records what it got so far.

The number of streams is what BigQuery gave, which can be fewer than the library asked for. In the benchmarks the library asked for 16 and got 4 for a 1M-row table.

Write span

BigQuery streaming write covers one writer, from opening its write stream until its background task ends. insert() and the CDC writer go through the same writer, so they have the same span.

FieldWhat it meansWhere it comes from
/bigquery/tablethe table being writtenthe writer, when the span opens
/bigquery/write_modeDefault, Committed, Pending or Bufferedthe writer’s options, when the span opens
/bigquery/rows_appendedrows in the batches BigQuery acknowledgedcounted by the writer, the same as rows_written in BigQueryWriteSummary
/bigquery/bytes_sentthe encoded size of every AppendRows request sent, resends included, before gRPC framingcounted by the writer, the same as bytes_sent in BigQueryWriteSummary
/bigquery/appendsAppendRows requests sent, resends includedcounted by the writer
/bigquery/retriesrequests that sent a batch againcounted by the writer

The four counts are recorded when the writer’s background task ends: after finish() or finalize(), when the writer is dropped, and when it fails for good.

BigQuery commit write streams has only /bigquery/table; the commit time is in the result of the commit.

Testing with the fake

The library provides a fake BigQuery for your tests, in the bigquery::testing module. BigQueryFake runs a gRPC server on a loopback port and hands out a real BigQueryDb connected to it, so the code under test runs unchanged, through the same requests, codecs, retries and errors as against BigQuery. It needs no credentials and no network.

A test scripts the fake with your own serde types:

  • a query rule answers the statement it matches with rows, DML counts or a failure;
  • a table holds rows, which reads serve and writes append to, and which the test reads back;
  • a fault makes the calls of one RPC fail.

The tests need an async runtime in your dev-dependencies:

[dev-dependencies]
tokio = { version = "1", features = ["macros", "rt"] }

The examples below share a row type and the code under test, and each of them is the body of a #[tokio::test] async fn ...() -> BigQueryResult<()>:

use bigquery::*;
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Order {
    id: i64,
    customer: String,
    total: f64,
}

const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";

async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
    db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
}

async fn save(db: &BigQueryDb, orders: &[Order]) -> BigQueryResult<BigQueryWriteSummary> {
    db.fluent().insert().into(SHOP.table(ORDERS)).objects(orders).execute().await
}

Queries

There is no SQL engine in the fake. A query answers only as a rule scripts it:

use bigquery::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Order { id: i64, customer: String, total: f64 }
const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";
async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
    db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
}
#[tokio::main(flavor = "current_thread")]
async fn main() -> BigQueryResult<()> {
use bigquery::testing::BigQueryFake;

let fake = BigQueryFake::start().await?;
let alice = vec![Order { id: 1, customer: "Alice".into(), total: 120.0 }];
let rule = fake
    .when_query_match(ORDERS_OF)
    .param("customer", "Alice")
    .returns_rows(|columns| columns.from_type::<Order>(), &alice)?;

assert_eq!(orders_of(fake.db(), "Alice").await?, alice);
assert_eq!(rule.calls(), 1);
Ok(())
}

The rows are any Serialize type, and their columns are declared as a table schema declares them, here with |columns| columns.from_type::<Order>(). A rule matches:

  • the SQL text exactly, with no whitespace or case folding;
  • a statement from sql_file! by the text of its file;
  • any parameters, unless the rule declares some with .param(..), .params(..) or .positional_param(..). Then the call’s named parameters must equal them as a set, and its positional ones in order.

Rules are tried in the order they were registered, and the first that matches answers. .times(n) limits a rule to n calls, retries included, so a later rule answers the calls after them. Besides returns_rows, a rule answers with returns_dml for a DML statement and the rows it changed, returns_statement for DDL, fails for a refused call and fails_job for a job that fails once it runs. One rule answers every terminal of a call: the rows for query(), the stats for execute() and the schema for dry_run(), with .bytes_processed(..) if set.

fake.when_query(|query| ...) matches with your own code instead of the SQL text. The closure gets the call’s SQL and its parameters, read back as any Deserialize type:

use bigquery::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Order { id: i64, customer: String, total: f64 }
const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";
async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
    db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
}
#[tokio::main(flavor = "current_thread")]
async fn main() -> BigQueryResult<()> {
use bigquery::testing::BigQueryFake;

let fake = BigQueryFake::start().await?;
let alice = vec![Order { id: 1, customer: "Alice".into(), total: 120.0 }];
let rule = fake
    .when_query(|query| {
        query.sql().contains("FROM shop.orders")
            && query.param::<String>("customer").is_some_and(|name| name.starts_with('A'))
    })
    .returns_rows(|columns| columns.from_type::<Order>(), &alice)?;

assert_eq!(orders_of(fake.db(), "Alice").await?, alice);
assert_eq!(rule.calls(), 1);
Ok(())
}

query.param::<T>(name) reads a named parameter, with the name compared ignoring case as BigQuery compares it, and query.positional_param::<T>(index) a positional one, the first at 0. Both return None when the call has no such parameter or its value does not read as T. A STRUCT reads as a struct or a map, an ARRAY as a Vec, and a RANGE as BigQueryRange. .param(..) and .positional_param(..) narrow a when_query rule as they narrow a SQL one.

Tables, reads and writes

A table on the fake holds rows. fake.table(..) creates it with its columns, and its dataset if that is missing, and .rows(..) adds rows a read is served:

use bigquery::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Order { id: i64, customer: String, total: f64 }
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[tokio::main(flavor = "current_thread")]
async fn main() -> BigQueryResult<()> {
use bigquery::testing::BigQueryFake;

let fake = BigQueryFake::start().await?;
let orders = vec![
    Order { id: 1, customer: "Alice".into(), total: 120.0 },
    Order { id: 2, customer: "Bob".into(), total: 80.5 },
];
fake.table(SHOP.table(ORDERS), |columns| columns.from_type::<Order>())
    .rows(&orders)
    .create()?;

let read: Vec<Order> = fake.db().fluent().select().from(SHOP.table(ORDERS)).obj().query().await?;
assert_eq!(read, orders);
Ok(())
}

Writes append to the table, and fake.rows::<T>(table) reads back the rows written:

use bigquery::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Order { id: i64, customer: String, total: f64 }
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
async fn save(db: &BigQueryDb, orders: &[Order]) -> BigQueryResult<BigQueryWriteSummary> {
    db.fluent().insert().into(SHOP.table(ORDERS)).objects(orders).execute().await
}
#[tokio::main(flavor = "current_thread")]
async fn main() -> BigQueryResult<()> {
use bigquery::testing::BigQueryFake;

let fake = BigQueryFake::start().await?;
fake.table(SHOP.table(ORDERS), |columns| columns.from_type::<Order>())
    .create()?;
let orders = vec![
    Order { id: 1, customer: "Alice".into(), total: 120.0 },
    Order { id: 2, customer: "Bob".into(), total: 80.5 },
];

let summary = save(fake.db(), &orders).await?;

assert_eq!(summary.rows_written, 2);
let written: Vec<Order> = fake.rows(SHOP.table(ORDERS))?;
assert_eq!(written, orders);
Ok(())
}

rows returns the visible rows, in the order they were acknowledged. Rows become visible as BigQuery makes them visible for each write mode:

  • the default and committed streams: when acknowledged;
  • a buffered stream: up to the last flushed offset;
  • a pending stream: at its commit, and never without one.

Rows are matched to the table’s columns by name, ignoring case, and the fields of a RECORD column the same way. Rows written before a column or a RECORD field was added, and flushed or committed after, read it as NULL, as do the rows the table already holds.

A table read is served the table’s rows, projected to the selected columns. The fake evaluates no filter, so a read with a row restriction needs a rule, fake.when_read(table).row_restriction(..), with the rows it returns. fake.when_rows_rejected::<T>(table, reason, |row| ...) fails each append request that has a row the predicate accepts, with a row error per such row, and writes nothing of that request, as BigQuery does. A row written from a serde type fails the same way when it has a value beyond the precision of its NUMERIC(p,s) or BIGNUMERIC(p,s) column, and a value with more decimal places than the scale is rounded.

A CDC writer’s rows are recorded as changes and not applied to the table’s rows. fake.changes::<T>(table) returns them, each with its change type and sequence number.

Datasets and tables, from fake.create_dataset(..), fake.table(..) or the code under test, are the fake’s state, and the admin calls are served from it.

Faults and retries

A query rule’s fails scripts the failure of a query. fake.when_fault(rpc).fails(..) does the same for every other RPC, and .on_table(..) narrows it to the calls on one table. A fault answers before every rule and before the tables, as one of:

  • BigQueryFakeFault::status(code, message), a gRPC status, which the client maps as it maps BigQuery’s own: NotFound is a DataNotFoundError, Unavailable a retryable DatabaseError, etc.;
  • BigQueryFakeFault::ConnectionDropped, a connection closed without an answer, which the client retries as a transport error;
  • BigQueryFakeFault::Hang, a call that never answers. The client waits until its own timeout, if it has one.

Retries against the fake wait no backoff, so a test that scripts failures runs at full speed. A failure a retry gets past, then an outage the client gives up on:

use bigquery::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Order { id: i64, customer: String, total: f64 }
const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";
async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
    db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
}
#[tokio::main(flavor = "current_thread")]
async fn main() -> BigQueryResult<()> {
use bigquery::testing::{BigQueryFake, BigQueryFakeCode, BigQueryFakeFault};

let fake = BigQueryFake::start().await?;
let alice = vec![Order { id: 1, customer: "Alice".into(), total: 120.0 }];
let unavailable =
    || BigQueryFakeFault::status(BigQueryFakeCode::Unavailable, "backend went away");
let lost = fake.when_query_match(ORDERS_OF).times(1).fails(unavailable())?;
let answer = fake
    .when_query_match(ORDERS_OF)
    .param("customer", "Alice")
    .returns_rows(|columns| columns.from_type::<Order>(), &alice)?;
let outage = fake
    .when_query_match(ORDERS_OF)
    .param("customer", "Bob")
    .fails(unavailable())?;

assert_eq!(orders_of(fake.db(), "Alice").await?, alice);
assert_eq!((lost.calls(), answer.calls()), (1, 1));

let failed = orders_of(fake.db(), "Bob").await;
assert!(matches!(&failed, Err(err) if err.retry_possible()), "{failed:?}");
assert_eq!(outage.calls(), 4, "the first attempt and max_retries = 3 retries");
Ok(())
}

On AppendRows, ConnectionDropped writes the request and then drops the connection, as an acknowledgement lost in transit. The client sends the request again, so on the default stream, which is at least once, its rows are written twice. A write with .exactly_once() sends it with its offset, and the table keeps one copy:

use bigquery::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Order { id: i64, customer: String, total: f64 }
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
#[tokio::main(flavor = "current_thread")]
async fn main() -> BigQueryResult<()> {
use bigquery::testing::{BigQueryFake, BigQueryFakeFault, BigQueryFakeRpc};

let fake = BigQueryFake::start().await?;
fake.table(SHOP.table(ORDERS), |columns| columns.from_type::<Order>())
    .create()?;
let lost = fake
    .when_fault(BigQueryFakeRpc::AppendRows)
    .times(1)
    .fails(BigQueryFakeFault::ConnectionDropped)?;
let orders = vec![Order { id: 1, customer: "Alice".into(), total: 120.0 }];

fake.db()
    .fluent()
    .insert()
    .into(SHOP.table(ORDERS))
    .objects(&orders)
    .exactly_once()
    .execute()
    .await?;

assert_eq!(fake.rows::<Order>(SHOP.table(ORDERS))?, orders);
assert_eq!(lost.calls(), 1);
Ok(())
}

Unmatched calls

A call that nothing answers, such as a query no rule matches or an RPC the fake does not serve, fails with Unimplemented, which the client does not retry. The fake records it, and fake.verify() panics with every such call and the rules that were there to answer it. It also panics for a rule limited with .times(n) that did not answer exactly n calls.

verify runs when the fake is dropped, so a test fails on an unscripted call even when the code under test swallows its error:

use bigquery::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Order { id: i64, customer: String, total: f64 }
const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";
async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
    db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
}
#[tokio::main(flavor = "current_thread")]
async fn main() -> BigQueryResult<()> {
use bigquery::testing::BigQueryFake;

let fake = BigQueryFake::start().await?;
fake.when_query_match(ORDERS_OF)
    .param("customer", "Alice")
    .returns_rows(|columns| columns.from_type::<Order>(), Vec::<Order>::new())?;

let unmatched = orders_of(fake.db(), "Bob").await;
assert!(matches!(&unmatched, Err(err) if !err.retry_possible()), "{unmatched:?}");

drop(fake); // Panics: no rule answers the query for Bob
Ok(())
}

If the test is already panicking when the fake is dropped, the problems are logged with tracing::error! instead, so the first panic is the one the test reports.

Choosing the fake or BigQuery at startup

fake.db() is a real BigQueryDb connected to the fake’s loopback server, so the fake needs no trait shared with the real client. Code that takes a BigQueryDb, as an argument or a field of the application state, runs unchanged against either. This also makes it possible to run the whole application against the fake, with a factory that picks one at startup from the application’s own configuration, such as an environment variable. The library itself reads no variable for this.

The factory returns an owner of the client. Dropping a BigQueryFake stops its server and runs verify(), so the owner keeps the fake alive next to the client it hands out:

use bigquery::*;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
struct Order { id: i64, customer: String, total: f64 }
const SHOP: BigQueryDatasetId = BigQueryDatasetId::from_static("shop");
const ORDERS: BigQueryTableId = BigQueryTableId::from_static("orders");
const ORDERS_OF: &str = "SELECT id, customer, total FROM shop.orders WHERE customer = @customer";
async fn orders_of(db: &BigQueryDb, customer: &str) -> BigQueryResult<Vec<Order>> {
    db.fluent().query(ORDERS_OF).param("customer", customer).obj().query().await
}
pub struct AppBigQuery {
    db: BigQueryDb,
    _fake: Option<bigquery::testing::BigQueryFake>,
}

impl AppBigQuery {
    pub fn db(&self) -> &BigQueryDb {
        &self.db
    }
}

/// BigQuery in `project`, or a seeded fake when `APP_BIGQUERY=fake` is set.
pub async fn bigquery_db(project: &str) -> BigQueryResult<AppBigQuery> {
    if std::env::var("APP_BIGQUERY").as_deref() == Ok("fake") {
        let fake = bigquery::testing::BigQueryFake::start().await?;
        seed(&fake)?;
        return Ok(AppBigQuery {
            db: fake.db().clone(),
            _fake: Some(fake),
        });
    }
    Ok(AppBigQuery {
        db: BigQueryDb::new(project).await?,
        _fake: None,
    })
}

fn seed(fake: &bigquery::testing::BigQueryFake) -> BigQueryResult<()> {
    let alice = vec![Order { id: 1, customer: "Alice".into(), total: 120.0 }];
    fake.table(SHOP.table(ORDERS), |columns| columns.from_type::<Order>())
        .rows(&alice)
        .create()?;
    fake.when_query_match(ORDERS_OF)
        .returns_rows(|columns| columns.from_type::<Order>(), &alice)?;
    Ok(())
}

#[tokio::main(flavor = "current_thread")]
async fn main() -> BigQueryResult<()> {
// The rest of the application sees only the client
let bigquery = bigquery_db("my-gcp-project-id").await?;
let orders = orders_of(bigquery.db(), "Alice").await?;
let _ = orders;
Ok(())
}

The fake answers a query only from a rule, so the fake branch seeds a rule for every query the application sends, besides the tables it reads and writes. An application run against the fake with APP_BIGQUERY=fake cargo run panics when the fake is dropped if a call went unanswered, as a test does. An application that wants the fake out of its production builds can put the fake branch behind a feature of its own.

Limits

The fake binds its own port, so tests run in parallel, and a fake is Send + Sync, so a test can share it with the tasks it spawns. A call sees the rules registered before it arrives.

Tokio’s paused time, #[tokio::test(start_paused = true)], is not supported. A paused runtime jumps to its next timer whenever it is idle, and waiting on the fake’s loopback socket counts as idle.

The fake does not provide:

  • SQL evaluation. The DDL that .sync() sends for a rename, drop or widen reaches the query rules and does not change the table;
  • read filters. A row restriction needs a read rule, and sample_percentage and snapshot_time are served the current rows;
  • failures in the middle of a read stream, and partial acknowledgements of an append;
  • polling of running jobs, other than for fails_job and queries with a destination table;
  • CDC changes applied to the table’s rows;
  • Arrow appends in Arrow types other than the table’s own. A decimal is the exception: it is rescaled to the scale of its NUMERIC(p,s) or BIGNUMERIC(p,s) column and checked against its precision;
  • the length checks of STRING(n) and BYTES(n), default values, request limits and quotas;
  • a read that selects a field of a STRUCT column, such as .fields(["address.city"]);
  • ListJobs, DeleteJob, models, routines, row access policies, table snapshots and copies.

Full example available here.

Benchmarks

The library was compared with Google’s own clients on the same machine, region and data:

  • the official Rust crate google-cloud-bigquery;
  • Python google-cloud-bigquery with google-cloud-bigquery-storage and pyarrow;
  • the bq CLI, for query latency only.

The numbers are from 2026-10-05, on master after 0.5.0, commit 5f30c65: the full run of every scenario, a second run of the query scenarios with bq, the query run, and a rerun of the Arrow write. They depend a lot on the network, so treat them as a comparison between the clients on one connection, not as absolute figures.

Method

  • Machine: Intel Core i7-10700K, 16 threads, 64 GB RAM, Linux 7.2, CPU governor powersave. A home connection in Sweden.
  • Region: every table in europe-north2 (Stockholm), and every query sent with that location, so generated rows are computed there too.
  • Endpoints: the global defaults for every client, bigquery.googleapis.com and bigquerystorage.googleapis.com. The TCP connect to both took about 10 ms and the TLS handshake about 29 ms. A small authenticated call, datasets.get over the library’s warm gRPC channel, took about 0.1 s.
  • Data: generated once with CREATE TABLE ... AS SELECT over GENERATE_ARRAY, in a scratch dataset deleted at the end. The scan table has 1M rows and 20 columns: INT64, FLOAT64, STRING, BOOL, NUMERIC, DATE and TIMESTAMP, a STRUCT and an ARRAY, some of them nullable. It is about 215 MB as BigQuery counts it.
  • Runs: 1 warm-up and 5 measured runs for every client and scenario, timed inside the client process (outside of it for bq). The tables show the median and the range.
  • Order: one client at a time. The clients run round by round, run 1 of every client, then run 2, etc., with the order rotated each round, so drifts in the machine or the network spread over all of them.
  • Quiet machine: before each run the harness waited for a 1-minute load under 1.6 (10% of the cores), no other process above 20% of a core, no compiler or linker running and under 2 MB/s of idle network traffic. During the run it sampled the machine every 2 s and repeated a run with a bad sample. In the full run 12 attempts were repeated, all because of the browser, the desktop compositor or a snapshot daemon on the same machine. One Arrow write run stayed contended after 4 attempts, so that scenario was run again on its own, with no run repeated. The query run repeated none. Every reported run started at a 1-minute load under 0.9 and had other processes using under 0.9 cores, and the network traffic during the runs matched what the clients themselves moved.
  • Settings: the defaults of each client, query cache off everywhere. Where a client leaves a setting to the caller, the harness set it:
    • Storage Read on the official crate: 16 streams asked for (the library asks for the machine’s parallelism), LZ4 buffers, one task per stream;
    • Storage Write on the official crate: 25,000 rows per Arrow request, about 5.5 MB, so a request stays under the 10 MB limit, and 8 requests in flight, which is the library’s default window. The Arrow write of the library takes the same 25,000-row batches.

Python asks Storage Read for max_stream_count = 0 by default, which lets BigQuery decide. It got 1 stream every time, while on the 1M-row table scan the library and the official crate got 4 of the 16 they asked for.

Versions

ClientVersions
bigquery (this library)master after 0.5.0, commit 5f30c65; gcloud-sdk 0.32.4, tonic 0.14.6, arrow 60.0.0
google-cloud-bigquery0.18.0, google-cloud-bigquery-v2 1.0.0, google-cloud-gax 1.15.0, google-cloud-auth 1.17.0
PythonPython 3.14.7, google-cloud-bigquery 3.46.1, google-cloud-bigquery-storage 2.42.0, pyarrow 25.0.1, pandas 3.0.6, grpcio 1.84.0
bqBigQuery CLI 2.1.39 (Google Cloud SDK 587.0.0)

Rust code was built in release mode with rustc 1.98.1.

What each client does

The paths below were checked in each client’s source for the versions above, and the harness records which one each run actually took:

  • This library: everything over gRPC. A query goes through the v2 Query call with an Arrow result and JOB_CREATION_OPTIONAL by default (.job_creation_required() turns it off); a result that does not fit in the first response is read through Storage Read. Typed rows are decoded from Arrow with serde. Writes go to Storage Write as protobuf rows, or as Arrow for record batches.
  • google-cloud-bigquery: queries over REST (google-cloud-bigquery-v2), rows as JSON pages converted with #[derive(FromRow)]. It sends queries with JOB_CREATION_OPTIONAL by default, so a short query may run without a job. Storage Read is a raw generated client that returns Arrow IPC bytes and leaves the decoding to you; in 0.18 it is not behind any cfg flag. Storage Write takes Arrow you serialize yourself; its protobuf writer is not public.
  • Python: query_and_wait() over REST, and to_arrow() or to_dataframe() through Storage Read for large results and table scans.
  • bq: one CLI process per query.

Small query latency

The library is measured twice, in its default short query mode and with .job_creation_required(), so you can see what a job costs. Both run in the same harness run as the other clients. The tables are from the query run.

SELECT 1 AS x:

ClientMedianRange
bigquery0.117 s0.101-0.169 s
bigquery, .job_creation_required()0.198 s0.178-0.282 s
google-cloud-bigquery0.107 s0.091-0.109 s
Python0.270 s0.152-0.328 s
bq2.480 s2.390-2.524 s

1,000 generated rows:

ClientMedianRangeRows/s
bigquery0.135 s0.111-0.169 s7,411
bigquery, .job_creation_required()0.224 s0.197-0.284 s4,458
google-cloud-bigquery0.126 s0.109-0.152 s7,935
Python0.218 s0.181-0.271 s4,582
bq2.488 s2.364-2.542 s402

The library and the official crate are about the same speed here, and both are faster than Python, because of the job:

  • The official crate sets JobCreationMode::JobCreationOptional on every query (google-cloud-bigquery 0.18.0, src/query/builder.rs, line 73), BigQuery’s short query mode, where BigQuery may answer without creating a job. The library does the same by default. In these runs neither got a job back, while Python and the library’s required mode got one every time.
  • Creating the job is the whole difference: 81 ms on the constant query in the query run, and 165 ms in the full run.
  • The library was 10 ms behind the official crate in the query run and 41 ms in the full run (0.130 s against 0.089 s). To check whether that is the code, the 0.5.0 release, this commit and the official crate ran SELECT 1 interleaved in one script, 20 rounds each, every round on a quiet machine: 0.094 s, 0.096 s and 0.091 s. The cause of the larger gap in the runs above was not measured.

About 0.6 s of every bq run is the CLI starting up (bq version timed in each run); the rest was not broken down.

Large query result

200,000 generated rows of 4 columns, read as typed rows, from the query run:

ClientPathMedianRangeRows/s
bigqueryStorage Read, 1 stream1.164 s1.137-1.328 s171,805
bigquery, .job_creation_required()Storage Read, 1 stream1.210 s1.079-1.227 s165,301
google-cloud-bigqueryREST JSON pages4.387 s3.829-4.696 s45,585
Python (list(rows))REST JSON pages4.550 s4.367-5.118 s43,954

A result this large always gets a job, so the two modes are the same here. The full run gave 1.315 s, 3.779 s and 4.680 s. One run of the library in the full run took 4.550 s on the same path, the others 1.218-1.429 s.

The same result as Arrow, from the full run:

ClientPathMedianRangeRows/s
bigquery (record_batches())Storage Read, 1 stream1.175 s1.082-1.284 s170,243
Python (to_arrow())Storage Read, 1 stream2.478 s2.283-2.797 s80,698
google-cloud-bigqueryn/a: its query client returns JSON rows only

The library reads the destination table through Storage Read once the result does not fit in the first response, while the official crate and Python iterate REST pages. The official crate spent 0.018 s of its 3.8-4.7 s in FromRow, so nearly all of its time is the paging itself.

Python takes the same Storage Read path for to_arrow() and is still twice as slow. Where its extra time goes was not measured.

Table scan

The whole 1M-row table, every column, from the full run. MB/s is the table’s 215 MB divided by the median.

Typed rows:

ClientPathMedianRangeRows/sMB/s
bigquery (obj::<T>())Storage Read, 4 streams9.592 s8.622-10.090 s104,25522.4
Python (to_dataframe())Storage Read, 1 stream10.695 s9.741-11.309 s93,50220.1
google-cloud-bigquery (FromRow)SELECT * query, REST JSON pages95.211 s94.464-97.281 s10,5032.3

Arrow:

ClientPathMedianRangeRows/sMB/s
bigquery (record_batches())Storage Read, 4 streams9.475 s9.364-9.516 s105,53922.7
Python (to_arrow())Storage Read, 1 stream9.474 s9.254-9.624 s105,55622.7
google-cloud-bigquery (raw client + arrow-ipc)Storage Read, 4 streams10.175 s9.680-10.344 s98,28321.1

The official crate has no typed Storage Read, so its typed path is a query, and a SELECT * query bills the table: 215 MB per run. Its FromRow took about 1.09 s of the 95 s.

Raw Arrow scans are about the same speed in all three clients, and the library has no advantage there. The official crate was 7% slower in this run and 1% faster in the previous one, on 2026-10-04, so the difference is within the variation between runs. Every client moved about 150 MB per scan on the network interface at the same 14-16 MB/s.

Python’s to_dataframe() builds a pandas frame with typed columns, not row objects, so it is the closest Python has to typed rows rather than the same thing.

Storage Write

1M rows of the scan table’s shape into the default stream, per run.

From Rust structures, from the full run:

ClientFormatMedianRangeRows/sMB sent
bigquery (insert().objects())protobuf from serde33.976 s33.089-37.866 s29,432180.5
google-cloud-bigqueryArrow built from the same rows35.433 s33.568-41.872 s28,222221.4
Pythonn/a: its writer takes requests you build yourself, protobuf rows with a hand-made descriptor

Both times include turning the Rust rows into the wire format. The library sent 18% fewer bytes and was slightly faster, but the ranges overlap. On the network interface the library ran at about 5.8 MB/s and the official crate at about 6.7 MB/s, so the upload of this connection is not the only limit. Whether the smaller requests explain the difference was not measured.

From Arrow record batches, from the rerun:

ClientFormatMedianRangeRows/sMB sent
bigquery (insert().record_batches())Arrow37.031 s34.438-37.606 s27,004221.4
google-cloud-bigqueryArrow35.223 s33.516-38.493 s28,390221.4

Both clients get the same 40 batches of 25,000 rows, built before the timer starts, so the times are the IPC encoding and the appends. Both sent the same bytes, 40 requests each. The library was 5% slower here and 6% slower in the full run (38.958 s against 36.711 s), but this is within the noise. Two later sessions on 0.6.0 gave 37.907 s against 35.840 s (10 runs each) and 37.446 s against 39.607 s (6 runs each), so the order flips. Over all 26 and 25 runs the medians are 37.58 s and 36.36 s, and a rank test does not tell the two clients apart (z = 1.07).

Both writes are limited by the same thing: how fast BigQuery takes in one AppendRows stream. From here it is about 6 MB/s, one 5.5 MB request acknowledged every 0.9 s or so, with the same cadence in both clients. The HTTP/2 window is 1 MiB and TCP had only about 64 KB in flight, so neither the client nor the upload of the connection is the limit. Two inserts in parallel, each with half of the batches, wrote the same 1M rows in 20.1-21.6 s over three runs. If you need more throughput, split the batches over several writers.

Decode cost

The scan table’s 1M rows already in memory as Arrow batches, decoded into structs on one thread, from the full run:

ClientMedianRangeRows/s
bigquery0.658 s0.639-0.670 s1,519,533
google-cloud-bigqueryn/a

The official crate has no Arrow to struct decoder. Its FromRow converts the JSON rows of a live query, and Row has no public constructor, so it cannot be fed the same batches. Inside the REST scan above its conversion took about 1.09 s per 1M rows, but on already parsed JSON values, so the two numbers are not comparable.

In the library the decode runs on each read stream’s task, next to the network reads, and typed and Arrow scans take almost the same time.

Cost

The full run billed 1.29 GB of queries, all of them the official crate’s SELECT * scans. The query run and the Arrow write rerun billed 0 bytes. Every other query read generated rows and billed 0 bytes. The scans went through the Storage Read free tier, and the writes ingested about 215 MB per 1M rows under the Storage Write free tier.

Reproducing

The harness is in bench-compare, an unpublished crate with the Rust contenders and a uv project for Python. You need application default credentials and optionally the bq CLI, found in PATH or through BQ_BIN:

bench-compare/run.sh --project my-project

It creates the scratch dataset in europe-north2 (--location changes it), runs everything and deletes the dataset, also on failure. The raw results land in bench-compare/results/<run label>/results.json, with the machine state of every run. A full run takes about 45 minutes and costs a few cents. --only query_const,query_1k,query_200k_rows runs just the query scenarios, --only write_arrow just the Arrow write. The summary of the runs above is in bench-compare/results-2026-10-05-europe-north2.json, and the previous ones, on the code before 0.5.0, in bench-compare/results-2026-10-04-europe-north2.json.