Query

yaml

type: "io.kestra.plugin.jdbc.duckdb.Query"

Query a DuckDb Database.

Examples

Execute a query that reads a csv, and outputs another csv.

yaml

id: query_duckdb
namespace: company.team

tasks:
  - id: http_download
    type: io.kestra.plugin.core.http.Download
    uri: "https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv"

  - id: query
    type: io.kestra.plugin.jdbc.duckdb.Query
    url: 'jdbc:duckdb:'
    timeZoneId: Europe/Paris
    sql: |-
      CREATE TABLE new_tbl AS SELECT * FROM read_csv_auto('{{ workingDir }}/in.csv', header=True);

      COPY (SELECT order_id, customer_name FROM new_tbl) TO '{{ outputFiles.out }}' (HEADER, DELIMITER ',');
    inputFiles:
      in.csv: "{{ outputs.http_download.uri }}"
    outputFiles:
       - out

Execute a query that reads from an existing database file using a URL.

yaml

id: query_duckdb
namespace: company.team

tasks:
  - id: query1
    type: io.kestra.plugin.jdbc.duckdb.Query
    url: jdbc:duckdb:/{{ vars.dbfile }}
    sql: SELECT * FROM table_name;
    fetchType: STORE

  - id: query2
    type: io.kestra.plugin.jdbc.duckdb.Query
    url: jdbc:duckdb:/temp/folder/duck.db
    sql: SELECT * FROM table_name;
    fetchType: STORE

Execute a query that reads from an existing database file using the databaseFile variable.

yaml

id: query_duckdb
namespace: company.team

tasks:
  - id: query1
    type: io.kestra.plugin.jdbc.duckdb.Query
    url: jdbc:duckdb:
    databaseFile: {{ vars.dbfile }}
    sql: SELECT * FROM table_name;
    fetchType: STORE

  - id: query2
    type: io.kestra.plugin.jdbc.duckdb.Query
    url: jdbc:duckdb:
    databaseFile: /temp/folder/duck.db
    sql: SELECT * FROM table_name;
    fetchType: STORE

Properties

`fetchType`

Type: string
Dynamic: ❌
Required: ✔️
Default: NONE
Possible Values:
- STORE
- FETCH
- FETCH_ONE
- NONE

The way you want to store data.

FETCH_ONE - output the first row. FETCH - output all rows as output variable. STORE - store all rows to a file. NONE - do nothing.

`url`

Type: string
Dynamic: ✔️
Required: ✔️
Default: jdbc:duckdb:

The JDBC URL to connect to the database.

The default value, jdbc:duckdb:, will use a local in-memory database. Set this property when connecting to a persisted database instance, for example jdbc:duckdb:md:my_database?motherduck_token=<my_token> to connect to MotherDuck.

`autoCommit`

Type: boolean
Dynamic: ❌
Required: ❌
Default: true

Whether autocommit is enabled.

Sets this connection's auto-commit mode to the given state. If a connection is in auto-commit mode, then all its SQL statements will be executed and committed as individual transactions. Otherwise, its SQL statements are grouped into transactions that are terminated by a call to either the method commit or the method rollback. By default, new connections are in auto-commit mode except when you are using store property in which case the auto-commit will be disabled.

`fetch`

⚠ Deprecated

Type: boolean
Dynamic: ❌
Required: ❌
Default: false

DEPRECATED, please use fetchType: FETCH instead. Whether to fetch the data from the query result to the task output. This parameter is evaluated after fetchOne and store.

`fetchOne`

⚠ Deprecated

Type: boolean
Dynamic: ❌
Required: ❌
Default: false

DEPRECATED, please use fetchType: FETCH_ONE instead. Whether to fetch only one data row from the query result to the task output. This parameter is evaluated before store and fetch.

`fetchSize`

Type: integer
Dynamic: ❌
Required: ❌
Default: 10000

Number of rows that should be fetched.

Gives the JDBC driver a hint as to the number of rows that should be fetched from the database when more rows are needed for this ResultSet object. If the fetch size specified is zero, the JDBC driver ignores the value and is free to make its own best guess as to what the fetch size should be. Ignored if autoCommit is false.

`inputFiles`

Type: object
SubType: string
Dynamic: ✔️
Required: ❌

Input files to be loaded from DuckDb.

Describe a files map that will be written and usable by DuckDb. You can reach files using a workingDir variable, example: SELECT * FROM read_csv_auto('{{ workingDir }}/myfile.csv');

`outputFiles`

Type: array
SubType: string
Dynamic: ❌
Required: ❌

Output file list that will be uploaded to internal storage.

List of keys that will generate temporary files. On the SQL query, you can just use a variable named outputFiles.key for the corresponding file. If you add a file with ["first"], you can use the special vars COPY tbl TO '{{ outputFiles.first }}' (HEADER, DELIMITER ','); and use this file in others tasks using {{ outputs.taskId.outputFiles.first }}.

`password`

Type: string
Dynamic: ✔️
Required: ❌

The database user's password.

`sql`

Type: string
Dynamic: ✔️
Required: ❌

The SQL query to run.

`store`

⚠ Deprecated

Type: boolean
Dynamic: ❌
Required: ❌
Default: false

DEPRECATED, please use fetchType: FETCH_STORE instead. Whether to fetch data row(s) from the query result to a file in internal storage. File will be saved as Amazon Ion (text format). See Amazon Ion documentation This parameter is evaluated after fetchOne but before fetch.