10 releases (5 breaking)

0.6.0 Feb 25, 2025
0.5.0 Oct 10, 2024
0.4.0 Sep 19, 2024
0.3.2 Jun 24, 2024
0.1.2 Mar 4, 2024

#255 in Database interfaces

Download history 937/week @ 2024-11-20 670/week @ 2024-11-27 550/week @ 2024-12-04 502/week @ 2024-12-11 628/week @ 2024-12-18 419/week @ 2024-12-25 636/week @ 2025-01-01 592/week @ 2025-01-08 531/week @ 2025-01-15 600/week @ 2025-01-22 330/week @ 2025-01-29 745/week @ 2025-02-05 651/week @ 2025-02-12 705/week @ 2025-02-19 920/week @ 2025-02-26 1019/week @ 2025-03-05

3,414 downloads per month

Apache-2.0

115KB
2K SLoC

Rust 1.5K SLoC // 0.1% comments JavaScript 525 SLoC // 0.2% comments

JavaScript UDF for Apache Arrow

Crate Docs

Usage

Add the following lines to your Cargo.toml:

[dependencies]
arrow-udf-js = "0.6"

Create a Runtime and define your JS functions in string form. Note that the function must be exported and its name must match the one you pass to add_function.

use arrow_udf_js::{FunctionOptions, Runtime};

let mut runtime = Runtime::new().await?;
runtime
    .add_function(
        "gcd",
        arrow_schema::DataType::Int32,
        r#"
        export function gcd(a, b) {
            while (b != 0) {
                let t = b;
                b = a % b;
                a = t;
            }
            return a;
        }
        "#,
        FunctionOptions::default().return_null_on_null_input(),
    )
    .await?;

You can then call the JS function on a RecordBatch:

let input: RecordBatch = ...;
let output: RecordBatch = runtime.call("gcd", &input).await?;

If you print the input and output batch, it will be like this:

 input     output
+----+----+-----+
| a  | b  | gcd |
+----+----+-----+
| 15 | 25 | 5   |
|    | 1  |     |
+----+----+-----+

For set-returning functions (or so-called table functions), define the function as a generator:

use arrow_udf_js::{FunctionOptions, Runtime};

let mut runtime = Runtime::new().await?;
runtime
    .add_function(
        "range",
        arrow_schema::DataType::Int32,
        r#"
        export function* range(n) {
            for (let i = 0; i < n; i++) {
                yield i;
            }
        }
        "#,
        FunctionOptions::default().return_null_on_null_input(),
    )
    .await?;

You can then call the table function via call_table_function:

let chunk_size = 1024;
let input: RecordBatch = ...;
let outputs = runtime.call_table_function("range", &input, chunk_size)?;
while let Some(output) = outputs.next().await {
    // do something with the output
}

If you print the output batch, it will be like this:

+-----+-------+
| row | range |
+-----+-------+
| 0   | 0     |
| 2   | 0     |
| 2   | 1     |
| 2   | 2     |
+-----+-------+

The JS code will be run in an embedded QuickJS interpreter.

See the example for more details.

Type Mapping

The following table shows the type mapping between Arrow and JavaScript:

Arrow Type JS Type
Null null
Boolean boolean
Int8 number
Int16 number
Int32 number
Int64 number
UInt8 number
UInt16 number
UInt32 number
UInt64 number
Float32 number
Float64 number
String string
LargeString string
Date32 Date
Timestamp Date
Decimal128 BigDecimal
Decimal256 BigDecimal
Binary Uint8Array
LargeBinary Uint8Array
List(Int8) Int8Array
List(Int16) Int16Array
List(Int32) Int32Array
List(Int64) BigInt64Array
List(UInt8) Uint8Array
List(UInt16) Uint16Array
List(UInt32) Uint32Array
List(UInt64) BigUint64Array
List(Float32) Float32Array
List(Float64) Float64Array
List(others) Array
Struct object

This crate also supports the following Arrow extension types:

Extension Type Physical Type ARROW:extension:name JS Type
JSON String, Binary, LargeBinary arrowudf.json any (parsed by JSON.parse(string))
Decimal String arrowudf.decimal BigDecimal

Async Functions and Fetch API

An async function is a JavaScript function that returns a promise. If the function involves IO operations, it's usually more efficient to use async functions.

We provide a Fetch API to allow making HTTP requests from JavaScript UDFs in async way. To use it, you need to enable it in the Runtime:

runtime.enable_fetch();

To enable async functions, you need to set async_mode() when adding the function. Then you can use the async fetch() function in your JavaScript code.

runtime
    .add_function(
        "echo",
        DataType::Utf8View,
        r#"
export async function my_fetch_udf(id) {
    const response = await fetch("https://api.example.com/" + id);
    const data = await response.json();
    return data.value;
}
"#,
        FunctionOptions::default().return_null_on_null_input().async_mode(),
    )
    .await
    .unwrap();

See the README of the fetch module for more details.

Batched Function

When a function is batched, it will be called once for all rows in the input RecordBatch. The input arguments will be an array of values.

runtime
    .add_function(
        "echo",
        DataType::Utf8View,
        r#"
export function echo(vals) {
    return vals.map(v => v + "!")
}
"#,
        FunctionOptions::default().return_null_on_null_input().batched(),
    )
    .await
    .unwrap();

Currently, table functions can not be batched.

Dependencies

~17–29MB
~496K SLoC