macropipe.lib

Macropipe built-in recipe functions.

Classes

Functions

All built-in recipe functions.

Module Contents

class macropipe.lib.Functions(lf: polars.LazyFrame)

All built-in recipe functions.

apply(pipeline: macropipe.core.MacroPipe) → polars.LazyFrame

Convert transformation recipes to Polars expressions and apply to structured pipeline.

head(n: str) → polars.LazyFrame

Get the first n rows.

Input:  All records.
Recipe: "head:30"
Output: Filtered records.

tail(n: str) → polars.LazyFrame

Get the last n rows.

Input:  All records.
Recipe: "tail:30"
Output: Filtered records.

TODO: Improve with automatic frame method mapping, see head.

first() → polars.LazyFrame

Get the first value.

Input:  All records.
Recipe: "first"
Output: Filtered records.

TODO: Improve with automatic frame method mapping, see head.

last() → polars.LazyFrame

Get the last value.

Input:  All records.
Recipe: "last"
Output: Filtered records.

TODO: Improve with automatic frame method mapping, see head.

cast(column_names: str | List[str], dtype: str) → polars.LazyFrame

Cast multiple columns by type name.

Input:  {"float": 42.42, "int": 42, "str": "42"}
Recipe: "cast:float,int,str:float"
Output: {"float": 42.42, "int": 42.0, str: 42.0}

select(column_names: str | List[str]) → polars.LazyFrame

Select multiple columns by name.

Input:  {"ts": 1754784000000, "data": "Hotzenplotz", "foo": "42"}
Recipe: "select:ts,data"
Output: {"ts": 1754784000000, "data": "Hotzenplotz"}

drop(column_names: str | List[str]) → polars.LazyFrame

Drop multiple columns by name.

Input:  {"ts": 1754784000000, "data": "Hotzenplotz", "foo": "42"}
Recipe: "drop:foo"
Output: {"ts": 1754784000000, "data": "Hotzenplotz"}

rename(source_column: str, target_column: str) → polars.LazyFrame

Rename a single column.

Input:  {"_id": "01kp0w38"}
Recipe: "rename:_id:__id"
Output: {"__id": "01kp0w38"}

concat(column_names: str | List[str], separator: str, target_column: str, options: str | None = None) → polars.LazyFrame

Combine multiple columns by joining them. Optionally drop the original columns.

Input:  {"firstname": "Räuber", "lastname": "Hotzenplotz"}
Recipe: "concat:firstname,lastname: :combined:drop=true"
Output: {"name": "Räuber Hotzenplotz"}

format(f_string: str, column_names: str | List[str], target_column: str, options: str | None = None) → polars.LazyFrame

Create column from existing columns and format expressions, optionally dropping origin.

Input:  {"a": ["a", "b", "c"], "b": [1, 2, 3]}
Recipe: "format:foo_{}_bar_{}:a,b:value:drop=true"
Output: {"value": ["foo_a_bar_1", "foo_b_bar_2", "foo_c_bar_3"]}

filter(clause: str) → polars.LazyFrame

Transform result by filtering records using SQL WHERE expression clauses.

Input:  [{"ts": 1754784000000, "data": "foo"}, {"ts": 1754785000000, "data": "bar"}]
Recipe: "filter:ts < 1754785000000"
Output: [{"ts": 1754784000000, "data": "foo"}]

• https://docs.pola.rs/user-guide/sql/intro/

• https://docs.pola.rs/user-guide/transformations/time-series/filter/

• https://docs.pola.rs/api/python/stable/reference/expressions/api/polars.sql_expr.html

scale(column_name: str, factor: float) → polars.LazyFrame

Scale value in a single column by multiplying by a factor.

Input:  {"value": 4242}
Recipe: "scale:value:0.01"
Output: {"value": 42.42}

iso_to_unixtime(column_name: str) → polars.LazyFrame

Convert ISO 8601 / RFC 3339 date & time format to epoch timestamp (Unix time).

Input:  {"value": "2026-03-03T12:12:12"}
Recipe: "iso_to_unixtime:value"
Output: {"value": 1772539932}

unixtime_to_iso(column_name: str) → polars.LazyFrame

Convert epoch timestamp (Unix time) to ISO 8601 / RFC 3339 date & time format.

Input:  {"value": 1772539932}
Recipe: "unixtime_to_iso:value"
Output: {"value": "2026-03-03T12:12:12.000000"}

json_array_to_wkt_point(col_name: str) → polars.LazyFrame

Convert coordinates list [long, lat] in JSON format to WKT POINT (long lat) format.

Input:  {"coordinates": "[9.757, 47.389]"}
Recipe: "json_array_to_wkt_point:coordinates"
Output: {"coordinates": "POINT ( 9.757 47.389 )"}

python_to_json(col_name: str) → polars.LazyFrame

Convert Python-encoded dictionary into pure JSON.

Input:  {"data": "{'temperature': 42.42}"}
Recipe: "python_to_json:data"
Output: {"data": '{"temperature": 42.42}'}

columns_to_json_array(source_columns: str | List[str], target_column: str, options: str | None = None) → polars.LazyFrame

Combine individual columns into JSON-encoded array. Optionally drop the original columns.

Input:  {"longitude": 9.757, "latitude": 47.389}"}
Recipe: "columns_to_json_array:longitude,latitude:coordinates:drop=true"
Output: {"coordinates": "[9.757 47.389]"}

json_fields_to_columns(source_column: str, extract_columns: str | List[str], dtype: str, options: str | None = None) → polars.LazyFrame

Extract JSON fields from single column into individual columns, optionally dropping origin.

Input:  {"data": '{"longitude": 9.757, "latitude": 47.389, "more": "anything"}'}
Recipe: "json_fields_to_columns:data:longitude,latitude:float:drop=true"
Output: {"longitude": 9.757, "latitude": 47.389}

TODO: An advanced version could provide extracting

individual columns with individual dtypes.

json_fields_to_wkt_point(source_column: str, longitude_field: str, latitude_field: str, target_column: str, options: str | None = None) → polars.LazyFrame

Extract longitude and latitude fields from JSON and encode them into WKT POINT format. Optionally drop the original columns.

Input:  {"data": '{"longitude": 9.757, "latitude": 47.389}'}
Recipe: "json_fields_to_wkt_point:data:longitude:latitude:coordinates:drop=true"
Output: {"coordinates": "POINT ( 9.757 47.389 )"}