macropipe.core¶
A small macro language on top of Polars.
This pipeline wrapper uses macro-like commands / a text-based expression language, that uses Polars pipes to apply compiled UDFs to a LazyFrame in a structured way.
https://macropipe.readthedocs.io/
Classes¶
A miniature transformation engine based on Polars. |
Module Contents¶
- class macropipe.core.MacroPipe¶
A miniature transformation engine based on Polars.
- expressions: List[str]¶
- registry: ClassVar[macropipe.registry.Registry]¶
- classmethod from_recipes(*recipes: str) MacroPipe¶
Create MacroPipe from list of recipes (text-based macro commands).
- resolve_function(name: str, lf: polars.LazyFrame | None = None) Callable¶
Resolve macro function by name, either from builtins or from user-registered function.
- static decode_expression(expression: str) Tuple[str, List[str]]¶
Tokenize the expression and convert it to macro invocation descriptor.
In contrast to the Python-based API, Macropipe invents a simple text-based macro language that compiles to Polars LazyFrame transformations: Function name and positional arguments are separated by colons
:, that’s it. Use\:to represent a literal colon inside an argument.<function>:<arg1>:<arg2>:<arg3>Todo
The expression language is currently pretty flat. It can certainly be improved in future iterations. Any suggestions are very much welcome.
- apply(lf: polars.LazyFrame) polars.LazyFrame¶
Convert transformation recipes to Polars expressions and apply to structured pipeline.