2.11.1
Spark
Directory
Classes
| Class | Description |
|---|---|
ParquetToSparkDecoder |
|
Spark |
Use this to configure a SparkContext for a your task. |
SparkToParquetEncoder |
Methods
| Method | Description |
|---|---|
register_spark_df_transformers() |
Register Spark DataFrame encoders and decoders with the DataFrameTransformerEngine. |
Methods
register_spark_df_transformers()
def register_spark_df_transformers()Register Spark DataFrame encoders and decoders with the DataFrameTransformerEngine.
This function is called automatically via the flyte.plugins.types entry point when flyte.init() is called with load_plugin_type_transformers=True (the default).