Use Spark with Unity Catalog

Pick matching Spark, Delta, and Unity Catalog connector versions, register a Unity Catalog catalog in a SparkSession, and choose how Spark authenticates to the server.

Apache Spark connects to a Unity Catalog server through the Unity Catalog Spark connector, so that Spark SQL can name catalog.schema.table and read and write the tables it holds. The connector registers one Unity Catalog catalog as a Spark catalog. Delta Spark handles the table format, and from Delta 4.3.0 it reads and commits catalog-managed tables through the server's Delta API.

Choose matching versions

The connector is built per Spark minor version. Its artifact name carries the Spark version, and it needs a Delta Spark build for the same Spark version:

With a Delta Spark release older than 4.3.0, the connector quietly skips the Delta API, and catalog-managed tables don't work.

Connector versions by release

Each Unity Catalog release publishes one connector artifact per Spark minor version, named unitycatalog-spark_<spark>_2.13. All of them are built for Scala 2.13. The release notes name a Delta Spark release that the connector works with. The connector's own tests build against the version in the last column, so where the two differ, the release notes name the newer release.

Releases before 0.5.0 shipped a single unitycatalog-spark_2.13 artifact.

Configure the session

Register the catalog under a Spark catalog name and point it at the server. The Spark name is your choice. Using the Unity Catalog catalog's own name, here unity, keeps three-part names the same in Spark and in the catalog:

from pyspark.sql import SparkSession

UC_SPARK = "io.unitycatalog:unitycatalog-spark_4.1_2.13:0.7.0"
DELTA_SPARK = "io.delta:delta-spark_4.1_2.13:4.3.1"

spark = (
    SparkSession.builder.appName("uc-spark")
    .config("spark.jars.packages", f"{UC_SPARK},{DELTA_SPARK}")
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config(
        "spark.sql.catalog.spark_catalog",
        "org.apache.spark.sql.delta.catalog.DeltaCatalog",
    )
    .config("spark.sql.catalog.unity", "io.unitycatalog.spark.UCSingleCatalog")
    .config("spark.sql.catalog.unity.uri", "http://localhost:8080")
    .config("spark.sql.catalog.unity.auth.type", "static")
    .config("spark.sql.catalog.unity.auth.token", "not-used")
    .config("spark.sql.defaultCatalog", "unity")
    .getOrCreate()
)

Each setting has a job:

The same keys work as --conf options for spark-submit, spark-sql, and pyspark, with --packages in place of spark.jars.packages.

Authenticate to the server

The connector always needs an authentication type, even when the server has authorization disabled. Without one, the first query fails with Required configuration key 'type' is missing or empty.

• Static token. Set auth.type to static and auth.token to a Unity Catalog access token. The local server above accepts any value. Older configurations set token instead, which still works. Set only one of the two.
• OAuth client credentials. Set auth.type to oauth, plus auth.oauth.uri, auth.oauth.clientId, and auth.oauth.clientSecret. auth.oauth.scope defaults to all-apis. The connector fetches and renews the token itself. This page doesn't test the OAuth flow.

Keep secrets out of source code. Read the token or client secret from the environment or a secret store when you build the session.

Check the connection

List the catalog's schemas and the tables in one of them:

spark.sql("SHOW SCHEMAS").show()
spark.sql("SHOW TABLES IN default").show()

The sample catalog's default schema lists tables such as numbers and marksheet. These sample tables live inside the server's container, so this local Spark can list them but can't read their files. Create your own tables under a storage root that both sides can reach, as in Create and update a catalog-managed Delta table.

Tune storage credentials

When Spark reads or writes a table, the connector asks the server for short-lived storage credentials for that table's location. It applies them per table. Three catalog options control this, and the defaults suit most setups:

Set them like the others, for example spark.sql.catalog.unity.renewCredential.enabled. For how the server decides what each credential may access, see Credential vending.

Option names and defaults come from the connector source, OptionsUtil.java and AuthConfigs.java.

Next steps

• Create and update a catalog-managed Delta table puts this session to work.
• Use DuckDB with Unity Catalog and Query Unity Catalog tables from Python DataFrame libraries cover other engines.