# Enabling GCP resources > This bundle contains all pages in the Enabling GCP resources section. > Source: https://www.union.ai/docs/v2/union/deployment/byoc/enabling-gcp-resources/ === PAGE: https://www.union.ai/docs/v2/union/deployment/byoc/enabling-gcp-resources === # Enabling GCP resources > **📝 Note** > > An LLM-optimized bundle of this entire section is available at [`section.md`](https://www.union.ai/docs/v2/union/deployment/byoc/section.md). > This single file contains all pages in this section, optimized for AI coding agent context. Components of your Union.ai data plane will need to connect to and communicate with other resources in your cloud environment such as **BYOC deployment > Enabling GCP resources > Enabling Google Cloud Storage**, **BYOC deployment > Enabling GCP resources > Enabling Google Artifact Registry**, **BYOC deployment > Enabling GCP resources > Enabling BigQuery**, and so forth. > [!NOTE] Secret management > We strongly recommend using the [Union.ai secrets manager](https://www.union.ai/docs/v2/union/user-guide/task-configuration/secrets/page.md) to manage secrets rather than Google Secret Manager. If your organization must use Google Secret Manager, however, see **BYOC deployment > Enabling GCP resources > Enabling Google Secret Manager**. As much as possible, access to the resources you need will be pre-configured by the Union.ai team when they set up your data plane. For example, if you want your task code to have access to a specific Cloud Storage bucket or BigQuery, this can be pre-configured. **You just have to inform the team of your specific requirements before the setup process begins**. As your projects evolve, your needs may change. You can always contact the Union.ai team for help enabling additional resources as required. **There are also some cases where you may want to configure things on your own.** **Below we give a general overview of these self-configuration options.** **The sub-pages of this section give examples for specific resources.** ## Types of access Broadly speaking, there are two categories of access that you are likely to have to deal with: * **Infrastructure access**: Enabling access to a resource for your data plane infrastructure. The most common case occurs when you are using Artifact Registry for your task container images and it resides in a project other than the one containing your data plane. In that case, some configuration is required to enable the Union.ai operator on your data plane to pull images from the registry when registering your workflows and tasks. **If you are using an Artifact Registry instance within the same project as your data plane, then access is enabled by default and no further configuration is needed.** * **Task code access**: Enabling access to a resource for your task code. For example, your task code might need to access Cloud Storage or Secret Manager at runtime. This involves granting permission to a Google Service Account (GSA) that is attached to the Kubernetes cluster within which your task code runs. ## Infrastructure-level access The only infrastructure-level access issue you are likely to encounter is around access to an Artifact Registry _in a GCP project other than the one in which your data plane resides_. **If your task container images are stored in an Artifact Registry in the same GCP project as your data plane, then access is already enabled. You do not have to do anything.** If your task container images reside in an Artifact Registry instance in **another GCP project** you will need to configure that instance to allow access from your data plane. See Enabling Artifact Registry for details. ## Task code access When your task code runs, it executes within a pod in the Kubernetes cluster in your data plane. To enable your task code to access cloud resources you must grant the appropriate permissions to the Google Service Account (GSA) attached to the Kubernetes cluster. There are two main options for setting this up: * **Domain-scoped access**: With this arrangement, you define the permissions you want to grant to your task code, and those permissions are applied only to a specific domain. * **Global access**: With this arrangement, you define the permissions you want to grant to your task code, and those permissions are then applied to code in all your projects and domains. > [!NOTE] GCP only supports scoping by domain > In AWS-based data planes, scoping by both project _and_ domain is supported. > However, due to intrinsic architectural constraints, GCP-based data planes only support scoping by domain. Global access is recommended for most use cases since it is simpler, but if you have a compelling reason to restrict access, then the project-domain-scoped access is available, at the cost of some additional complexity in setup. > [!NOTE] Relationship with RBAC > The permissions being discussed here are attached to a domain. > This is independent of the permissions granted to users and machine applications through Union.ai's role-based access control (see [User management](https://www.union.ai/docs/v2/union/user-guide/user-management/page.md)). > But, the two types of permissions are related. > > For example, for a user (or machine application) to have read access to a Cloud Storage bucket, two things are required: > > * The user (or machine application) must have **execute** permission for the project and domain where the code that does the reading resides. > * The domain must have read permission for the Cloud Storage bucket. ## Domain-scoped access **Because of the way that GCP works internally, domain-scoped access can only be configured by the Union.ai team.** Please work directly with the Union.ai team if you have requirements that involve domain-scoped access to cloud resources. If you need to add or change domain-scoped access after your data plane has been set up, you should also contact the team. ## Globally-scoped access You can manage the configuration of globally-scoped access to GCP resources yourself without involving the Union.ai team. In a GCP-based Union.ai data plane, globally-scoped access to resources is mediated by a single Google Service Account (GSA) that is configured as part of the data plane setup. We refer to it as ``. `` is bound to all the pods in your data plane's Kubernetes cluster that run your Flyte code. To enable access to a resource in GCP you grant ``access to that resource and assign it a role that includes the permissions that you want your code to have. > [!NOTE] `` > Here we refer to the default global-access GSA as``because the precise name differs across installations. > This GSA is identified by name and email of the following form: > > * Name: `-userflyterol-` > * Email: `-userflyterol-@-gcp-dataplane.iam.gserviceaccount.com` > [!NOTE] Google Service Account (GSA) > We use the term Google Service Account (GSA) to refer to the accounts that are managed in the GCP console under **IAM & Admin > Service Accounts**. > This is to distinguish them from Kubernetes Service Accounts (KSAs). > KSAs are a distinct type of service account managed _within_ the Kubernetes cluster. You will not normally encounter these at the data plane level. ## Find the actual name of `` In this section we refer to the default global-access GSA as``because the precise name differs across installations. The actual name and email of this GSA have the following forms: * Name: `-userflyterol-` * Email: `-userflyterol-@-gcp-dataplane.iam.gserviceaccount.com` **You will need to have the email identifier of this role on hand when you enable access to resources for your task code.** To find the actual name of this GSA do the following: * In the GCP data plane project, go to **IAM & Admin > Service accounts**. * In the list of service account, find the one whose name and email match the pattern above. For example: ![](../../../_static/images/user-guide/integrations/enabling-gcp-resources/user-flyte-gsa.png) * Copy this name to document in an editor. You will need it later to configure each specific resource. === PAGE: https://www.union.ai/docs/v2/union/deployment/byoc/enabling-gcp-resources/enabling-google-cloud-storage === # Enabling Google Cloud Storage For Union.ai customers whose data plane is in GCP, we walk through setting up access to your own Google Cloud Storage bucket. > [!NOTE] Google Cloud Storage in the Union.ai environment > Your data plane is set up with a Kubernetes cluster and other resources. > Among these are a number of Google Cloud Storage (GCS) buckets used internally by the Union.ai operator running in the cluster (see [Platform architecture](https://www.union.ai/docs/v2/union/deployment/byoc/platform-architecture)) to store things like workflow metadata. > > **These are not the GCS buckets we are talking about in this section.** > > **We are discussing the case where you have **_**your own GCS bucket**_** that you set up to store input and output data used by your workflows.** ## Grant `` access to the bucket To enable access to a GCS bucket you have to add the `` Google Service Account as a principal to that bucket and assign it a role that includes the permissions that you want your code to have. * Find the actual name and email of the `` in your Union.ai data plane GCP project (See [Find the actual name of ``](_index#find-the-actual-name-of-userflytegsa)) * Go to **Cloud Storage > Buckets** and select the bucket to which you want to grant access. * In the **Bucket details** view select the **Permissions** tab and then select **GRANT ACCESS**: ![](../../../_static/images/user-guide/integrations/enabling-gcp-resources/enabling-google-cloud-storage/bucket-details.png) * In the **Grant access** panel: * Under **Add principals**, paste the actual name (in email form) of the `` into the **New principals** field. * Under **Assign roles** add as many roles as you need. In the example below we add the roles enabling reading and writing: **Storage Object Viewer** and **Storage Object Creator**. ![](../../../_static/images/user-guide/integrations/enabling-gcp-resources/enabling-google-cloud-storage/grant-access-to-bucket.png) * Click **SAVE**. Your bucket should now be **globally accessible** to task code in all Flyte projects and domains in your Union.ai organization. > [!NOTE] Domain-scoped permissions are not self-service > If you want to assign permissions in a more fine-grained way, per project and/or domain, you need to contact the Union.ai team. > See [Domain-scoped access](_index#domain-scoped-access). === PAGE: https://www.union.ai/docs/v2/union/deployment/byoc/enabling-gcp-resources/enabling-google-artifact-registry === # Enabling Google Artifact Registry ## Access to Artifact Registry in the same project is enabled by default When registering tasks and workflows, the Union.ai infrastructure in your data plane must have access to the container registry that holds the task container images you will be using. If your data plane is on GCP then you may want to use Google Artifact Registry (GAR) to store these images. **In most cases, you will be using a GAR repository in the same GCP project as your data plane.** **If this is the case, then you do not need to configure anything.** **Access to GAR in the same project is enabled by default.** ## Enabling cross-project access to Artifact Registry If you want to store your task container images in a GAR repository in a GCP project _other than the one that holds your data plane_, you must enable the node pool of your data plane to access that GAR. This is the infrastructure-level access that we discussed [earlier](_index#infrastructure-level-access). It is mediated by the a specific Google Service Account (GSA) which we will refer to here as `` (recall that this is in contrast to the task code access, which is mediated by a different default GSA, ``). > [!NOTE] `` > Here we refer to the default global-access GSA as``because the precise name differs across installations. > This GSA is identified by name and email of the following form: > > * Name: `-flyteworker-` > * Email: `-flyteworker-@-gcp-dataplane.iam.gserviceaccount.com` To enable access to the GAR repository in the other account, do the following: * In your data plane GCP project, go to **IAM > Service Accounts**. Find the GSA `` and copy its email. We will call this ``. * In the other GCP project account (the one that contains the GAR instance), go to **Artifact Registry > Repositories**. * Find the GAR repository you want to enable and select the checkbox beside it. * Under **Permissions** in the side panel, select **Add Principal**. * Specify the `` as a **Principal** and assign (at least) the role **Artifact Registry Reader**. * Select **Save**. Your Union.ai data plane infrastructure should now be able to pull images from the GAR repository. === PAGE: https://www.union.ai/docs/v2/union/deployment/byoc/enabling-gcp-resources/enabling-google-secret-manager === # Enabling Google Secret Manager > [!NOTE] > This documentation exists for customers who must use Google Secret Manager for organizational reasons. For everyone else, we strongly recommend using the > [Union.ai secrets manager](https://www.union.ai/docs/v2/union/user-guide/task-configuration/secrets) to manage secrets rather than Google Secret Manager. Access to a secret stored in Secret Manager in the same GCP project as the data plane is enabled by default. All you need to do is: * Create your secrets in Secret Manager. * Retrieve your secrets from within your task code. To access a secret stored in Secret Manager in a GCP project _other than the one that holds your data plane_ requires one additional step: Granting the `` (see **BYOC deployment > Enabling GCP resources > Enabling Google Secret Manager**) access to top the secret in the other projects. ## Create your secrets Create your secrets in **Secret Manager** (see the [Secret Manager documentation](https://cloud.google.com/secret-manager/docs) for details): * Go to **Security > Secret Manager**. * Select **CREATE SECRET** at the top of the page. * Fill in the **Name**, **Value,** and (optionally) the other parameters. * Select **CREATE SECRET** at the bottom of the page. Your secret should now be on the secrets list: ![](../../../_static/images/user-guide/integrations/enabling-gcp-resources/enabling-google-secret-manager/secret-manager.png) Above we see a secret named `example-secret`. Clicking on it will bring us to the **Secret details** page: ![](../../../_static/images/user-guide/integrations/enabling-gcp-resources/enabling-google-secret-manager/secret-details.png) The secret has three important identifiers: * The **GCP secret name**, in this case `example-secret`. You will need this if you are accessing a secret in the same project as your data plane. * The **GCP secret path**, in this case `projects/956281974034/secrets/example-secret`. You will need this if you are accessing a secret in a different project from your data plane project. * The **GCP secret version**, in this case `1`. This is required for both same- and cross-project cases. ## Same-project secrets If your secret is stored in the Secret Manager of the same project as your data plane then the `` will have access to it out-of-the-box. No further configuration is necessary. To use a same-project GCP secret in your task code, do the following: * Define a `Secret` object where * `Secret.group` is the **GCP secret name**, in this case `example-secret`(optionally, you can use the **GCP secret path** instead, but the simple name is sufficient). * `Secret.group_version` is the **GCP secret version** (in this case `1`) * `Secret.mount_requirement` is `Secret.MountType.FILE` * Pass that `Secret` object in the `secret_requests` parameter of the `@union.task` decorator. * Inside the task code, retrieve the value of the secret with a call to `union.current_context().secrets.get(SECRET_GROUP, group_version=SECRET_GROUP_VERSION)`. Here is an example: ```python import union SECRET_GROUP = "example-secret" SECRET_GROUP_VERSION = "1" SECRET_REQUEST = Secret( group=SECRET_GROUP, group_version=SECRET_GROUP_VERSION, mount_requirement=union.Secret.MountType.FILE ) @union.task(secret_requests=[SECRET_REQUEST]) def t1(): secret_val = union.current_context().secrets.get( SECRET_GROUP, group_version=SECRET_GROUP_VERSION ) ``` ## Cross-project secrets If your secret is stored in the Secret Manager of a project other than the one containing your data plane, then you will first need to grant the `` permission to access it: * Find the **email identifier** of the `` in your data plane GCP project (see **BYOC deployment > Enabling GCP resources > Enabling Google Secret Manager** for details). * Go to **Security > Secret Manager** in the GCP project that contains your secret. * Select the secret that you want to access and select **GRANT ACCESS**. * In the subsequent panel, under **Add principals**, paste in the email identifier of the `` that you found above. * Under **Assign roles** add at least the role **Secret Manager Secret Accessor**. * Save the changes. At this point, your task code will have access to the secret in the other project. To use that secret in your task code, do the following: * Define a `union.Secret` object where * `union.Secret.group` is the **GCP secret path** (in this case, `projects/956281974034/secrets/example-secret`) * `union.Secret.group_version` is the **GCP secret version** (in this case `1`) * `union.Secret.mount_requirement` is `union.Secret.MountType.FILE` * Pass that `union.Secret` object in the `secret_requests` parameter of the `@union.task` decorator. * Inside the task code, retrieve the value of the secret with a call to\ `union.current_context().secrets.get(SECRET_GROUP, group_version=SECRET_GROUP_VERSION)` > [!NOTE] GCP secret name vs GCP secret path > In your task code, the only difference between using a same-project secret and a cross-project secret is > > * With a _same-project secret,_ you can use either the **GCP secret name** or the **GCP secret path** as the value of the parameter `union.Secret.group`. > * With a _cross-project secret,_ you must use the **GCP secret path** as the value of the parameter `union.Secret.group`. Here is an example: ```python import union SECRET_GROUP = "projects/956281974034/secrets/example-secret" SECRET_GROUP_VERSION = "1" SECRET_REQUEST = union.Secret( group=SECRET_GROUP, group_version=SECRET_GROUP_VERSION, mount_requirement=union.Secret.MountType.FILE ) @union.task(secret_requests=[SECRET_REQUEST]) def t1(): secret_val = union.current_context().secrets.get( SECRET_GROUP, group_version=SECRET_GROUP_VERSION ) # do something with the secret. For example, communication with an external API. ... ``` > [!WARNING] > Do not return secret values from tasks, as this will expose secrets to the control plane. === PAGE: https://www.union.ai/docs/v2/union/deployment/byoc/enabling-gcp-resources/enabling-bigquery === # Enabling BigQuery For customers using the Google Cloud Platform as the data plane, Union.ai lets you easily pull data from BigQuery into your workflows. For most users on GCP, access to BigQuery should be enabled by default and bound to the service account used by the BigQuery connector.