Start transferring data into your BigQuery project by creating a transfer.
Before you begin
Before creating a data transfer in BigQuery, make sure you have enrolled the data source. It's best practice to create a dataset as a destination for your data before creating the transfer, although a new dataset can also be created in the data transfer configuration.
Creating datasets and transfers requires specific user permissions in BigQuery. The easiest option is using the role BigQuery Admin to grant the user permissions for both. Another option is using a custom role for the user creating transfers, and granting the bigquery.transfer.get and bigquery.transfer.update permissions to that user on the dataset level.
Supermetrics' data sources for BigQuery require your dataset to be located in one of the following Google Cloud Regions:
- us
- europe
- asia-northeast1
- asia-southeast1
- europe-west2
- australia-southeast1
Prerequisites for VPC Service Controls
When you have VPC Service Controls (VPC SC) enabled, it prevents any service from exfiltrating data in/out the perimeter.
To make the BigQuery transfers work with VPC SC enabled, you must add the Supermetrics-owned project into your security perimeter or set up ingress/egress rules. Follow the instructions by Google.
You need the following details for the security perimeter or setting up ingress/egress rules:
- Project ID: supermetrics-sql
- Project number: 377904780319
If VPC SC blocks BigQuery, the following error message is shown:
- Failed to start job for table [TABLE NAME] with error PERMISSION_DENIED: VPC Service Controls: Request is prohibited by organization’s policy. File: [PATH TO SUPERMETRICS’ CLOUD STORAGE BUCKET]; JobID: [BIGQUERY JOB ID]
Instructions
- Log in to the Google Cloud Platform.
- Navigate to BigQuery → Data transfers.
- Click +Create transfer.
- Select the Supermetrics data source you want to use as the source type.
- Fill in the name and schedule details for the transfer.
- Select the destination dataset from your existing datasets or create a new destination dataset.
- Set the Refresh window — the number of days you want to include in each data transfer. The default value is 2 days.
- Click Connect source.
- A popup will open, asking you to enable a third-party connection. Click Accept agreement to proceed.
- Follow the steps in the Supermetrics login window to log in to the data source and define the transfer settings.
- Click Submit. You will be notified of a successful connection to the data source.
- Click Save to store the transfer.
Once your transfer has been saved, you’ll be taken to the Run history of the newly created transfer where you can follow its status.
Table and partition expiration
When you're setting up your transfer, make sure to set the table and partition expiration time to "never" or as many days as possible. Setting the table and partition expiration times as long as possible will make sure that you won't accidentally lose data when running a backfill.
See Google’s documentation for: