Google BigQuery is Google's fully managed, petabyte scale, low-cost enterprise data warehouse for analytics. Here are some things you can expect from the Google BigQuery/Grow integration.
In this article, we will cover:
Connecting to BigQuery
Before you get started you will need your Google BigQuery Project ID, Client Email and Private Key or you can upload a JSON file with your project ID, email, and private key.
- Click on the Data tab at the top left of the global navigation. Then click on Connections in the Data Library section.
- Click on the blue Connect button and search for Google BigQuery.
- Click on the Google BigQuery logo.
- The most straightforward way to connect your BigQuery account is to click Fill in manually on the authorization page. You will need the BigQuery project ID you want to connect, the email address associated with the BigQuery account, and your private key.
- If you do not know what your private key is or how to get it, Google has a helpful article on how to do so. You have to do a bit of scrolling through the page to get to what you need, so we have copied the important details below:
To generate a private key in JSON or PKCS12 format in Google BigQuery:
- Open the list of credentials in the Google Cloud Platform Console.
- Click Create credentials.
- Select Service account key. A Create service account key window opens.
- Click the drop-down box below Service account, then click New service account.
- Enter a name for the service account in Name.
- Use the default Service account ID or generate a different one.
- Select the Key type: JSON or P12.
- Click Create. A Service account created window is displayed and the private key for the Key type you selected is downloaded automatically. If you selected a P12 key, the private key's password ("notasecret") is displayed.
- Highlight and copy your private key.
- You can then come back to Grow and paste the key into the Private Key field. You'll need to also enter your email associated with BigQuery and your Project ID. Alternatively, if you have the know-how and would prefer to upload a JSON file with your project ID, email, and private key, that will work as well.
- When you've entered in this information, press the Connect button. You will see a green confirmation message that the connection has been made.
Google BigQuery is on the Grow Data Warehouse, which we highly recommend you use. For more information about warehousing your data, visit the Data Warehouse Help Article.
The list of tables available will depend on your unique Google BigQuery project. You can expect to see one table per dataset associated with the project Id in your connection. Below is an example:
Warehousing your Data
- Starting with the Connections Overview Page, navigate to your Google BigQuery connection and click on it to open the Connection Details page.
- Select the Manage Connection button inside the Connection Details section to open the warehouse table selection flow.
- If your data source isn't already connected, refer to the Connecting to a Data Source for more details. If your data source is already connected, you can immediately select which tables of data you want to warehouse.Once you have finished selecting the data that you want stored in the Data Warehouse, click the Sync & Store button at the bottom of the Manage Connection page. This will begin the initial population of your warehouse with the tables you defined. Populating your data warehouse for the first time may take a while, even up to several hours.
- While your data is "In Queue" or "Processing", you may continue working within the app.
The default sync interval is 1 week.
Incremental Sync is available for all Google BigQuery warehoused tables. The default incremental sync interval is 1 hour.
To learn more about sync intervals and how to adjust the default settings, please visit the Sync section of the Data Warehouse help article.
A direct query connection can be used to pull data into Grow. With abnormal usage, Direct Query can run into API limitations depending on the data source. For this reason, we recommend that you use Grow's Data Warehouse for a more performant, reliable experience.
Below is a list of the reports you can pull from BigQuery, along with a description of the fields available with each report.
- Datasets: All datasets associated with projects under the connected BigQuery account, including kind, dataset ID, dataset reference, and project ID.
- Tables: All tables for the selected dataset, including kind, table ID, project, dataset, table name, type, and unix epoch creation time.
- Table Data: All table data for the selected dataset and table. The fields returned here will vary based on the specifics of the datasets created with BigQuery.
- Custom SQL Query: Write custom SQL queries to hone in on specific fields when building a metric.
This is for reference only. You should not need to consult these to use the connection in Grow.
Permissions for Connecting to BigQuery using the Grow Warehouse
When connecting to Google BigQuery using the Grow Warehouse make sure you have the correct roles and permissions assigned. The user who is connecting BigQuery to Grow needs to have the assigned role of roles/bigquery.jobUser then also have the permission bigquery.jobs.create. Assigning this role and permission will allow you to run jobs including queries inside projects.
For more in-depth information on roles and permissions see Predefined roles and permissions.
Unix Epoch Dates
Some datatable information fields include creation dates in unix epoch format. This will just look like a string of random numbers. Luckily, there is a quick way to fix that, either through the table date format function in the edit metric screen in Grow, or with a relatively simple SQL query.
How can I make sure my data is secure?
We have several items in place to keep your data safe. Check out our security article to see what steps you can take to keep control of your data.