BigQuery is a Bigdata solution created by Google that allows the storage and analysis of data on a large scale. The data integration from BigQuery to the Data Warehouse created by Kondado allows you to access your BigQuery data in your analytical cloud.

Adding the data source

To automate the ETL from BigQuery with Kondado to your database, follow the steps below:

1. Log in to the Kondado platform, go to the page to add data sources and select the Google BigQuery data source
2. Click on AUTHORIZE
3. Select the account you will use
4. In the next screen, check ALL the permissions that are requested and click Continue
5. You will be redirected to the Kondado page with access already authorized. Follow the rest of the step-by-step to find the rest of the necessary information
6. Access your BigQuery console in GCP and copy the project IDs (in red in the figure below) and the dataset (in green)
7. Again on the page from step (5), paste the project and dataset values obtained in step (6) and give a name to your data source

ATTENTION: Make sure that the Dataset ID does NOT contain the project ID separated by ":". For example, "project_id:dataset_id" is wrong. The correct way is for the project ID and Dataset ID to be each separately in their respective fields

Now just save the data source and start integrating your BigQuery data into the Data Lake or Data Warehouse.

Don't forget to click on "TEST CONNECTION" to check if everything is right!

Tables and Views

With our integration, you will be able to integrate tables and also views

If your table has a datetime/timestamp type column that marks when a record was changed/inserted, your integration can be incremental

It will be necessary to define the primary key, which can be defined by several columns and refers to the column (or set of columns) that define a record as being unique