{"id":8700,"date":"2026-09-17T07:36:39","date_gmt":"2026-09-17T07:36:39","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=8700"},"modified":"2026-09-18T16:53:53","modified_gmt":"2026-09-18T16:53:53","slug":"adding-gcs-file-into-bq","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/adding-gcs-file-into-bq\/","title":{"rendered":"Adding GCS file into BigQuery"},"content":{"rendered":"\r\n

Loading a Google Cloud Storage File Into BigQuery<\/h1>\r\n

Cloud Storage is the standard place to stage a file before it becomes a BigQuery table, especially once a file is too large for the direct upload option in the console. This walks through loading a CSV already sitting in a bucket into a new BigQuery table, then querying it.<\/p>\r\n

Loading data from Google Cloud Storage (GCS) into BigQuery is a fundamental process that forms the backbone of many data ingestion pipelines within Google’s comprehensive cloud ecosystem. This introduction will explore the intricacies of data loading from GCS to BigQuery, providing insights into the mechanisms, best practices, and benefits associated with this essential operation.<\/p>\r\n

Google Cloud Storage serves as a scalable and durable object storage solution, ideal for storing a wide array of data types, including structured, semi-structured, and unstructured data. BigQuery, on the other hand, offers a powerful, fully managed data warehouse platform with lightning-fast SQL queries and built-in machine learning capabilities.<\/p>\r\n

Step by Step Process of Loading Data from GCS into BigQuery<\/h2>\r\n

The process of loading data from GCS into BigQuery involves several steps, including defining the data schema, configuring the load job settings, and initiating the data transfer. Users can choose from a variety of file formats supported by both GCS and BigQuery, such as CSV, JSON, Avro, Parquet, and more, ensuring compatibility and flexibility in data loading operations.<\/p>\r\n

Step One: Get the Sample File Into a Bucket<\/h3>\r\n

Download this sample CSV, a small subset of New York taxi trip data:<\/p>\r\n\r\n\r\n\r\n

https:\/\/storage.googleapis.com\/cloud-training\/OCBL013\/nyc_tlc_yellow_trips_2018_subset_1.csv<\/a><\/p>\r\n\r\n\r\n\r\n

Upload it into a Cloud Storage bucket in your project.<\/p>\r\n\r\n\r\n\r\n

Step Two: Open BigQuery<\/h3>\r\n

Open the console, then the menu, then BigQuery.<\/p>\r\n\r\n\r\n\r\n

Open Menu > BigQuery.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Three: Create a Dataset<\/h3>\r\n

Open your project, then click Create Dataset.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Give it a dataset ID, then click Create Dataset.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Four: Create a Table<\/h3>\r\n

Click Create Table.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Select Google Cloud Storage as the source, then click Browse.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Choose the file from bucket. Click Select<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

The File will be selected with file format.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Choose the Destination table.<\/p>\r\n\r\n\r\n\r\n

Give project name, dataset name and table name.<\/p>\r\n\r\n\r\n\r\n

Click on auto detect. Click create table.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

The table is created and loaded with the file’s data.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Five: Query the Table<\/h3>\r\n

In the query editor, replace the dataset and table names below with the ones you actually created, then run it to see the ten highest fares in the dataset:<\/p>\r\n

SELECT
\u00a0\u00a0*
FROM
\u00a0\u00a0your_dataset.your_table_name
ORDER BY
\u00a0\u00a0fare_amount DESC
LIMIT
\u00a0\u00a010;<\/p>\r\n

Click Run.The Query will be executed.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

This second version narrows the same question down to January only, using EXTRACT to pull the month out of the pickup timestamp:<\/p>\r\n

SELECT
\u00a0\u00a0*
FROM
\u00a0\u00a0your_dataset.your_table_name
WHERE
\u00a0\u00a0EXTRACT(MONTH FROM pickup_datetime) = 1
ORDER BY
\u00a0\u00a0fare_amount DESC
LIMIT
\u00a0\u00a010;<\/p>\r\n\r\n\r\n\r\n

Click Run.The Query will be executed.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n

 <\/p>\r\n

Load Job Versus External Table<\/h2>\r\n

Loading, which is what the steps above do, copies the file’s data into BigQuery’s own storage as a native table. There is a second option worth knowing about: creating an external table that points at the file in Cloud Storage without copying it in at all. BigQuery reads the file directly from the bucket each time the external table is queried.<\/p>\r\n