Uploading local files to BigQuery alternatives

Integrating local files into BigQuery represents a vital aspect of expanding data accessibility and leveraging the platform’s analytical capabilities. This introduction will elucidate the process of seamlessly incorporating local files into BigQuery, elucidating the methods, considerations, and advantages inherent in this operation.

While BigQuery excels in processing vast datasets stored in Google Cloud Storage or other cloud-based repositories, there are scenarios where local files, residing within an organization’s infrastructure or on individual workstations, contain valuable data that warrants inclusion in analytical workflows.

The process of adding local files into BigQuery involves a few essential steps, such as preparing the data in a compatible format, uploading it to Google Cloud Storage, and then utilizing BigQuery’s native capabilities to ingest the data into its tables. Supported file formats include CSV, JSON, Avro, Parquet, and more, ensuring flexibility and compatibility with various data sources.

Prerequisites

GCP account

Open Console.

Open Menu > Bigquery > SQL Workspace

In Resources, select your project.

Click on Create  Dataset

Give the name for Dataset. Rest of it no need to change.

Click Create Dataset.

The dataset will be created.

Download one file by pressing the below link.

https://storage.googleapis.com/cloud-training/OCBL013/nyc_tlc_yellow_trips_2018_subset_1.csv

In the created dataset, click create Table.

Choose the source as upload.

Click Browse and choose the downloaded file.

It will automatically detect the file format. If not, select it as CSV

Give the Project name and Dataset name. Usually it will be auto filled.

Give the table name.

Tick the Auto Detect Schema. Click Create table.

It will create the table with the downloaded file content.

Paste the below query in query editor.

SELECT

     *

FROM

     nyc_taxi.2018_trips

ORDER BY

     fare_amount DESC

LIMIT

     10

And Click run. It will give the output in query results.

Uploading local files to BigQuery alternatives