{"id":8045,"date":"2026-09-17T10:15:10","date_gmt":"2026-09-17T10:15:10","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=8045"},"modified":"2026-09-19T15:19:11","modified_gmt":"2026-09-19T15:19:11","slug":"etl-processing-on-google-cloud-using-dataflow-and-big-query","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/","title":{"rendered":"ETL Processing on Google Cloud Using Dataflow and Big Query"},"content":{"rendered":"\r\n

ETL Processing With Dataflow and BigQuery<\/h1>\r\n

Dataflow<\/b><\/strong>\u00a0is Google Cloud’s managed service for running Apache Beam pipelines, code that reads data, transforms it, and writes it somewhere else, without you having to provision or manage the machines that actually do the work. You submit a pipeline, Dataflow spins up however many workers it needs, runs it, and shuts them down again once it finishes.<\/p>\r\n

Pairing it with BigQuery is a common ETL pattern for a specific reason. BigQuery is excellent at querying data once it is already clean and structured, but not every transformation is easy to express as SQL, and raw source data is often messy before it gets anywhere near a table. Dataflow handles that in between step in Python, reshaping and enriching the data, then lands the result directly in BigQuery, ready to query.<\/p>\r\n

This builds that pattern in three concrete stages, ingesting a raw CSV file, transforming it, then enriching it further, with each stage’s output landing in BigQuery before the next one runs.<\/p>\r\n

Step by Step Process of ETL on Google Cloud With Dataflow and BigQuery<\/h2>\r\n

Step One: Activate Cloud Shell<\/h3>\r\n

Open the console and click Activate Cloud Shell.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n

 <\/p>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Two: Confirm and Set Your Project<\/h3>\r\n

gcloud config list project
gcloud config set project my-demo-project-306417<\/p>\r\n

The first command lists your current project. The second sets the active project, replacing the example ID with your own.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Three: Confirm the Compute Service Account<\/h3>\r\n

Open the menu, then IAM and Admin.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Confirm that the default compute Service Account\u00a0is present and has the\u00a0editor\u00a0role assigned.<\/p>\r\n\r\n\r\n\r\n

{project-number}-compute@developer.gserviceaccount.com<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n

Step Four: Find Your Project Number<\/h3>\r\n<\/div>\r\n\r\n\r\n\r\n

To check the project number, Go to Menu > Home > Dashboard<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

The project number is shown under Project Info.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Five: Copy the Sample Pipeline Code<\/h3>\r\n

gsutil -m cp -R gs:\/\/spls\/gsp290\/dataflow-python-examples .<\/p>\r\n

This copies the sample Dataflow Python code from Google Cloud’s own training bucket into your Cloud Shell home directory.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Six: Set Your Project as an Environment Variable<\/h3>\r\n

export PROJECT=YOUR-PROJECT-ID
gcloud config set project $PROJECT<\/p>\r\n

Replace YOUR-PROJECT-ID with your actual project ID. Do not change PROJECT itself once it is set, later commands rely on that exact variable name.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n

Step Seven: Create a Bucket<\/h3>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0gsutil mb -c regional -l us-central1 gs:\/\/$PROJECT\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To create new Bucket<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Eight: Copy Sample Data Into the Bucket<\/h3>\r\n

gsutil cp gs:\/\/spls\/gsp290\/data_files\/usa_names.csv gs:\/\/$PROJECT\/data_files\/<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Nine: Create a BigQuery Dataset<\/h3>\r\n

bq mk lake<\/p>\r\n

This creates a dataset named lake, which the pipeline will load results into.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Ten: Set Up a Python Virtual Environment<\/h3>\r\n

cd dataflow-python-examples\/
python3 -m venv venv
source venv\/bin\/activate
pip install “apache-beam[gcp]”<\/p>\r\n

Current Cloud Shell images include Python’s own venv module, so a plain python3 -m venv works without needing to separately install virtualenv or use sudo with pip, which current guidance advises against. Installing apache-beam without a version pin gets you the current supported release, rather than a six year old one that Google’s own documentation no longer recommends.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0 \u00a0virtualenv -p python3 venv\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Creating Virtual Environement<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0source venv\/bin\/activate\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Activating Virtual Environment<\/p>\r\n\r\n\r\n\r\n

\u00a0 pip install apache-beam[gcp]==2.24.0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Installing Apache-beam<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Eleven: Open the Cloud Shell Editor<\/h3>\r\n

In the Cloud Shell window, click Open Editor.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Twelve: Review the Sample Code<\/h3>\r\n

The dataflow-python-examples folder appears in the workspace. Take a moment to look through the Python files inside dataflow_python_examples before running anything.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Thirteen: Run the Data Ingestion Pipeline<\/h3>\r\n

python dataflow_python_examples\/data_ingestion.py \\
–project=$PROJECT \\
–region=us-central1 \\
–runner=DataflowRunner \\
–staging_location=gs:\/\/$PROJECT\/test \\
–temp_location=gs:\/\/$PROJECT\/test \\
–input=gs:\/\/$PROJECT\/data_files\/usa_names.csv \\
–save_main_session<\/p>\r\n

This runs a pipeline that provisions its own workers, does the work, then shuts the workers down when finished. It takes a few minutes.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n

Step Fourteen: Check the Job Status<\/h3>\r\n<\/div>\r\n\r\n\r\n\r\n

Open the console and click Menu > Dataflow > Jobs<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

The job shows Running until it finishes, then Succeeded.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n

Step Fifteen: Confirm the Table in BigQuery<\/h3>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0Big Query > SQL Workspace<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

The usa_names table now appears under the lake dataset.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Sixteen: Run the Data Transformation Pipeline<\/h3>\r\n

python dataflow_python_examples\/data_transformation.py \\
–project=$PROJECT \\
–region=us-central1 \\
–runner=DataflowRunner \\
–staging_location=gs:\/\/$PROJECT\/test \\
–temp_location=gs:\/\/$PROJECT\/test \\
–input=gs:\/\/$PROJECT\/data_files\/head_usa_names.csv \\
–save_main_session<\/p>\r\n

Same pattern as before, this pipeline provisions workers, transforms the data, and shuts down when done.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Seventeen: Edit the Data Enrichment Script<\/h2>\r\n

In the Cloud Shell editor, open dataflow-python-examples, then dataflow_python_examples, then data_enrichment.py. Find this line, around line 83:<\/p>\r\n

values = [x.decode(‘utf8’) for x in csv_row]<\/p>\r\n

Replace it with:<\/p>\r\n

values = [x for x in csv_row]<\/p>\r\n

This removes an old Python 2 style decode call that is not needed under Python 3, and lets the pipeline actually populate BigQuery. If you open this file and the line already looks like the replacement, the sample repository has already been updated upstream, and you can skip this step.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n
\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Step Eighteen: Run the Data Enrichment Pipeline<\/h3>\r\n

python dataflow_python_examples\/data_enrichment.py \\
–project=$PROJECT \\
–region=us-central1 \\
–runner=DataflowRunner \\
–staging_location=gs:\/\/$PROJECT\/test \\
–temp_location=gs:\/\/$PROJECT\/test \\
–input=gs:\/\/$PROJECT\/data_files\/head_usa_names.csv \\
–save_main_session<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n

Step Nineteen: Confirm the Final Result<\/h3>\r\n<\/div>\r\n\r\n\r\n\r\n

Open Menu > Big Query > SQL Workspace<\/p>\r\n\r\n\r\n\r\n

It will show the populated dataset in lake.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n

Step Twenty: Review the Pipeline Graph<\/h3>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0Dataflow > Jobs.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Click on the job which you have done.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

It will show the Data Flow pipeline of the work.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n

Common Mistakes to Avoid<\/h2>\r\n