{"id":9121,"date":"2026-09-21T13:04:02","date_gmt":"2026-09-21T13:04:02","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=9121"},"modified":"2026-09-22T15:21:43","modified_gmt":"2026-09-22T15:21:43","slug":"working-with-dataflow-using-bigquery","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/working-with-dataflow-using-bigquery\/","title":{"rendered":"Working With Dataflow using BigQuery"},"content":{"rendered":"\r\n

Building a Dataflow Pipeline From a BigQuery Source<\/h1>\r\n

This runs a Dataflow pipeline that reads a table already sitting in BigQuery, processes it with a Python script using Apache Beam, and writes the result into a Cloud Storage bucket. It is a useful third variation alongside the other two Dataflow guides in this series, ETL Processing With Dataflow and BigQuery reads from a file and writes into BigQuery, and Pub\/Sub Stream Processing With Dataflow reads a continuous stream of messages, while this one starts from data that is already a BigQuery table.<\/p>\r\n

This walks through querying the source table directly in BigQuery first, then running the same underlying data through a Dataflow pipeline that finds Java projects with a high ratio of comments suggesting they need help, such as TODO or FIXME notes.<\/p>\r\n

Step by Step Process of Building a Dataflow Pipeline from a BigQuery Source<\/h2>\r\n

Step One: Open BigQuery<\/h3>\r\n

Open the console, then the menu, then BigQuery.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Two: Run a Sample Query<\/h3>\r\n

SELECT
\u00a0\u00a0content
FROM
\u00a0\u00a0fh-bigquery.github_extracts.contents_java_2016
LIMIT
\u00a0\u00a010<\/p>\r\n

Click Run.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

It will display the results.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Three: Run a Count Query<\/h3>\r\n

SELECT
\u00a0\u00a0COUNT(*)
FROM
\u00a0\u00a0fh-bigquery.github_extracts.contents_java_2016<\/p>\r\n

Click Run.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

This gives you the total number of rows in the table.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Four: Clone the Training Repository<\/h3>\r\n

Activate Cloud Shell, then run:<\/p>\r\n

git clone https:\/\/github.com\/GoogleCloudPlatform\/training-data-analyst<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Five: List the Contents<\/h3>\r\n

ls<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Six: Set a Bucket Variable<\/h3>\r\n

Create a bucket in the console first, using the same name as your project ID, then set it as a variable in Cloud Shell:<\/p>\r\n

BUCKET=”your-bucket-name”
echo $BUCKET<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Seven: Move Into the Lab Directory<\/h3>\r\n

cd training-data-analyst\/courses\/data_analysis\/lab2\/python
ls<\/p>\r\n

The lab files will be displayed.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Eight: Open the Script<\/h3>\r\n

nano JavaProjectsThatNeedHelp.py<\/p>\r\n\r\n\r\n\r\n

 <\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

This is a good point to actually read through the script before running it, so you know what it is doing to the data rather than just running it blind.<\/p>\r\n

Exit with control x.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Nine: Run the Pipeline<\/h3>\r\n

python3 JavaProjectsThatNeedHelp.py –bucket $BUCKET –project $DEVSHELL_PROJECT_ID –DataFlowRunner<\/p>\r\n

This specific lab script expects that exact set of arguments, bucket and project as regular flags, with DataFlowRunner as a separate flag that switches it from running locally to running on Dataflow. If your copy of the script has been updated since this was written, check its own argument parser for the current flag names before assuming the command is wrong.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n

Step Ten: Watch the Job Run<\/h3>\r\n\r\n\r\n\r\n

Go to DataFlow > Jobs The jobs will be running<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Open the running job<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Watch its progress.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Eleven: Confirm Success<\/h3>\r\n

Once finished, the job shows a Succeeded status.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Twelve: Check the Output<\/h3>\r\n

Open the menu, then Cloud Storage, then your bucket.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Open the javahelp folder.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

The pipeline’s output is stored there.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n

How This Fits With the Other Dataflow Guides<\/h2>\r\n

Across this series, the three Dataflow examples cover the three common shapes a pipeline’s source and destination can take. ETL Processing With Dataflow and BigQuery reads a file and writes into BigQuery. Pub\/Sub Stream Processing With Dataflow reads a continuous stream and writes to Cloud Storage. This one reads a table already inside BigQuery and writes back out to Cloud Storage, showing that Dataflow treats BigQuery as just another data source, not only a destination.<\/p>\r\n

Common Mistakes to Avoid<\/h2>\r\n