{"id":9078,"date":"2026-09-21T11:55:53","date_gmt":"2026-09-21T11:55:53","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=9078"},"modified":"2026-09-22T15:36:56","modified_gmt":"2026-09-22T15:36:56","slug":"working-with-dataflow","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/working-with-dataflow\/","title":{"rendered":"Working with Dataflow"},"content":{"rendered":"\r\n

Working With Dataflow: From a Local Run to the Cloud<\/h1>\r\n

This is the foundational Dataflow lab that the other Dataflow guides in this series build on. It runs the same basic idea, a script that filters text for a pattern, three times: once locally on your own machine, once on Dataflow itself, and once more with a slightly more advanced example that groups and ranks results, the same pattern used in the ETL and BigQuery source Dataflow guides elsewhere in this series.<\/p>\r\n

This walks through setting up the lab environment, running a simple pipeline locally to see what it does before involving the cloud at all, then running the equivalent version on Dataflow, and finally a more advanced example that writes ranked output back into Cloud Storage.<\/p>\r\n

Step by Step Process of Working with Dataflow<\/h2>\r\n

Step 1: Clone the Training Repository<\/h2>\r\n

Activate Cloud Shell, then run:<\/p>\r\n

git clone https:\/\/github.com\/GoogleCloudPlatform\/training-data-analyst<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ls<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step 2: Set a Bucket Variable<\/h3>\r\n

Create a bucket in the console first, using the same name as your project ID, then set it as a variable:<\/p>\r\n

BUCKET=”your-bucket-name”
echo $BUCKET<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n

Step 3: Enable the Dataflow API<\/h3>\r\n\r\n\r\n\r\n

Open Menu > API services > Library<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Search for Dataflow, then click the Dataflow API.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Click Enable<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step 4: Move Into the Lab Directory<\/h3>\r\n

cd training-data-analyst\/courses\/data_analysis\/lab2\/python
ls<\/p>\r\n

The lab files will be displayed.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step 5: Install the Required Packages<\/h3>\r\n

Open install_packages.sh to see what it does before running it:<\/p>\r\n

nano install_packages.sh<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

This file installs the Python components the lab needs.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 sudo .\/install_packages.sh<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step 6: Confirm Your Python Version<\/h3>\r\n

pip -V
pip3 -V<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step 7: Run a Simple Pipeline Locally<\/h3>\r\n

Open grep.py and look at what it does. This script searches text files for a pattern and runs on Beam’s DirectRunner by default, meaning it executes right there in Cloud Shell, with no Dataflow job created at all.<\/p>\r\n

nano grep.py<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n
\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 python3 grep.py<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Confirm it actually produced output:<\/p>\r\n

ls \/tmp<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

cat \/tmp\/output-*<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step 8: Copy Sample Files Into Your Bucket<\/h3>\r\n

gsutil cp ..\/javahelp\/src\/main\/java\/com\/google\/cloud\/training\/dataanalyst\/javahelp\/*.java gs:\/\/$BUCKET\/javahelp<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Confirm they arrived. Open the menu, then Cloud Storage, then your bucket.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

The files should be listed there.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step 9: Run the Same Idea on Dataflow<\/h3>\r\n

Confirm your project and bucket variables are still set correctly:<\/p>\r\n

echo $DEVSHELL_PROJECT_ID
echo $BUCKET<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Open grepc.py, the cloud version of the same script:<\/p>\r\n

nano grepc.py<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Edit its PROJECT and BUCKET values to match your own project ID and bucket name. If both happen to be the same value, you can use that same ID for both.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Save and exit with control x, then y, then enter, and run it:<\/p>\r\n

python3 grepc.py<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n

Step 10: Watch the Job in Dataflow<\/h3>\r\n\r\n\r\n\r\n

Open Console >Dataflow > Jobs<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Open the Job which is executed.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Click the Job Graph.<\/p>\r\n\r\n\r\n\r\n

The Graph is displayed.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

In Job Graph on right side you can see the Job info and resource metrics.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step 11: A More Advanced Example<\/h3>\r\n

is_popular.py goes further than a simple filter, grouping and ranking results rather than just matching a pattern. Open it and look through it before running it.<\/p>\r\n

ls
nano is_popular.py<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

It will open the file is_popular.py<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Run it, then check the output:<\/p>\r\n

python3 .\/is_popular.py
cat \/tmp\/output-*<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 python3 .\/is_popular.py\u00a0 –output_prefix=\/tmp\/myoutput<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nano \/tmp\/myoutput-00000-of-00001<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

It will open the file with output.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step 12: Check the Final Output in Your Bucket<\/h3>\r\n

Open the menu, then Cloud Storage, then your bucket.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Open javahelp\/ folder<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

The outputs will be stored in it.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n

DirectRunner Versus DataflowRunner<\/h2>\r\n

The reason this page runs grep.py before grepc.py is to show the same underlying idea two different ways. DirectRunner, the default when no runner is specified, executes a pipeline on the machine you are already on, useful for quickly checking that your logic works before spending time and money running it at scale. DataflowRunner submits the exact same kind of pipeline to run as a managed job in the cloud, which is what you actually want once the data is too large for one machine, or once the job needs to keep running without you watching it.<\/p>\r\n

How This Page Fits With the Other Dataflow Guides<\/h2>\r\n

This page is the right starting point in this series, since it shows the DirectRunner versus DataflowRunner distinction that the other three assume you already understand. From here, ETL Processing With Dataflow and BigQuery shows a complete batch pipeline reading a file and writing into BigQuery, Pub\/Sub Stream Processing With Dataflow shows a continuously running pipeline instead of a one time job, and Building a Dataflow Pipeline From a BigQuery Source shows Dataflow reading from BigQuery itself as the input.<\/p>\r\n

Common Mistakes to Avoid<\/h2>\r\n