{"id":9397,"date":"2021-05-31T09:33:24","date_gmt":"2021-05-31T09:33:24","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=9397"},"modified":"2024-03-20T06:44:59","modified_gmt":"2024-03-20T06:44:59","slug":"pubsub-stream-processing-with-dataflow","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/","title":{"rendered":"Pubsub stream processing with dataflow"},"content":{"rendered":"\r\n

Cloud Pub\/Sub and Dataflow for Stream Processing<\/span><\/h2>\r\n

 <\/p>\r\n

Prerequisites<\/strong><\/p>\r\n\r\n\r\n\r\n

GCP<\/a> account<\/p>\r\n\r\n\r\n\r\n

Open console.<\/p>\r\n\r\n\r\n\r\n

Open Console. Click on Activate cloud shell<\/p>\r\n\r\n\r\n\r\n

NB: If any permission error arises, give owner permission for the default service account<\/p>\r\n\r\n\r\n\r\n

Create variables for your bucket<\/a>, project<\/a>, pubsub<\/a> topic and region. Select a Dataflow\u00a0region\u00a0close to where you run the commands<\/p>\r\n\r\n\r\n\r\n

BUCKET_NAME=your-bucket-name<\/em>
PROJECT_ID=(gcloud config get-value project)
TOPIC_ID=your-topic-id<\/em>
REGION=dataflow-region<\/em><\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Create a Cloud Scheduler job in this project.\u00a0<\/p>\r\n\r\n\r\n\r\n

$ gcloud scheduler jobs create pubsub publisher-job \\<\/p>\r\n\r\n\r\n\r\n

–schedule=”* * * * *” \\<\/p>\r\n\r\n\r\n\r\n

–topic=$TOPIC_ID –message-body=”Hello!”<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Start the job.<\/p>\r\n\r\n\r\n\r\n

gcloud scheduler jobs run publisher-job<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Use the following commands to clone the quickstart repository and navigate to the sample code directory:<\/p>\r\n\r\n\r\n\r\n

git clone https:\/\/github.com\/GoogleCloudPlatform\/python-docs-samples.git<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

cd python-docs-samples\/pubsub\/streaming-analytics<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

pip3 install -r requirements.txt \u00a0<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

NB : If error arises as apache beam version is not available, open requirements.txt file (nano requirements.txt) and change version to 2.24.0<\/p>\r\n\r\n\r\n\r\n

Open File<\/p>\r\n\r\n\r\n\r\n

$ nano PubSubToGCS.py<\/p>\r\n\r\n\r\n\r\n

paste the below code<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n
\r\n
\r\n
\r\n
\r\n
\r\n
\r\n

# Copyright 2019 Google LLC.<\/p>\r\n\r\n\r\n\r\n\r\n\r\n

Licensed under the Apache License, Version 2.0 (the “License”);<\/p>\r\n\r\n\r\n\r\n

\u00a0you may not use this file except in compliance with the License.<\/p>\r\n\r\n\r\n\r\n

You may obtain a copy of the License at<\/p>\r\n\r\n\r\n\r\n\r\n\r\n

http:\/\/www.apache.org\/licenses\/LICENSE-2.0<\/p>\r\n\r\n\r\n\r\n\r\n\r\n

\u00a0Unless required by applicable law or agreed to in writing, software<\/p>\r\n\r\n\r\n\r\n

distributed under the License is distributed on an “AS IS” BASIS,<\/p>\r\n\r\n\r\n\r\n

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.<\/p>\r\n\r\n\r\n\r\n

See the License for the specific language governing permissions and<\/p>\r\n\r\n\r\n\r\n

# limitations under the License.<\/h3>\r\n\r\n\r\n\r\n

# [START pubsub_to_gcs]<\/p>\r\n\r\n\r\n\r\n

import argparse<\/p>\r\n\r\n\r\n\r\n

from datetime import datetime<\/p>\r\n\r\n\r\n\r\n

import logging<\/p>\r\n\r\n\r\n\r\n

import random<\/p>\r\n\r\n\r\n\r\n

from apache_beam import DoFn, GroupByKey, io, ParDo, Pipeline, PTransform, WindowInto, WithKeys<\/p>\r\n\r\n\r\n\r\n

#from apache_beam.options.pipeline_options import PipelineOptions<\/p>\r\n\r\n\r\n\r\n

from apache_beam.transforms.window import FixedWindows<\/p>\r\n\r\n\r\n\r\n

class GroupMessagesByFixedWindows(PTransform):<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 “””A composite transform that groups Pub\/Sub messages based on publish time<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 and outputs a list of tuples, each containing a message and its publish time.<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 “””<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 def __init__(self, window_size, num_shards=5):<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # Set window size to 60 seconds.<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 self.window_size = int(window_size * 60)<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 self.num_shards = num_shards<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 def expand(self, pcoll):<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 return (<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pcoll<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0Bind window info to each element using element timestamp (or publish time).<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 | “Window into fixed intervals”<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 >> WindowInto(FixedWindows(self.window_size))<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 | “Add timestamp to windowed elements” >> ParDo(AddTimestamp())<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Assign a random key to each windowed element based on the number of shards.<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 | “Add key” >> WithKeys(lambda _: random.randint(0, self.num_shards – 1))<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Group windowed elements by key. All the elements in the same window must fit<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 memory for this. If not, you need to use `beam.util.BatchElements`.<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 | “Group by key” >> GroupByKey()<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 )<\/p>\r\n\r\n\r\n\r\n

class AddTimestamp(DoFn):<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 def process(self, element, publish_time=DoFn.TimestampParam):<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “””Processes each windowed element by extracting the message body and its<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 publish time into a tuple.<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “””<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 yield (<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 element.decode(“utf-8”),<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 datetime.utcfromtimestamp(float(publish_time)).strftime(<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “%Y-%m-%d %H:%M:%S.%f”<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ),<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 )<\/p>\r\n\r\n\r\n\r\n

class WriteToGCS(DoFn):<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 def __init__(self, output_path):<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 self.output_path = output_path<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 def process(self, key_value, window=DoFn.WindowParam):<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “””Write messages in a batch to Google Cloud Storage.”””<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ts_format = “%H:%M”<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 window_start = window.start.to_utc_datetime().strftime(ts_format)<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 window_end = window.end.to_utc_datetime().strftime(ts_format)<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 shard_id, batch = key_value<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 filename = “-“.join([self.output_path, window_start, window_end, str(shard_id)])<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 with io.gcsio.GcsIO().open(filename=filename, mode=”w”) as f:<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 for message_body, publish_time in batch:<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 f.write(f”{message_body},{publish_time}”.encode(“utf-8”))<\/p>\r\n\r\n\r\n\r\n

def run(input_topic, output_path, window_size=1.0, num_shards=5, pipeline_args=None):<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 Set `save_main_session` to True so DoFns can access globally imported modules.<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 pipeline_options = PipelineOptions(<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pipeline_args, streaming=True, save_main_session=True<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 )<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 with Pipeline(options=pipeline_options) as pipeline:<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 (<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pipeline<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Because `timestamp_attribute` is unspecified in `ReadFromPubSub`, Beam<\/h3>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 binds the publish time returned by the Pub\/Sub server for each message<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 to the element’s timestamp parameter, accessible via `DoFn.TimestampParam`.<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0https:\/\/beam.apache.org\/releases\/pydoc\/current\/apache_beam.io.gcp.pubsub.html<\/p>\r\n

apache_beam.io.gcp.pubsub.ReadFromPubSub<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “Read from Pub\/Sub” >> io.ReadFromPubSub(topic=input_topic)<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “Window into” >> GroupMessagesByFixedWindows(window_size, num_shards)<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 “Write to GCS” >> ParDo(WriteToGCS(output_path))<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 )<\/p>\r\n\r\n\r\n\r\n

if __name__ == “__main__”:<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 logging.getLogger().setLevel(logging.INFO)<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 parser = argparse.ArgumentParser()<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 parser.add_argument(<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “–input_topic”,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 help=”The Cloud Pub\/Sub topic to read from.”<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ‘”projects\/<PROJECT_ID>\/topics\/<TOPIC_ID>”.’,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 )<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 parser.add_argument(<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “–window_size”,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 type=float,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 default=1.0,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 help=”Output file’s window size in minutes.”,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 )<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 parser.add_argument(<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “–output_path”,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 help=”Path of the output GCS file including the prefix.”,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 )<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 parser.add_argument(<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 “–num_shards”,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 type=int,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 default=5,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 help=”Number of shards to use when writing windowed elements to GCS.”,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 )<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 known_args, pipeline_args = parser.parse_known_args()<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 run(<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 known_args.input_topic,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 known_args.output_path,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 known_args.window_size,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 known_args.num_shards,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pipeline_args,<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 )<\/p>\r\n\r\n\r\n\r\n

# [END pubsub_to_gcs]<\/p>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

To save and exit press Ctrl+ x then press Y to confirm then press Enter.<\/p>\r\n\r\n\r\n\r\n

Paste the below code in shell.<\/p>\r\n\r\n\r\n\r\n

$ python3 PubSubToGCS.py \\<\/p>\r\n\r\n\r\n\r\n

\u00a0 –project=$PROJECT_ID \\<\/p>\r\n\r\n\r\n\r\n

\u00a0 –region=$REGION \\<\/p>\r\n\r\n\r\n\r\n

\u00a0 –input_topic=projects\/$PROJECT_ID\/topics\/$TOPIC_ID \\<\/p>\r\n\r\n\r\n\r\n

\u00a0 –output_path=gs:\/\/$BUCKET_NAME\/samples\/output \\<\/p>\r\n\r\n\r\n\r\n

\u00a0 –runner=DataflowRunner \\<\/p>\r\n\r\n\r\n\r\n

\u00a0 –window_size=2 \\<\/p>\r\n\r\n\r\n\r\n

\u00a0 –num_shards=2 \\<\/p>\r\n\r\n\r\n\r\n

\u00a0 –temp_location=gs:\/\/$BUCKET_NAME\/temp<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

In console, Open Menu > Dataflow<\/a>.<\/p>\r\n\r\n\r\n\r\n

You can see the Job progress in Dataflow Console<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Wait little bit time and check the bucket. It will have samples folder and inside that the output will be created. Or<\/p>\r\n\r\n\r\n\r\n

$ gsutil ls gs:\/\/${BUCKET_NAME}\/samples\/<\/p>\r\n\r\n\r\n\r\n

It will display the output created<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Delete the Cloud Scheduler job.<\/p>\r\n\r\n\r\n\r\n

$ gcloud scheduler jobs delete publisher-job<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n","protected":false},"excerpt":{"rendered":"

Cloud Pub\/Sub and Dataflow for Stream Processing   Prerequisites GCP account Open console. Open Console. Click on Activate cloud shell NB: If any permission error arises, give owner permission for the default service account Create variables for your bucket, project, pubsub topic and region. Select a Dataflow\u00a0region\u00a0close to where you run the commands BUCKET_NAME=your-bucket-namePROJECT_ID=(gcloud config […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1636,1],"tags":[605,699,700,984,617,683,684,685,611,1400,692,1523,1524,1525,1531,1530],"class_list":["post-9397","post","type-post","status-publish","format-standard","hentry","category-dataflow","category-google-cloud-platform","tag-gcp","tag-gcp-certification","tag-gcp-cloud-console","tag-gcp-course","tag-google-cloud","tag-google-cloud-certification","tag-google-cloud-console","tag-google-cloud-courses","tag-google-cloud-platform","tag-google-cloud-platform-tutorial","tag-google-cloud-training","tag-pubsub","tag-pubsub-google-cloud","tag-pubsub-message","tag-pubsub-to-bucket","tag-pubsub-to-gcs"],"acf":[],"yoast_head":"\nCloud Pub\/Sub and Dataflow for Stream Processing - Prwatech<\/title>\n<meta name=\"description\" content=\"Master Cloud Pub\/Sub and Dataflow for Stream Processing - Dive deep with our expert instructors and comprehensive curriculum.\" \/>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Cloud Pub\/Sub and Dataflow for Stream Processing - Prwatech\" \/>\n<meta property=\"og:description\" content=\"Master Cloud Pub\/Sub and Dataflow for Stream Processing - Dive deep with our expert instructors and comprehensive curriculum.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/\" \/>\n<meta property=\"og:site_name\" content=\"Prwatech\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prwatech.in\/\" \/>\n<meta property=\"article:published_time\" content=\"2021-05-31T09:33:24+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-03-20T06:44:59+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-648.png\" \/>\n<meta name=\"author\" content=\"Prwatech\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:site\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Prwatech\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/\",\"url\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/\",\"name\":\"Cloud Pub\/Sub and Dataflow for Stream Processing - Prwatech\",\"isPartOf\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-648.png\",\"datePublished\":\"2021-05-31T09:33:24+00:00\",\"dateModified\":\"2024-03-20T06:44:59+00:00\",\"author\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\"},\"description\":\"Master Cloud Pub\/Sub and Dataflow for Stream Processing - Dive deep with our expert instructors and comprehensive curriculum.\",\"breadcrumb\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/#primaryimage\",\"url\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-648.png\",\"contentUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-648.png\",\"width\":628,\"height\":64},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/prwatech.in\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Pubsub stream processing with dataflow\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/prwatech.in\/blog\/#website\",\"url\":\"https:\/\/prwatech.in\/blog\/\",\"name\":\"Prwatech\",\"description\":\"Share Ideas, Start Something Good.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/prwatech.in\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\",\"name\":\"Prwatech\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"caption\":\"Prwatech\"},\"url\":\"https:\/\/prwatech.in\/blog\/author\/prwatech123\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Cloud Pub\/Sub and Dataflow for Stream Processing - Prwatech","description":"Master Cloud Pub\/Sub and Dataflow for Stream Processing - Dive deep with our expert instructors and comprehensive curriculum.","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"en_US","og_type":"article","og_title":"Cloud Pub\/Sub and Dataflow for Stream Processing - Prwatech","og_description":"Master Cloud Pub\/Sub and Dataflow for Stream Processing - Dive deep with our expert instructors and comprehensive curriculum.","og_url":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/","og_site_name":"Prwatech","article_publisher":"https:\/\/www.facebook.com\/prwatech.in\/","article_published_time":"2021-05-31T09:33:24+00:00","article_modified_time":"2024-03-20T06:44:59+00:00","og_image":[{"url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-648.png","type":"","width":"","height":""}],"author":"Prwatech","twitter_card":"summary_large_image","twitter_creator":"@Eduprwatech","twitter_site":"@Eduprwatech","twitter_misc":{"Written by":"Prwatech","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/","url":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/","name":"Cloud Pub\/Sub and Dataflow for Stream Processing - Prwatech","isPartOf":{"@id":"https:\/\/prwatech.in\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/#primaryimage"},"image":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/#primaryimage"},"thumbnailUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-648.png","datePublished":"2021-05-31T09:33:24+00:00","dateModified":"2024-03-20T06:44:59+00:00","author":{"@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3"},"description":"Master Cloud Pub\/Sub and Dataflow for Stream Processing - Dive deep with our expert instructors and comprehensive curriculum.","breadcrumb":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/#primaryimage","url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-648.png","contentUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-648.png","width":628,"height":64},{"@type":"BreadcrumbList","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/pubsub-stream-processing-with-dataflow\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/prwatech.in\/blog\/"},{"@type":"ListItem","position":2,"name":"Pubsub stream processing with dataflow"}]},{"@type":"WebSite","@id":"https:\/\/prwatech.in\/blog\/#website","url":"https:\/\/prwatech.in\/blog\/","name":"Prwatech","description":"Share Ideas, Start Something Good.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/prwatech.in\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3","name":"Prwatech","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","caption":"Prwatech"},"url":"https:\/\/prwatech.in\/blog\/author\/prwatech123\/"}]}},"_links":{"self":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/9397","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/comments?post=9397"}],"version-history":[{"count":4,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/9397\/revisions"}],"predecessor-version":[{"id":10991,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/9397\/revisions\/10991"}],"wp:attachment":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/media?parent=9397"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/categories?post=9397"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/tags?post=9397"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}