{"id":8045,"date":"2021-04-09T10:15:10","date_gmt":"2021-04-09T10:15:10","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=8045"},"modified":"2024-03-20T06:37:50","modified_gmt":"2024-03-20T06:37:50","slug":"etl-processing-on-google-cloud-using-dataflow-and-big-query","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/","title":{"rendered":"ETL Processing on Google Cloud Using Dataflow and Big Query"},"content":{"rendered":"\r\n

Dataflow and BigQuery for ETL on Google Cloud<\/span><\/h2>\r\n

 <\/p>\r\n

Prerequisites<\/strong><\/h3>\r\n\r\n\r\n\r\n

Hardware\u00a0 :\u00a0 GCP<\/a><\/h3>\r\n\r\n\r\n\r\n

Google account<\/p>\r\n\r\n\r\n\r\n

Open the console.<\/p>\r\n\r\n\r\n\r\n

Click on activate cloud shell.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n

 <\/p>\r\n<\/div>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 gcloud config list project\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To List Projects<\/a><\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 gcloud config set project my-demo-project-306417\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To Select Project<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

In the Cloud console, click\u00a0Menu > IAM & Admin.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Confirm that the default compute Service Account\u00a0is present and has the\u00a0editor\u00a0role assigned.<\/p>\r\n\r\n\r\n\r\n

{project-number}-compute@developer.gserviceaccount.com<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

To check the project number, Go to Menu > Home > Dashboard<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

In the dashboard under the project info project number will be displayed.<\/h4>\r\n\r\n\r\n\r\n
\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

In the console, copy the below code.<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 gsutil -m cp -R gs:\/\/spls\/gsp290\/dataflow-python-examples .\u00a0\u00a0\u00a0 # To copy the \u00a0Dataflow Python Examples from\u00a0Google Cloud’s professional services in GitHub<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Export the name of your cloud project as an environment variable.<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 export PROJECT=<YOUR-PROJECT-ID><\/p>\r\n\r\n\r\n\r\n

Select the project which has been exported. [Don\u2019t change $PROJECT]<\/p>\r\n\r\n\r\n\r\n

\u00a0 gcloud config set project $PROJECT<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0gsutil mb -c regional -l us-central1 gs:\/\/$PROJECT\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To create new Bucket<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0 gsutil cp gs:\/\/spls\/gsp290\/data_files\/usa_names.csv gs:\/\/$PROJECT\/data_files\/\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To copy the files into the bucket<\/a> which we created<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0 bq mk lake\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To make Big Query Lake<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Go to the directory<\/p>\r\n\r\n\r\n\r\n

\u00a0cd dataflow-python-examples\/<\/p>\r\n\r\n\r\n\r\n

Install Virtual Environment<\/p>\r\n\r\n\r\n\r\n

sudo pip install virtualenv<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0 \u00a0virtualenv -p python3 venv\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Creating Virtual Environement<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0source venv\/bin\/activate\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Activating Virtual Environment<\/p>\r\n\r\n\r\n\r\n

\u00a0 pip install apache-beam[gcp]==2.24.0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Installing Apache-beam<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

In the Cloud Shell window, Click on Open Editor<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

In the Workspace, the dataflow-python-examples folder will be copied. Go through each python programs and try to understand the code.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Here we’ll run the Dataflow pipeline in cloud. It will add the workers required, and shut them down when complete<\/h3>\r\n\r\n\r\n\r\n

\u00a0 \u00a0python dataflow_python_examples\/data_ingestion.py –project=$PROJECT –region=us-central1 –runner=DataflowRunner –staging_location=gs:\/\/$PROJECT\/test –temp_location gs:\/\/$PROJECT\/test –input gs:\/\/$PROJECT\/data_files\/usa_names.csv –save_main_session<\/p>\r\n\r\n\r\n\r\n

It will take some time to assign the workers and to finish the job assigned.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Open the console and click Menu > Dataflow > Jobs<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

In Jobs, Check the status. It will show Succeeded after completion. It will show running if the work is not fully assigned.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0Big Query > SQL Workspace<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

It will show the usa_names table under the lake dataset.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Here we’ll run the Dataflow pipeline saved in data_transformation.py file. It will add the workers which is required, and shut it down when complete.<\/p>\r\n\r\n\r\n\r\n

\u00a0 \u00a0python dataflow_python_examples\/data_transformation.py –project=$PROJECT –region=us-central1 –runner=DataflowRunner –staging_location=gs:\/\/$PROJECT\/test –temp_location gs:\/\/$PROJECT\/test –input gs:\/\/$PROJECT\/data_files\/head_usa_names.csv –save_main_session<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0Cloud Shell Editor, open dataflow-python-examples > dataflow_python_examples > data_enrichment.py<\/p>\r\n\r\n\r\n\r\n

In Line 83 replace<\/p>\r\n\r\n\r\n\r\n

values = [x.decode(‘utf8’) for x in csv_row]<\/p>\r\n\r\n\r\n\r\n

With\u00a0\u00a0\u00a0 $\u00a0\u00a0\u00a0\u00a0\u00a0 values = [x for x in csv_row]<\/p>\r\n\r\n\r\n\r\n

This code will populate the data in BigQuery.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n
\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Now again we’ll run the Dataflow pipeline saved in data_enrichment.py. It will add the workers, and shut down when finished.<\/p>\r\n\r\n\r\n\r\n

python dataflow_python_examples\/data_enrichment.py –project=$PROJECT –region=us-central1 –runner=DataflowRunner –staging_location=gs:\/\/$PROJECT\/test –temp_location gs:\/\/$PROJECT\/test –input gs:\/\/$PROJECT\/data_files\/head_usa_names.csv –save_main_session<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Open Menu > Big Query > SQL Workspace<\/p>\r\n\r\n\r\n\r\n

It will show the populated dataset in lake.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

\u00a0Dataflow > Jobs.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

Click on the job which you have done.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n\r\n\r\n\r\n

It will show the Data Flow pipeline of the work.<\/p>\r\n\r\n\r\n\r\n

\r\n
\"\"<\/figure>\r\n<\/div>\r\n

Dataflow and BigQuery for ETL on Google Cloud<\/span><\/h2>\r\n

 <\/p>\r\n","protected":false},"excerpt":{"rendered":"

Dataflow and BigQuery for ETL on Google Cloud   Prerequisites Hardware\u00a0 :\u00a0 GCP Google account Open the console. Click on activate cloud shell. $\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 gcloud config list project\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To List Projects $\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 gcloud config set project my-demo-project-306417\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To Select Project In the Cloud console, click\u00a0Menu > IAM & Admin. Confirm that the default compute Service […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1632,1636,1],"tags":[975,605,641,699,700,711,683,684,685,611],"class_list":["post-8045","post","type-post","status-publish","format-standard","hentry","category-bigquery","category-dataflow","category-google-cloud-platform","tag-etl-processing","tag-gcp","tag-gcp-bucket","tag-gcp-certification","tag-gcp-cloud-console","tag-gcp-instance","tag-google-cloud-certification","tag-google-cloud-console","tag-google-cloud-courses","tag-google-cloud-platform"],"acf":[],"yoast_head":"\nDataflow and BigQuery for ETL on Google Cloud - Prwatech<\/title>\n<meta name=\"description\" content=\"Master Dataflow and BigQuery for ETL on Google Cloud with Prwatech - Dive deep with our expert instructors and comprehensive curriculum.\" \/>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Dataflow and BigQuery for ETL on Google Cloud - Prwatech\" \/>\n<meta property=\"og:description\" content=\"Master Dataflow and BigQuery for ETL on Google Cloud with Prwatech - Dive deep with our expert instructors and comprehensive curriculum.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/\" \/>\n<meta property=\"og:site_name\" content=\"Prwatech\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prwatech.in\/\" \/>\n<meta property=\"article:published_time\" content=\"2021-04-09T10:15:10+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-03-20T06:37:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/04\/1-8.jpg\" \/>\n<meta name=\"author\" content=\"Prwatech\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:site\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Prwatech\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/\",\"url\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/\",\"name\":\"Dataflow and BigQuery for ETL on Google Cloud - Prwatech\",\"isPartOf\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/04\/1-8.jpg\",\"datePublished\":\"2021-04-09T10:15:10+00:00\",\"dateModified\":\"2024-03-20T06:37:50+00:00\",\"author\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\"},\"description\":\"Master Dataflow and BigQuery for ETL on Google Cloud with Prwatech - Dive deep with our expert instructors and comprehensive curriculum.\",\"breadcrumb\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/#primaryimage\",\"url\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/04\/1-8.jpg\",\"contentUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/04\/1-8.jpg\",\"width\":506,\"height\":215},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/prwatech.in\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"ETL Processing on Google Cloud Using Dataflow and Big Query\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/prwatech.in\/blog\/#website\",\"url\":\"https:\/\/prwatech.in\/blog\/\",\"name\":\"Prwatech\",\"description\":\"Share Ideas, Start Something Good.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/prwatech.in\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\",\"name\":\"Prwatech\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"caption\":\"Prwatech\"},\"url\":\"https:\/\/prwatech.in\/blog\/author\/prwatech123\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Dataflow and BigQuery for ETL on Google Cloud - Prwatech","description":"Master Dataflow and BigQuery for ETL on Google Cloud with Prwatech - Dive deep with our expert instructors and comprehensive curriculum.","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"en_US","og_type":"article","og_title":"Dataflow and BigQuery for ETL on Google Cloud - Prwatech","og_description":"Master Dataflow and BigQuery for ETL on Google Cloud with Prwatech - Dive deep with our expert instructors and comprehensive curriculum.","og_url":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/","og_site_name":"Prwatech","article_publisher":"https:\/\/www.facebook.com\/prwatech.in\/","article_published_time":"2021-04-09T10:15:10+00:00","article_modified_time":"2024-03-20T06:37:50+00:00","og_image":[{"url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/04\/1-8.jpg","type":"","width":"","height":""}],"author":"Prwatech","twitter_card":"summary_large_image","twitter_creator":"@Eduprwatech","twitter_site":"@Eduprwatech","twitter_misc":{"Written by":"Prwatech","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/","url":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/","name":"Dataflow and BigQuery for ETL on Google Cloud - Prwatech","isPartOf":{"@id":"https:\/\/prwatech.in\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/#primaryimage"},"image":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/#primaryimage"},"thumbnailUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/04\/1-8.jpg","datePublished":"2021-04-09T10:15:10+00:00","dateModified":"2024-03-20T06:37:50+00:00","author":{"@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3"},"description":"Master Dataflow and BigQuery for ETL on Google Cloud with Prwatech - Dive deep with our expert instructors and comprehensive curriculum.","breadcrumb":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/#primaryimage","url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/04\/1-8.jpg","contentUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/04\/1-8.jpg","width":506,"height":215},{"@type":"BreadcrumbList","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataflow\/etl-processing-on-google-cloud-using-dataflow-and-big-query\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/prwatech.in\/blog\/"},{"@type":"ListItem","position":2,"name":"ETL Processing on Google Cloud Using Dataflow and Big Query"}]},{"@type":"WebSite","@id":"https:\/\/prwatech.in\/blog\/#website","url":"https:\/\/prwatech.in\/blog\/","name":"Prwatech","description":"Share Ideas, Start Something Good.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/prwatech.in\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3","name":"Prwatech","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","caption":"Prwatech"},"url":"https:\/\/prwatech.in\/blog\/author\/prwatech123\/"}]}},"_links":{"self":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/8045","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/comments?post=8045"}],"version-history":[{"count":12,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/8045\/revisions"}],"predecessor-version":[{"id":10989,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/8045\/revisions\/10989"}],"wp:attachment":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/media?parent=8045"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/categories?post=8045"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/tags?post=8045"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}