{"id":8755,"date":"2026-09-16T11:18:11","date_gmt":"2026-09-16T11:18:11","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=8755"},"modified":"2026-09-17T06:34:34","modified_gmt":"2026-09-17T06:34:34","slug":"data-fusion-with-bigquery","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/","title":{"rendered":"Data Fusion with BigQuery"},"content":{"rendered":"\r\n<h1>Cloud Data Fusion and BigQuery<\/h1>\r\n<p class=\"wp-block-paragraph\">Cloud Data Fusion is Google Cloud&#8217;s fully managed, visual data integration service, built on the open source CDAP project. Instead of writing code, you build a data pipeline by connecting boxes on a canvas, each one a source, a transformation, or a destination, and Data Fusion handles running it as an actual Spark job underneath.<\/p>\r\n<p>This walks through a real pipeline: cleaning up a raw CSV of New York taxi trips using Data Fusion&#8217;s Wrangler tool, joining it against a BigQuery public dataset to turn numeric zone IDs into readable zone names and boroughs, and writing the joined result back into BigQuery as a new table.<\/p>\r\n<h2>Step by Step Process of Cloud Data Fusion and BigQuery<\/h2>\r\n<h3>Step One: Prepare Your Files in Cloud Shell<\/h3>\r\n<p>Open the console and activate Cloud Shell.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"230\" height=\"95\" class=\"wp-image-8771\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-156.png\" alt=\"\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Run the following, one line at a time:<\/p>\r\n<p>echo $GOOGLE_CLOUD_PROJECT<br \/><br \/>gsutil cp gs:\/\/cloud-training\/OCBL017\/ny-taxi-2018-sample.csv gs:\/\/$GOOGLE_CLOUD_PROJECT<br \/><br \/>gsutil mb gs:\/\/$GOOGLE_CLOUD_PROJECT-temp<\/p>\r\n<p>The first line confirms your project ID. The second copies a sample taxi trip CSV from a public training bucket into your own project&#8217;s default bucket. The third creates a second, temporary bucket, which Data Fusion will use later to stage data during the pipeline run.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-8772 alignnone\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-157.png\" alt=\"\" width=\"722\" height=\"136\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-157.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-157-300x57.png 300w\" sizes=\"auto, (max-width: 722px) 100vw, 722px\" \/><\/figure>\r\n<p>Confirm both buckets by opening the menu, then Cloud Storage, then Browser.<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Open\u00a0 Menu &gt; Cloud Storage &gt; Browser<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"394\" height=\"190\" class=\"wp-image-8773\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-158.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-158.png 394w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-158-300x145.png 300w\" sizes=\"auto, (max-width: 394px) 100vw, 394px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">The temporary\u00a0bucket is created and the file is copied in main bucket.<\/p>\r\n\r\n\r\n\r\n<h3>Step Two: Open Data Fusion and Start Wrangler<\/h3>\r\n<p class=\"wp-block-paragraph\">Open the menu, then Data Fusion, then click View Instance.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"614\" height=\"182\" class=\"wp-image-8775\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-160.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-160.png 614w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-160-300x89.png 300w\" sizes=\"auto, (max-width: 614px) 100vw, 614px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Dismiss the welcome prompt by clicking &#8220;No thanks&#8221;.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"442\" class=\"wp-image-8776\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-161.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-161.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-161-300x211.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click Wrangler, the tool used to clean and reshape raw data before it enters a pipeline.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"437\" class=\"wp-image-8777\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-162.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-162.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-162-300x209.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n<p>Under Default Cloud Storage, open the bucket you copied the CSV into.<\/p>\r\n\r\n\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"303\" class=\"wp-image-8778\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-163.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-163.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-163-300x145.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Select the copied file.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"275\" class=\"wp-image-8779\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-164.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-164.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-164-300x131.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Three: Clean the Data in Wrangler<\/h3>\r\n<p class=\"wp-block-paragraph\">Click the dropdown arrow on the loaded column.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"348\" height=\"309\" class=\"wp-image-8780\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-165.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-165.png 348w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-165-300x266.png 300w\" sizes=\"auto, (max-width: 348px) 100vw, 348px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click Parse &gt; CSV<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"443\" height=\"387\" class=\"wp-image-8781\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-166.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-166.png 443w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-166-300x262.png 300w\" sizes=\"auto, (max-width: 443px) 100vw, 443px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Select comma and tick set first row as header.<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click Apply<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"525\" height=\"376\" class=\"wp-image-8782\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-167.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-167.png 525w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-167-300x215.png 300w\" sizes=\"auto, (max-width: 525px) 100vw, 525px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">In drop down of body column, Click delete column<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"265\" height=\"501\" class=\"wp-image-8783\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-168.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-168.png 265w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-168-159x300.png 159w\" sizes=\"auto, (max-width: 265px) 100vw, 265px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">In Trip distance drop down &gt; change data type to Float<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"494\" height=\"425\" class=\"wp-image-8784\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-169.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-169.png 494w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-169-300x258.png 300w\" sizes=\"auto, (max-width: 494px) 100vw, 494px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">\u00a0open Filter &gt; Custom condition<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"454\" height=\"514\" class=\"wp-image-8785\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-170.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-170.png 454w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-170-265x300.png 265w\" sizes=\"auto, (max-width: 454px) 100vw, 454px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Give Condition as &gt;0.0<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click Apply<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"426\" height=\"280\" class=\"wp-image-8786\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-171.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-171.png 426w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-171-300x197.png 300w\" sizes=\"auto, (max-width: 426px) 100vw, 426px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">In right side of table, we can see the transformations which we did in our table<\/p>\r\n\r\n\r\n\r\n<h4>Step Four: Turn Wrangler Into a Pipeline<\/h4>\r\n<p class=\"wp-block-paragraph\">Click Create a pipeline.<\/p>\r\n<p>&nbsp;<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"491\" height=\"344\" class=\"wp-image-8787\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-172.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-172.png 491w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-172-300x210.png 300w\" sizes=\"auto, (max-width: 491px) 100vw, 491px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Choose Batch Pipeline, since this runs once on a fixed dataset rather than continuously on streaming data.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"119\" class=\"wp-image-8788\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-173.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-173.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-173-300x57.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">A pipeline canvas opens with a Wrangler node already placed, carrying every transformation you just defined.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"557\" height=\"182\" class=\"wp-image-8789\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-174.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-174.png 557w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-174-300x98.png 300w\" sizes=\"auto, (max-width: 557px) 100vw, 557px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Opening its properties confirms the same transformations are recorded here.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"423\" height=\"194\" class=\"wp-image-8790\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-175.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-175.png 423w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-175-300x138.png 300w\" sizes=\"auto, (max-width: 423px) 100vw, 423px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">On the right, the output schema is shown. Delete the column named extra, which is not needed downstream.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"467\" height=\"449\" class=\"wp-image-8791\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-176.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-176.png 467w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-176-300x288.png 300w\" sizes=\"auto, (max-width: 467px) 100vw, 467px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click Validate to check the node for configuration errors.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"503\" height=\"214\" class=\"wp-image-8792\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-177.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-177.png 503w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-177-300x128.png 300w\" sizes=\"auto, (max-width: 503px) 100vw, 503px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Close the validation panel once it confirms no errors were found.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"372\" height=\"109\" class=\"wp-image-8793\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-178.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-178.png 372w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-178-300x88.png 300w\" sizes=\"auto, (max-width: 372px) 100vw, 372px\" \/><\/figure>\r\n<h3>Step Five: Prepare the BigQuery Side of the Join<\/h3>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Open \u00a0Console &gt; BigQuery<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"273\" height=\"214\" class=\"wp-image-8794\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-179.png\" alt=\"\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click Create Dataset.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"101\" class=\"wp-image-8795\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-180.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-180.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-180-300x48.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Give dataset name. Click Create Dataset.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"485\" height=\"434\" class=\"wp-image-8796\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-181.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-181.png 485w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-181-300x268.png 300w\" sizes=\"auto, (max-width: 485px) 100vw, 485px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click More, then Query Settings.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"173\" class=\"wp-image-8797\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-182.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-182.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-182-300x83.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Tick Set destination table for query results, and give it your project, the dataset you just created, and a table name. Choose Write if empty or Overwrite table, so reruns do not fail on a table that already exists, then click Save.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"497\" height=\"302\" class=\"wp-image-8798\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-183.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-183.png 497w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-183-300x182.png 300w\" sizes=\"auto, (max-width: 497px) 100vw, 497px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click save<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"487\" height=\"260\" class=\"wp-image-8799\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-184.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-184.png 487w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-184-300x160.png 300w\" sizes=\"auto, (max-width: 487px) 100vw, 487px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">In the query editor, paste the following, which pulls zone reference data from a BigQuery public dataset:<\/p>\r\n<p>SELECT<br \/>\u00a0\u00a0zone_id,<br \/>\u00a0\u00a0zone_name,<br \/>\u00a0\u00a0borough<br \/>FROM<br \/>\u00a0\u00a0bigquery-public-data.new_york_taxi_trips.taxi_zone_geom<\/p>\r\n<p>Click Run.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"610\" height=\"344\" class=\"wp-image-8800\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-185.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-185.png 610w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-185-300x169.png 300w\" sizes=\"auto, (max-width: 610px) 100vw, 610px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">The result is written into the destination table you configured.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"427\" height=\"459\" class=\"wp-image-8801\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-186.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-186.png 427w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-186-279x300.png 279w\" sizes=\"auto, (max-width: 427px) 100vw, 427px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Six: Add BigQuery as a Second Source in the Pipeline<\/h3>\r\n<p class=\"wp-block-paragraph\">Back in Data Fusion, add a Source node, then choose BigQuery.<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">\u00a0Source &gt; BigQuery<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"304\" height=\"326\" class=\"wp-image-8802\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-187.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-187.png 304w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-187-280x300.png 280w\" sizes=\"auto, (max-width: 304px) 100vw, 304px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Open its properties in BigQuery.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"589\" height=\"396\" class=\"wp-image-8803\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-188.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-188.png 589w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-188-300x202.png 300w\" sizes=\"auto, (max-width: 589px) 100vw, 589px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Give it a reference name, then the dataset and table name you just created, and click Get Schema.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"458\" height=\"521\" class=\"wp-image-8804\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-189.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-189.png 458w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-189-264x300.png 264w\" sizes=\"auto, (max-width: 458px) 100vw, 458px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">The output schema appears on the right.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"623\" height=\"188\" class=\"wp-image-8805\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-190.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-190.png 623w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-190-300x91.png 300w\" sizes=\"auto, (max-width: 623px) 100vw, 623px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Provide the temporary bucket name you created earlier in Cloud Shell.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"254\" height=\"93\" class=\"wp-image-8806\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-191.png\" alt=\"\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click on Validate.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"503\" height=\"214\" class=\"wp-image-8807\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-192.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-192.png 503w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-192-300x128.png 300w\" sizes=\"auto, (max-width: 503px) 100vw, 503px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Close the panel once no errors are found.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"372\" height=\"109\" class=\"wp-image-8808\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-193.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-193.png 372w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-193-300x88.png 300w\" sizes=\"auto, (max-width: 372px) 100vw, 372px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Seven: Join the Two Data Streams<\/h3>\r\n<p class=\"wp-block-paragraph\">Add an Analytics node, then choose Joiner.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"615\" height=\"307\" class=\"wp-image-8809\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-194.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-194.png 615w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-194-300x150.png 300w\" sizes=\"auto, (max-width: 615px) 100vw, 615px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Drag a connecting arrow from both the Wrangler node and the BigQuery source node into the Joiner.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"269\" class=\"wp-image-8810\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-195.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-195.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-195-300x129.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"258\" class=\"wp-image-8811\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-196.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-196.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-196-300x123.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click on properties in joiner<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"251\" class=\"wp-image-8812\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-197.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-197.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-197-300x120.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Jointype : inner<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\"><strong>Join condition<\/strong><\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Wrangler : pickup_location_id<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">BigQuery : zone_id<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click on Get Schema.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"287\" class=\"wp-image-8813\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-198.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-198.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-198-300x137.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">In output schema remove pickup_location_id &amp; zone_id<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"470\" height=\"511\" class=\"wp-image-8814\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-199.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-199.png 470w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-199-276x300.png 276w\" sizes=\"auto, (max-width: 470px) 100vw, 470px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">remove zone_id.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"478\" height=\"485\" class=\"wp-image-8815\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-200.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-200.png 478w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-200-296x300.png 296w\" sizes=\"auto, (max-width: 478px) 100vw, 478px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Validate.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"503\" height=\"214\" class=\"wp-image-8816\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-201.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-201.png 503w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-201-300x128.png 300w\" sizes=\"auto, (max-width: 503px) 100vw, 503px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">close if no errors found.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"372\" height=\"109\" class=\"wp-image-8817\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-202.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-202.png 372w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-202-300x88.png 300w\" sizes=\"auto, (max-width: 372px) 100vw, 372px\" \/><\/figure>\r\n<h3>Step Eight: Send the Joined Result to BigQuery<\/h3>\r\n<p>Add a Sink node, then choose BigQuery.<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Sink &gt; bigQuery<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"176\" class=\"wp-image-8818\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-203.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-203.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-203-300x84.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Open its properties in second big query.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"225\" class=\"wp-image-8819\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-204.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-204.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-204-300x107.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Give it a reference name, a dataset and table name for the final joined output, and the same temporary bucket name used earlier.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"383\" class=\"wp-image-8820\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-205.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-205.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-205-300x183.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click on Validate.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"494\" height=\"112\" class=\"wp-image-8821\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-206.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-206.png 494w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-206-300x68.png 300w\" sizes=\"auto, (max-width: 494px) 100vw, 494px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">close if no errors found<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"372\" height=\"109\" class=\"wp-image-8822\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-207.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-207.png 372w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-207-300x88.png 300w\" sizes=\"auto, (max-width: 372px) 100vw, 372px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Now drag the arrow from Joiner<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"202\" class=\"wp-image-8823\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-208.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-208.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-208-300x96.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Nine: Deploy and Run the Pipeline<\/h3>\r\n<p class=\"wp-block-paragraph\">Give the pipeline a name, then click Save.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"277\" height=\"187\" class=\"wp-image-8824\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-209.png\" alt=\"\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click Save.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"430\" height=\"176\" class=\"wp-image-8825\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-210.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-210.png 430w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-210-300x123.png 300w\" sizes=\"auto, (max-width: 430px) 100vw, 430px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click Deploy.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"455\" height=\"179\" class=\"wp-image-8826\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-211.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-211.png 455w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-211-300x118.png 300w\" sizes=\"auto, (max-width: 455px) 100vw, 455px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">It will deploy pipeline<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"621\" height=\"182\" class=\"wp-image-8827\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-212.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-212.png 621w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-212-300x88.png 300w\" sizes=\"auto, (max-width: 621px) 100vw, 621px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Click Run. A full execution typically takes around 10 to 15 minutes, since Data Fusion provisions an actual Spark cluster to run it.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"231\" class=\"wp-image-8828\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-213.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-213.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-213-300x110.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">It will Start Execution. This execution will take arround 10-15 minutes.<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Execution is Provisioning<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"312\" class=\"wp-image-8829\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-214.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-214.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-214-300x149.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">\u00a0Starting<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"73\" class=\"wp-image-8830\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-215.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-215.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-215-300x35.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Running<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"101\" class=\"wp-image-8831\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-216.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-216.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-216-300x48.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">\u00a0Finished Successfully.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"265\" class=\"wp-image-8832\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-217.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-217.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-217-300x127.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Ten: Confirm the Result<\/h3>\r\n<p class=\"wp-block-paragraph\">In BigQuery, the destination dataset now has a new table holding the joined, cleaned data.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"219\" class=\"wp-image-8833\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-218.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-218.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-218-300x105.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Back in Data Fusion, click Logs to review the full execution log for the run.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"628\" height=\"501\" class=\"wp-image-8834\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-219.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-219.png 628w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-219-300x239.png 300w\" sizes=\"auto, (max-width: 628px) 100vw, 628px\" \/><\/figure>\r\n<h2>Common Mistakes to Avoid<\/h2>\r\n<ul>\r\n<li>Forgetting to click Get Schema after changing a join condition or removing columns. The schema shown in each node has to be refreshed manually whenever the fields feeding into it change, or later nodes will validate against an outdated schema.<\/li>\r\n<li>Reusing the wrong bucket for temporary storage. The temporary bucket used by BigQuery source and sink nodes needs to be one your Data Fusion service account can actually write to, which is why the dedicated temp bucket created in Cloud Shell matters.<\/li>\r\n<li>Choosing Write if empty instead of Overwrite table when rerunning a query or pipeline during testing. Write if empty will fail once the destination table already has data in it from a previous run.<\/li>\r\n<li>Skipping validation on a node before connecting the next one. Catching a schema mismatch at the node where it happened is far faster than tracing it back after the whole pipeline fails partway through a 10 minute run.<\/li>\r\n<\/ul>\r\n<p>That covers building a full Data Fusion pipeline that cleans, joins, and writes data back into BigQuery. To go further, explore <a href=\"https:\/\/prwatech.in\/gcp-training-institutes-in-pune\/\"><strong><b>Prwatech&#8217;s Google Cloud training<\/b><\/strong><\/a>\u00a0program, which includes placement assistance.<\/p>\r\n<p>&nbsp;<\/p>\r\n","protected":false},"excerpt":{"rendered":"<p>Cloud Data Fusion and BigQuery Cloud Data Fusion is Google Cloud&#8217;s fully managed, visual data integration service, built on the open source CDAP project. Instead of writing code, you build a data pipeline by connecting boxes on a canvas, each one a source, a transformation, or a destination, and Data Fusion handles running it as [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1632,1],"tags":[728,1401,1403,732,1402,1408,1407,1405,1406,605,729,641,699,700,984,617,683,684,685,611,1400,692],"class_list":["post-8755","post","type-post","status-publish","format-standard","hentry","category-bigquery","category-google-cloud-platform","tag-bigquery","tag-bigquery-console","tag-bigquery-documentation","tag-bigquery-sql","tag-bigquery-tutorial","tag-data-fusion-documentation","tag-data-fusion-examples","tag-data-fusion-techniques","tag-data-fusion-tutorial","tag-gcp","tag-gcp-bigquery","tag-gcp-bucket","tag-gcp-certification","tag-gcp-cloud-console","tag-gcp-course","tag-google-cloud","tag-google-cloud-certification","tag-google-cloud-console","tag-google-cloud-courses","tag-google-cloud-platform","tag-google-cloud-platform-tutorial","tag-google-cloud-training"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Cloud Data Fusion and BigQuery: Building a Join Pipeline<\/title>\n<meta name=\"description\" content=\"Build a Cloud Data Fusion pipeline that cleans a CSV in Wrangler, joins it against BigQuery, and writes the result back, step by step process\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Cloud Data Fusion and BigQuery: Building a Join Pipeline\" \/>\n<meta property=\"og:description\" content=\"Build a Cloud Data Fusion pipeline that cleans a CSV in Wrangler, joins it against BigQuery, and writes the result back, step by step process\" \/>\n<meta property=\"og:url\" content=\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/\" \/>\n<meta property=\"og:site_name\" content=\"Prwatech\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prwatech.in\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-09-16T11:18:11+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-09-17T06:34:34+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-156.png\" \/>\n\t<meta property=\"og:image:width\" content=\"230\" \/>\n\t<meta property=\"og:image:height\" content=\"95\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Prwatech\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:site\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Prwatech\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"18 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/\",\"url\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/\",\"name\":\"Cloud Data Fusion and BigQuery: Building a Join Pipeline\",\"isPartOf\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-156.png\",\"datePublished\":\"2026-09-16T11:18:11+00:00\",\"dateModified\":\"2026-09-17T06:34:34+00:00\",\"author\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\"},\"description\":\"Build a Cloud Data Fusion pipeline that cleans a CSV in Wrangler, joins it against BigQuery, and writes the result back, step by step process\",\"breadcrumb\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/#primaryimage\",\"url\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-156.png\",\"contentUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-156.png\",\"width\":230,\"height\":95},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/prwatech.in\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Data Fusion with BigQuery\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/prwatech.in\/blog\/#website\",\"url\":\"https:\/\/prwatech.in\/blog\/\",\"name\":\"Prwatech\",\"description\":\"Share Ideas, Start Something Good.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/prwatech.in\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\",\"name\":\"Prwatech\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"caption\":\"Prwatech\"},\"url\":\"https:\/\/prwatech.in\/blog\/author\/prwatech123\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Cloud Data Fusion and BigQuery: Building a Join Pipeline","description":"Build a Cloud Data Fusion pipeline that cleans a CSV in Wrangler, joins it against BigQuery, and writes the result back, step by step process","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/","og_locale":"en_US","og_type":"article","og_title":"Cloud Data Fusion and BigQuery: Building a Join Pipeline","og_description":"Build a Cloud Data Fusion pipeline that cleans a CSV in Wrangler, joins it against BigQuery, and writes the result back, step by step process","og_url":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/","og_site_name":"Prwatech","article_publisher":"https:\/\/www.facebook.com\/prwatech.in\/","article_published_time":"2026-09-16T11:18:11+00:00","article_modified_time":"2026-09-17T06:34:34+00:00","og_image":[{"width":230,"height":95,"url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-156.png","type":"image\/png"}],"author":"Prwatech","twitter_card":"summary_large_image","twitter_creator":"@Eduprwatech","twitter_site":"@Eduprwatech","twitter_misc":{"Written by":"Prwatech","Est. reading time":"18 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/","url":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/","name":"Cloud Data Fusion and BigQuery: Building a Join Pipeline","isPartOf":{"@id":"https:\/\/prwatech.in\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/#primaryimage"},"image":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/#primaryimage"},"thumbnailUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-156.png","datePublished":"2026-09-16T11:18:11+00:00","dateModified":"2026-09-17T06:34:34+00:00","author":{"@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3"},"description":"Build a Cloud Data Fusion pipeline that cleans a CSV in Wrangler, joins it against BigQuery, and writes the result back, step by step process","breadcrumb":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/#primaryimage","url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-156.png","contentUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-156.png","width":230,"height":95},{"@type":"BreadcrumbList","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/data-fusion-with-bigquery\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/prwatech.in\/blog\/"},{"@type":"ListItem","position":2,"name":"Data Fusion with BigQuery"}]},{"@type":"WebSite","@id":"https:\/\/prwatech.in\/blog\/#website","url":"https:\/\/prwatech.in\/blog\/","name":"Prwatech","description":"Share Ideas, Start Something Good.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/prwatech.in\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3","name":"Prwatech","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","caption":"Prwatech"},"url":"https:\/\/prwatech.in\/blog\/author\/prwatech123\/"}]}},"_links":{"self":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/8755","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/comments?post=8755"}],"version-history":[{"count":6,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/8755\/revisions"}],"predecessor-version":[{"id":11781,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/8755\/revisions\/11781"}],"wp:attachment":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/media?parent=8755"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/categories?post=8755"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/tags?post=8755"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}