{"id":9304,"date":"2026-09-14T05:31:21","date_gmt":"2026-09-14T05:31:21","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=9304"},"modified":"2026-09-15T07:08:04","modified_gmt":"2026-09-15T07:08:04","slug":"working-with-composer-using-airflow","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/","title":{"rendered":"Working with composer using airflow"},"content":{"rendered":"\r\n<h1>Cloud Composer and Airflow Integration<\/h1>\r\n<p>Cloud Composer is Google Cloud&#8217;s managed version of Apache Airflow, used for authoring, scheduling, and monitoring workflows made up of individual tasks called a DAG, short for directed acyclic graph. This walks through deploying a sample DAG, one that creates a Dataproc cluster, runs a Hadoop job on it, then tears the cluster down again, and watching it execute.<\/p>\r\n<h2>Prerequisites<\/h2>\r\n<ul>\r\n<li>A Google Cloud Platform account<\/li>\r\n<li>An existing Cloud Composer environment. Build it on Composer 3 rather than Composer 1, for the reasons above.<\/li>\r\n<\/ul>\r\n<h2>Complete Process of Cloud Composer and Airflow Integration<\/h2>\r\n<h3>Step One: Create a Storage Bucket<\/h3>\r\n<p>Open the console menu and go to Cloud Storage, then Browser.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"392\" height=\"269\" class=\"wp-image-9309\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-586.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-586.png 392w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-586-300x206.png 300w\" sizes=\"auto, (max-width: 392px) 100vw, 392px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Click on Create Bucket<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"372\" height=\"180\" class=\"wp-image-9310\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-587.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-587.png 372w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-587-300x145.png 300w\" sizes=\"auto, (max-width: 372px) 100vw, 372px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Name the bucket the same as your project ID, then click Create.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"497\" height=\"504\" class=\"wp-image-9311\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-588.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-588.png 497w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-588-296x300.png 296w\" sizes=\"auto, (max-width: 497px) 100vw, 497px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>The bucket will be created.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"492\" height=\"277\" class=\"wp-image-9312\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-589.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-589.png 492w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-589-300x169.png 300w\" sizes=\"auto, (max-width: 492px) 100vw, 492px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Two: Open the Airflow Web Interface<\/h3>\r\n<p>From your Composer environment, click Airflow to open its web interface.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"733\" height=\"187\" class=\"wp-image-9313\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-590.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-590.png 733w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-590-300x77.png 300w\" sizes=\"auto, (max-width: 733px) 100vw, 733px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Choose your login when prompted.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"472\" height=\"524\" class=\"wp-image-9314\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-591.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-591.png 472w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-591-270x300.png 270w\" sizes=\"auto, (max-width: 472px) 100vw, 472px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>The Airflow DAGs view opens.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"208\" class=\"wp-image-9315\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-592.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-592.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-592-300x84.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Three: About the Underlying Cluster<\/h3>\r\n<p>On Composer 1, opening the Kubernetes Engine section of the console menu and going to Clusters shows the GKE cluster Composer created for you behind the scenes. This is the step that does not carry over cleanly to Composer 3, since Composer 3 deliberately hides this infrastructure rather than exposing it in your project&#8217;s Kubernetes Engine list. If you are on Composer 1 or an older Composer 2 environment, you can still follow this step as shown.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"447\" height=\"255\" class=\"wp-image-9316\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-593.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-593.png 447w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-593-300x171.png 300w\" sizes=\"auto, (max-width: 447px) 100vw, 447px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>The cluster has been created.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"172\" class=\"wp-image-9317\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-594.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-594.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-594-300x70.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Four: Set Airflow Variables<\/h3>\r\n<p>In Airflow, go to Admin, then Variables.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"734\" height=\"272\" class=\"wp-image-9318\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-595.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-595.png 734w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-595-300x111.png 300w\" sizes=\"auto, (max-width: 734px) 100vw, 734px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Click on Create.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"295\" class=\"wp-image-9319\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-596.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-596.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-596-300x119.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<p><strong>Key\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Val<\/strong><\/p>\r\n\r\n\r\n\r\n<p>gcp_project\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0 &lt;your-project-id&gt;<\/p>\r\n\r\n\r\n\r\n<p>gcs_bucket\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0 gs:\/\/&lt;your-bucket-name&gt;<\/p>\r\n\r\n\r\n\r\n<p>gce_zone\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0 &lt;the zone of your cluster&gt;<\/p>\r\n\r\n\r\n\r\n<p>Do these one by one in Key and Val. And press Save and Add Another<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"593\" class=\"wp-image-9320\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-597.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-597.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-597-300x240.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"606\" class=\"wp-image-9321\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-598.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-598.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-598-300x245.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>On the last one, click Save instead.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"598\" class=\"wp-image-9322\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-599.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-599.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-599-300x242.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>The key and Value will be added.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"215\" class=\"wp-image-9323\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-600.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-600.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-600-300x87.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Five: Copy the Sample DAG Into Your Environment<\/h3>\r\n<p>Open your Composer environment and click the DAGs folder link.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"159\" class=\"wp-image-9324\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-601.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-601.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-601-300x64.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Copy the folder&#8217;s path.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"630\" height=\"279\" class=\"wp-image-9325\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-602.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-602.png 630w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-602-300x133.png 300w\" sizes=\"auto, (max-width: 630px) 100vw, 630px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Open Cloud Shell.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"382\" height=\"139\" class=\"wp-image-9326\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-603.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-603.png 382w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-603-300x109.png 300w\" sizes=\"auto, (max-width: 382px) 100vw, 382px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Run the following, pasting your copied DAG path in place of the placeholder, then press enter:<\/p>\r\n\r\n\r\n\r\n<p>$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 gsutil cp gs:\/\/cloud-training\/datawarehousing\/lab_assets\/hadoop_tutorial.py gs:\/\/&lt;paste the DAG path&gt;<\/p>\r\n\r\n\r\n\r\n<p>This copies the sample DAG file into your cluster&#8217;s bucket.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"127\" class=\"wp-image-9327\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-604.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-604.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-604-300x51.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<div class=\"flex-1 overflow-hidden\">\r\n<div class=\"react-scroll-to-bottom--css-frkrv-79elbk h-full\">\r\n<div class=\"react-scroll-to-bottom--css-frkrv-1n7m0yu\">\r\n<div class=\"flex flex-col text-sm pb-9\">\r\n<div class=\"w-full text-token-text-primary\" data-testid=\"conversation-turn-23\">\r\n<div class=\"px-4 py-2 justify-center text-base md:gap-6 m-auto\">\r\n<div class=\"flex flex-1 text-base mx-auto gap-3 md:px-5 lg:px-1 xl:px-5 md:max-w-3xl lg:max-w-[40rem] xl:max-w-[48rem] group final-completion\">\r\n<div class=\"relative flex w-full flex-col agent-turn\">\r\n<div class=\"flex-col gap-1 md:gap-3\">\r\n<div class=\"flex flex-grow flex-col max-w-full\">\r\n<div class=\"min-h-[20px] text-message flex flex-col items-start gap-3 whitespace-pre-wrap break-words [.text-message+&amp;]:mt-5 overflow-x-auto\" data-message-author-role=\"assistant\" data-message-id=\"2c3c081e-cf49-4d3e-b614-c9f2c5ca60a8\">\r\n<div class=\"markdown prose w-full break-words dark:prose-invert light\">\r\n<p>The file also shows up inside Airflow shortly after.<\/p>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"285\" class=\"wp-image-9328\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-605.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-605.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-605-300x115.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Six: Explore the DAG<\/h3>\r\n<p>In Airflow, click the DAG.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"147\" class=\"wp-image-9329\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-606.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-606.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-606-300x60.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<div class=\"flex-1 overflow-hidden\">\r\n<div class=\"react-scroll-to-bottom--css-frkrv-79elbk h-full\">\r\n<div class=\"react-scroll-to-bottom--css-frkrv-1n7m0yu\">\r\n<div class=\"flex flex-col text-sm pb-9\">\r\n<div class=\"w-full text-token-text-primary\" data-testid=\"conversation-turn-25\">\r\n<div class=\"px-4 py-2 justify-center text-base md:gap-6 m-auto\">\r\n<div class=\"flex flex-1 text-base mx-auto gap-3 md:px-5 lg:px-1 xl:px-5 md:max-w-3xl lg:max-w-[40rem] xl:max-w-[48rem] group final-completion\">\r\n<div class=\"relative flex w-full flex-col agent-turn\">\r\n<div class=\"flex-col gap-1 md:gap-3\">\r\n<div class=\"flex flex-grow flex-col max-w-full\">\r\n<div class=\"min-h-[20px] text-message flex flex-col items-start gap-3 whitespace-pre-wrap break-words [.text-message+&amp;]:mt-5 overflow-x-auto\" data-message-author-role=\"assistant\" data-message-id=\"3b6bacbb-1f7e-45c1-9ea9-fcd25140a940\">\r\n<div class=\"markdown prose w-full break-words dark:prose-invert light\">\r\n<p>Please click on the Graph View. You will see a graph-like structure.<\/p>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"380\" class=\"wp-image-9330\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-607.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-607.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-607-300x154.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Hover the curser to each one. You can see the details.<\/p>\r\n\r\n\r\n\r\n<p>Click any one of it.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"582\" height=\"246\" class=\"wp-image-9331\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-608.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-608.png 582w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-608-300x127.png 300w\" sizes=\"auto, (max-width: 582px) 100vw, 582px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Press View Log.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"546\" height=\"478\" class=\"wp-image-9332\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-609.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-609.png 546w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-609-300x263.png 300w\" sizes=\"auto, (max-width: 546px) 100vw, 546px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>You can see the log for the Execution.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"270\" class=\"wp-image-9333\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-610.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-610.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-610-300x109.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Seven: Check the Bucket<\/h3>\r\n<p>Open the bucket you created earlier.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"501\" height=\"280\" class=\"wp-image-9334\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-611.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-611.png 501w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-611-300x168.png 300w\" sizes=\"auto, (max-width: 501px) 100vw, 501px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>The file has been saved there.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"656\" height=\"288\" class=\"wp-image-9335\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-612.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-612.png 656w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-612-300x132.png 300w\" sizes=\"auto, (max-width: 656px) 100vw, 656px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Eight: Trigger the DAG<\/h3>\r\n<p>If the DAG has not run yet, open Airflow and find composer_hadoop_tutorial, then click Trigger DAG.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"277\" class=\"wp-image-9336\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-613.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-613.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-613-300x112.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Trigger<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"562\" height=\"226\" class=\"wp-image-9337\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-614.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-614.png 562w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-614-300x121.png 300w\" sizes=\"auto, (max-width: 562px) 100vw, 562px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Nine: Watch It Run<\/h3>\r\n<p>In the graph view, colors show each task&#8217;s current state as the DAG executes.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"663\" height=\"59\" class=\"wp-image-9338\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-615.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-615.png 663w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-615-300x27.png 300w\" sizes=\"auto, (max-width: 663px) 100vw, 663px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Its runnning create_dataproc_cluster.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"600\" height=\"344\" class=\"wp-image-9339\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-616.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-616.png 600w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-616-300x172.png 300w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Open Menu &gt; Dataproc &gt; Clusters<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"456\" height=\"339\" class=\"wp-image-9340\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-617.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-617.png 456w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-617-300x223.png 300w\" sizes=\"auto, (max-width: 456px) 100vw, 456px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>cluster\u00a0 created.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"186\" class=\"wp-image-9341\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-618.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-618.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-618-300x75.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Now the green border is on run_dataproc_hadoop. It is executing the content<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"646\" height=\"332\" class=\"wp-image-9342\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-619.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-619.png 646w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-619-300x154.png 300w\" sizes=\"auto, (max-width: 646px) 100vw, 646px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Then it changes to delete_dataproc_cluster. It will delete the cluster.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"591\" height=\"340\" class=\"wp-image-9343\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-620.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-620.png 591w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-620-300x173.png 300w\" sizes=\"auto, (max-width: 591px) 100vw, 591px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Check the cluster in dataproc. deleted.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"587\" height=\"199\" class=\"wp-image-9344\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-621.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-621.png 587w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-621-300x102.png 300w\" sizes=\"auto, (max-width: 587px) 100vw, 587px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Ten: Review the Job Output<\/h3>\r\n<p>In Dataproc, open Jobs, then open the job that ran.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"592\" height=\"186\" class=\"wp-image-9345\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-622.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-622.png 592w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-622-300x94.png 300w\" sizes=\"auto, (max-width: 592px) 100vw, 592px\" \/><\/figure>\r\n\r\n\r\n\r\n<div class=\"flex-1 overflow-hidden\">\r\n<div class=\"react-scroll-to-bottom--css-frkrv-79elbk h-full\">\r\n<div class=\"react-scroll-to-bottom--css-frkrv-1n7m0yu\">\r\n<div class=\"flex flex-col text-sm pb-9\">\r\n<div class=\"w-full text-token-text-primary\" data-testid=\"conversation-turn-29\">\r\n<div class=\"px-4 py-2 justify-center text-base md:gap-6 m-auto\">\r\n<div class=\"flex flex-1 text-base mx-auto gap-3 md:px-5 lg:px-1 xl:px-5 md:max-w-3xl lg:max-w-[40rem] xl:max-w-[48rem] group final-completion\">\r\n<div class=\"relative flex w-full flex-col agent-turn\">\r\n<div class=\"flex-col gap-1 md:gap-3\">\r\n<div class=\"flex flex-grow flex-col max-w-full\">\r\n<div class=\"min-h-[20px] text-message flex flex-col items-start gap-3 whitespace-pre-wrap break-words [.text-message+&amp;]:mt-5 overflow-x-auto\" data-message-author-role=\"assistant\" data-message-id=\"dca5ba9c-a459-4e69-af46-777103272e05\">\r\n<div class=\"markdown prose w-full break-words dark:prose-invert light\">\r\n<p>The output file itself is saved back into your bucket.<\/p>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"605\" height=\"743\" class=\"wp-image-9346\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-623.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-623.png 605w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-623-244x300.png 244w\" sizes=\"auto, (max-width: 605px) 100vw, 605px\" \/><\/figure>\r\n\r\n\r\n\r\n<div class=\"flex-1 overflow-hidden\">\r\n<div class=\"react-scroll-to-bottom--css-frkrv-79elbk h-full\">\r\n<div class=\"react-scroll-to-bottom--css-frkrv-1n7m0yu\">\r\n<div class=\"flex flex-col text-sm pb-9\">\r\n<div class=\"w-full text-token-text-primary\" data-testid=\"conversation-turn-27\">\r\n<div class=\"px-4 py-2 justify-center text-base md:gap-6 m-auto\">\r\n<div class=\"flex flex-1 text-base mx-auto gap-3 md:px-5 lg:px-1 xl:px-5 md:max-w-3xl lg:max-w-[40rem] xl:max-w-[48rem] group final-completion\">\r\n<div class=\"relative flex w-full flex-col agent-turn\">\r\n<div class=\"flex-col gap-1 md:gap-3\">\r\n<div class=\"flex flex-grow flex-col max-w-full\">\r\n<div class=\"min-h-[20px] text-message flex flex-col items-start gap-3 whitespace-pre-wrap break-words [.text-message+&amp;]:mt-5 overflow-x-auto\" data-message-author-role=\"assistant\" data-message-id=\"1469b91f-430a-4d35-a4f8-1e27ba43ad36\">\r\n<div class=\"markdown prose w-full break-words dark:prose-invert light\">\r\n<p>The system will save the file in the bucket.<\/p>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n<\/div>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"656\" height=\"288\" class=\"wp-image-9347\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-624.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-624.png 656w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-624-300x132.png 300w\" sizes=\"auto, (max-width: 656px) 100vw, 656px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Eleven: Review the DAG&#8217;s Code<\/h3>\r\n<p>Back in Airflow, click Code to see the DAG definition that produced this whole run.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"277\" class=\"wp-image-9348\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-625.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-625.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-625-300x112.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n<p>&nbsp;<\/p>\r\n\r\n\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"703\" class=\"wp-image-9349\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-626.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-626.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-626-300x285.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<h3>Step Twelve: Clean Up<\/h3>\r\n<p>To avoid ongoing charges, delete the Composer environment once you are finished. Click Delete.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"141\" class=\"wp-image-9350\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-627.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-627.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-627-300x57.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n\r\n\r\n\r\n<p>Confirm by pressing Delete again.<\/p>\r\n\r\n\r\n\r\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"741\" height=\"229\" class=\"wp-image-9351\" src=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-628.png\" alt=\"\" srcset=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-628.png 741w, https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-628-300x93.png 300w\" sizes=\"auto, (max-width: 741px) 100vw, 741px\" \/><\/figure>\r\n<p>&nbsp;<\/p>\r\n<h2>What Changes on Cloud Composer 3<\/h2>\r\n<ul>\r\n<li>Hidden infrastructure. The GKE cluster backing your environment is not exposed in your project&#8217;s Kubernetes Engine list the way it was on Composer 1, so Step Three above will not show you anything there.<\/li>\r\n<li>Simplified networking. Composer 3 reduces the number of networking options you need to configure directly when creating an environment.<\/li>\r\n<li>Evergreen versioning. Infrastructure updates and security patches are applied automatically, rather than requiring a manual environment upgrade.<\/li>\r\n<\/ul>\r\n<p>The Airflow concepts, DAGs, tasks, variables, and triggering, work the same way regardless of which Composer version sits underneath them, since Airflow itself has not changed, only how Google manages the infrastructure running it.<\/p>\r\n<h2>Common Mistakes to Avoid<\/h2>\r\n<ul>\r\n<li>Building a new environment on Composer 1 today. It is past or approaching its end of life, so any new work should start on Composer 3 instead.<\/li>\r\n<li>Expecting to find your environment&#8217;s cluster in the Kubernetes Engine console on Composer 3. Look at the Composer environment details instead, since the cluster itself is intentionally not shown there anymore.<\/li>\r\n<\/ul>\r\n<p>Mistyping a variable name. The sample DAG expects the exact variable names gcp_project, gcs_bucket, and gce_zone. A typo in any of them will cause the DAG to fail when it tries to read a variable that does not exist.<\/p>\r\n<p>That covers deploying, triggering, and monitoring an Airflow DAG in Cloud Composer. To go further, explore <a href=\"https:\/\/prwatech.in\/gcp-training-institutes-in-bangalore\/\"><strong><b>Prwatech&#8217;s Google Cloud training<\/b><\/strong>\u00a0program<\/a>, which includes placement assistance.<\/p>\r\n","protected":false},"excerpt":{"rendered":"<p>Cloud Composer and Airflow Integration Cloud Composer is Google Cloud&#8217;s managed version of Apache Airflow, used for authoring, scheduling, and monitoring workflows made up of individual tasks called a DAG, short for directed acyclic graph. This walks through deploying a sample DAG, one that creates a Dataproc cluster, runs a Hadoop job on it, then [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1632,1],"tags":[1519,1520,1521,1522,605,699,700,617,683,684,685,611,1400,686,692],"class_list":["post-9304","post","type-post","status-publish","format-standard","hentry","category-bigquery","category-google-cloud-platform","tag-cloud-composer","tag-cloud-composer-dag","tag-cloud-composer-documentation","tag-cloud-composer-environment","tag-gcp","tag-gcp-certification","tag-gcp-cloud-console","tag-google-cloud","tag-google-cloud-certification","tag-google-cloud-console","tag-google-cloud-courses","tag-google-cloud-platform","tag-google-cloud-platform-tutorial","tag-google-cloud-storage","tag-google-cloud-training"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Cloud Composer and Airflow Integration: DAGs, Composer 3<\/title>\n<meta name=\"description\" content=\"Learn Cloud Composer and Airflow Integration with Prwatech. Trigger and monitor an Airflow DAG in Cloud Composer, updated for Composer 1 &amp; move to Composer 3.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Cloud Composer and Airflow Integration: DAGs, Composer 3\" \/>\n<meta property=\"og:description\" content=\"Learn Cloud Composer and Airflow Integration with Prwatech. Trigger and monitor an Airflow DAG in Cloud Composer, updated for Composer 1 &amp; move to Composer 3.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/\" \/>\n<meta property=\"og:site_name\" content=\"Prwatech\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prwatech.in\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-09-14T05:31:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-09-15T07:08:04+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-586.png\" \/>\n\t<meta property=\"og:image:width\" content=\"392\" \/>\n\t<meta property=\"og:image:height\" content=\"269\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Prwatech\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:site\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Prwatech\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"13 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/\",\"url\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/\",\"name\":\"Cloud Composer and Airflow Integration: DAGs, Composer 3\",\"isPartOf\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-586.png\",\"datePublished\":\"2026-09-14T05:31:21+00:00\",\"dateModified\":\"2026-09-15T07:08:04+00:00\",\"author\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\"},\"description\":\"Learn Cloud Composer and Airflow Integration with Prwatech. Trigger and monitor an Airflow DAG in Cloud Composer, updated for Composer 1 & move to Composer 3.\",\"breadcrumb\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/#primaryimage\",\"url\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-586.png\",\"contentUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-586.png\",\"width\":392,\"height\":269},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/prwatech.in\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Working with composer using airflow\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/prwatech.in\/blog\/#website\",\"url\":\"https:\/\/prwatech.in\/blog\/\",\"name\":\"Prwatech\",\"description\":\"Share Ideas, Start Something Good.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/prwatech.in\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\",\"name\":\"Prwatech\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"caption\":\"Prwatech\"},\"url\":\"https:\/\/prwatech.in\/blog\/author\/prwatech123\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Cloud Composer and Airflow Integration: DAGs, Composer 3","description":"Learn Cloud Composer and Airflow Integration with Prwatech. Trigger and monitor an Airflow DAG in Cloud Composer, updated for Composer 1 & move to Composer 3.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/","og_locale":"en_US","og_type":"article","og_title":"Cloud Composer and Airflow Integration: DAGs, Composer 3","og_description":"Learn Cloud Composer and Airflow Integration with Prwatech. Trigger and monitor an Airflow DAG in Cloud Composer, updated for Composer 1 & move to Composer 3.","og_url":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/","og_site_name":"Prwatech","article_publisher":"https:\/\/www.facebook.com\/prwatech.in\/","article_published_time":"2026-09-14T05:31:21+00:00","article_modified_time":"2026-09-15T07:08:04+00:00","og_image":[{"width":392,"height":269,"url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-586.png","type":"image\/png"}],"author":"Prwatech","twitter_card":"summary_large_image","twitter_creator":"@Eduprwatech","twitter_site":"@Eduprwatech","twitter_misc":{"Written by":"Prwatech","Est. reading time":"13 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/","url":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/","name":"Cloud Composer and Airflow Integration: DAGs, Composer 3","isPartOf":{"@id":"https:\/\/prwatech.in\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/#primaryimage"},"image":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/#primaryimage"},"thumbnailUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-586.png","datePublished":"2026-09-14T05:31:21+00:00","dateModified":"2026-09-15T07:08:04+00:00","author":{"@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3"},"description":"Learn Cloud Composer and Airflow Integration with Prwatech. Trigger and monitor an Airflow DAG in Cloud Composer, updated for Composer 1 & move to Composer 3.","breadcrumb":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/#primaryimage","url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-586.png","contentUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-586.png","width":392,"height":269},{"@type":"BreadcrumbList","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/bigquery\/working-with-composer-using-airflow\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/prwatech.in\/blog\/"},{"@type":"ListItem","position":2,"name":"Working with composer using airflow"}]},{"@type":"WebSite","@id":"https:\/\/prwatech.in\/blog\/#website","url":"https:\/\/prwatech.in\/blog\/","name":"Prwatech","description":"Share Ideas, Start Something Good.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/prwatech.in\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3","name":"Prwatech","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","caption":"Prwatech"},"url":"https:\/\/prwatech.in\/blog\/author\/prwatech123\/"}]}},"_links":{"self":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/9304","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/comments?post=9304"}],"version-history":[{"count":6,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/9304\/revisions"}],"predecessor-version":[{"id":11771,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/9304\/revisions\/11771"}],"wp:attachment":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/media?parent=9304"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/categories?post=9304"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/tags?post=9304"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}