Cloud Composer and Airflow Integration
Prerequisites
GCP account
Open Console
Open Menu > Cloud Storage > Browser
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-586.png)
Click on Create Bucket
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-587.png)
Create one bucket with same name as the project ID. Click create
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-588.png)
The bucket will be created.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-589.png)
In Composer, Click on Airflow
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-590.png)
Choose the login
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-591.png)
DAG Airflow will be opened.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-592.png)
Go to Menu > Kubernetes Engine > Clusters
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-593.png)
The cluster has been created.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-594.png)
In Airflow, Go to Admin > Variables
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-595.png)
Click on Create.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-596.png)
Key Val
gcp_project <project-ID>
gcs_bucket gs://<bucket-name>
gce_zone <zone of cluster>
Do these one by one in Key and Val. And press Save and Add Another
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-597.png)
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-598.png)
In last one Press save
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-599.png)
The key and Value will be added.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-600.png)
Open Composer.
Click on DAGs Folder
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-601.png)
Copy the path.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-602.png)
Click on Activate Cloud Shell
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-603.png)
Paste the below code in shell. In DAG path paste the copied DAG path and press Enter
$ gsutil cp gs://cloud-training/datawarehousing/lab_assets/hadoop_tutorial.py gs://<paste the DAG path>
It will copy hadoop_tutorial.py into Cluster bucket
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-604.png)
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-605.png)
In Airflow, click on the DAG.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-606.png)
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-607.png)
Hover the curser to each one. You can see the details.
Click any one of it.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-608.png)
Press View Log.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-609.png)
You can see the log for the Execution.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-610.png)
Go to Bucket. Open the Bucket which we created.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-611.png)
file saved.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-612.png)
If it is not Executed, Open the Airflow > composer_hadoop_tutorial.
Trigger DAG
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-613.png)
Trigger
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-614.png)
Graph view. Here you can see the execution.
The below colors shows the execution state.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-615.png)
Its runnning create_dataproc_cluster.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-616.png)
Open Menu > Dataproc > Clusters
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-617.png)
cluster created.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-618.png)
Now the green border is on run_dataproc_hadoop. It is executing the content
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-619.png)
Then it changes to delete_dataproc_cluster. It will delete the cluster.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-620.png)
Check the cluster in dataproc. deleted.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-621.png)
Open the Dataproc > jobs. Open the job.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-622.png)
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-623.png)
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-624.png)
In Airflow click on Code.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-625.png)
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-626.png)
To delete the composer environment , Click on Delete.
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-627.png)
Press Delete
![](https://prwatech.in/blog/wp-content/uploads/2021/05/image-628.png)
Cloud Composer and Airflow Integration