Prerequisites

GCP account

Open Console

Open Menu > Cloud Storage > Browser

Click on Create Bucket

Create one bucket with same name as the project ID. Click create

The bucket will be created.

In Composer, Click on Airflow

Choose the login

DAG Airflow will be opened.

Go to Menu > Kubernetes Engine > Clusters

The cluster will be created.

In Airflow, Go to Admin > Variables

Click on Create.

Key                                        Val

gcp_project               <project-ID>

gcs_bucket                 gs://<bucket-name>

gce_zone                    <zone of cluster>

Do these one by one in Key and Val. And press Save and Add Another

In last one Press save

The key and Value will be added.

Open Composer.

Click on DAGs Folder

Copy the path.

Click on Activate Cloud Shell

Paste the below code in shell. In DAG path paste the copied DAG path and press Enter

$          gsutil cp gs://cloud-training/datawarehousing/lab_assets/hadoop_tutorial.py gs://<paste the DAG path>

It will copy hadoop_tutorial.py  into Cluster bucket

The file will be copied into cluster bucket as well as it will displayed in  Airfow

In Airflow, click on the DAG. It will be already executed

Click on Graph View. You can see the graph alike structure.

Hover the curser to each one. You can see the details.

Click any one of it.

Press View Log.

You can see the log for the Execution.

Go to Bucket. Open the Bucket which we created.

The file will be saved in it.

If it is not Executed, Open the Airflow > composer_hadoop_tutorial.

Click on Trigger DAG

Click on Trigger

Click on Graph view. Here you can see the execution.

The below colors shows the execution state.

Its runnning create_dataproc_cluster.

Open Menu > Dataproc > Clusters

Here you can see the cluster is getting created.

Now the green border is on run_dataproc_hadoop. It is executing the content

Then it changes to delete_dataproc_cluster. It will delete the cluster.

Check the cluster in dataproc. The created cluster will be deleted.

Open the Dataproc > jobs. Open the job.

See where the file is saved.

The file will be saved in the bucket

In Airflow click on Code.

You can see the code which is used for execution

To delete the composer environment , Click on Delete.

Press Delete