{"id":8915,"date":"2021-05-30T05:41:44","date_gmt":"2021-05-30T05:41:44","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=8915"},"modified":"2023-07-13T11:53:24","modified_gmt":"2023-07-13T11:53:24","slug":"working-with-dataproc","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/","title":{"rendered":"Working with Dataproc"},"content":{"rendered":"\r\n

Prerequisites<\/strong><\/p>\r\n\r\n\r\n\r\n

GCP <\/a>account<\/p>\r\n\r\n\r\n\r\n

Open Cloud Console.<\/p>\r\n\r\n\r\n\r\n

Open Menu > Dataproc<\/a> > Clusters<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Click the Cluster.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Click on VM Instances<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Click on SSH of master node<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Check whether the components is already installed.<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pyspark\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #opens pyspark<\/p>\r\n\r\n\r\n\r\n

To exit press ctrl +d<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hive\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To check hive is available or not<\/p>\r\n\r\n\r\n\r\n

To exit press ctrl +d<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 python \u2013V\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #to check python version<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 spark-shell\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # opens spark shell<\/p>\r\n\r\n\r\n\r\n

To exit press ctrl +d<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pwd\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To get path<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 mkdir ratingscounter\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #making directory named ratingscounter<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cd ratingscounter\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Change the directory into ratingscounter<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 wget https:\/\/s3.amazonaws.com\/sankethadoop\/u.data\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To get the data for dataproc<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ls\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Display the contents in the directory<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nano u.data\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #open the u.data file.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

It will display the content in u.data<\/p>\r\n\r\n\r\n\r\n

To exit press ctrl + x<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nano ratingscounter.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Creates and opens file ratingscounter.py<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Paste the below code into ratingscounter.py file<\/p>\r\n\r\n\r\n\r\n

from pyspark import SparkConf, SparkContext<\/p>\r\n\r\n\r\n\r\n

import collections<\/p>\r\n\r\n\r\n\r\n

conf = SparkConf().setMaster(“local”).setAppName(“Ratings”)<\/p>\r\n\r\n\r\n\r\n

sc = SparkContext(conf = conf)<\/p>\r\n\r\n\r\n\r\n

lines = sc.textFile(“sparkdata\/u.data”)<\/p>\r\n\r\n\r\n\r\n

ratings = lines.map(lambda x: x.split( )[2])<\/p>\r\n\r\n\r\n\r\n

result = ratings.countByValue()<\/p>\r\n\r\n\r\n\r\n

sortedResults = collections.OrderedDict(sorted(result.items()))<\/p>\r\n\r\n\r\n\r\n

for key, value in sortedResults.items():<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 print(“%s %i” % (key, value))<\/p>\r\n\r\n\r\n\r\n

This code is to count the films in each ratings.<\/p>\r\n\r\n\r\n\r\n

NB : if you are changing the name of directory or file, you may have to change it in the file lolcation also.<\/p>\r\n\r\n\r\n\r\n

To exit press \u2018ctrl + x\u2019 then press \u2018y\u2019 to confirm then \u2018Enter\u2019<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Create Schema structure<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hadoop fs -mkdir \/user\/<userid>\/sparkdata\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To create directory named sparkdata<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hadoop fs -put u.data sparkdata\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To copy u.data file into sparkdata<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hadoop fs -ls sparkdata\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0# to check the file is saved or not<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 spark-submit ratingscounter.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #execute the ratingscounter.py file<\/p>\r\n\r\n\r\n\r\n

It will display the result.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cd\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #change directory<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 mkdir totalspendbycustomer\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #make directory<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cd totalspendbycustomer\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #change directory to totalspendbycustomer<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 wget https:\/\/s3.amazonaws.com\/sankethadoop\/customer-orders.csv \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To copy file to disk<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 ls\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #list the contents in the directory<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nano customer-orders.csv\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Open the file customer-orders.csv<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Opens the file.<\/p>\r\n\r\n\r\n\r\n

To exit press ctrl +x<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pwd\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Displays the path<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nano totalspendbycustomer.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #creates and opens file totalspendbycustomer.py<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

paste the below code<\/p>\r\n\r\n\r\n\r\n

from pyspark import SparkConf, SparkContext<\/p>\r\n\r\n\r\n\r\n

conf = SparkConf().setMaster(“local”).setAppName(“SpendByCustomer”)<\/p>\r\n\r\n\r\n\r\n

sc = SparkContext(conf = conf)<\/p>\r\n\r\n\r\n\r\n

def extractCustomerPricePairs(line):<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 fields = line.split(‘,’)<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 return (int(fields[0]), float(fields[2]))<\/p>\r\n\r\n\r\n\r\n

input = sc.textFile(“sparkdata\/customer-orders.csv”)<\/p>\r\n\r\n\r\n\r\n

mappedInput = input.map(extractCustomerPricePairs)<\/p>\r\n\r\n\r\n\r\n

totalByCustomer = mappedInput.reduceByKey(lambda x, y: x + y)<\/p>\r\n\r\n\r\n\r\n

results = totalByCustomer.collect();<\/p>\r\n\r\n\r\n\r\n

for result in results:<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0 print(result)<\/p>\r\n\r\n\r\n\r\n

This code is to get the amount spent by the customers for movie<\/p>\r\n\r\n\r\n\r\n

NB : if you are changing the name of directory or file, you may have to change it in the file lolcation also.<\/p>\r\n\r\n\r\n\r\n

To exit press \u2018ctrl + x\u2019 then press \u2018y\u2019 to confirm then \u2018Enter\u2019<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hadoop fs -put customer-orders.csv sparkdata\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #moves file customer-orders.csv into sparkdata<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hadoop fs -ls sparkdata\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # to check the file is saved or not<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 spark-submit totalspendbycustomer.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Execute the file totalspendbycustomer.py<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

It will display the customer ID and total amount spend by customer<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

To find popular movies<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cd\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To Change Directory<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 mkdir popularmovies\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Make directory named popularmovies<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cd popularmovies\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To change directory into popularmovies<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 wget https:\/\/s3.amazonaws.com\/sankethadoop\/u.data \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To copy file to disk<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nano popularmovies.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Open file popularmovies.py<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

In the python file paste the below code.<\/p>\r\n\r\n\r\n\r\n

from pyspark import SparkConf, SparkContext<\/p>\r\n\r\n\r\n\r\n

conf = SparkConf().setMaster(“local”).setAppName(“PopularMovies”)<\/p>\r\n\r\n\r\n\r\n

sc = SparkContext(conf = conf)<\/p>\r\n\r\n\r\n\r\n

lines = sc.textFile(“sparkdata\/u.data”)<\/p>\r\n\r\n\r\n\r\n

movies = lines.map(lambda x: (int(x.split()[1]), 1))<\/p>\r\n\r\n\r\n\r\n

movieCounts = movies.reduceByKey(lambda x, y: x + y)<\/p>\r\n\r\n\r\n\r\n

flipped = movieCounts.map( lambda xy: (xy[1],xy[0]) )<\/p>\r\n\r\n\r\n\r\n

sortedMovies = flipped.sortByKey()<\/p>\r\n\r\n\r\n\r\n

results = sortedMovies.collect()<\/p>\r\n\r\n\r\n\r\n

for result in results:<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0 print(result)<\/p>\r\n\r\n\r\n\r\n

To save and exit, Press \u2018Ctrl + x\u2019 then \u2018y\u2019 then \u2018Enter\u2019<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hadoop fs -put u.data sparkdata\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To copy u.data file into sparkdata<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 spark-submit popularmovies.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 # To execute popularmovies.py<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

It will show the most popular movie ID and most number of votes.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

To find most 10 popular movies<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cd\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To Change Directory<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 mkdir 10popularmovies\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #Make directory named popularmovies<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cd 10popularmovies\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To change directory into popularmovies<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 wget https:\/\/s3.amazonaws.com\/sankethadoop\/u.item<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 wget https:\/\/s3.amazonaws.com\/sankethadoop\/u.data<\/p>\r\n\r\n\r\n\r\n

It will copy the file into disk.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nano u.item\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To open the file content<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

To exit press ctrl+ x<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 nano 10popular.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Create and open the file 10popular.py<\/p>\r\n\r\n\r\n\r\n

Paste the below code.<\/p>\r\n\r\n\r\n\r\n

from pyspark.sql import SparkSession<\/p>\r\n\r\n\r\n\r\n

from pyspark.sql import Row<\/p>\r\n\r\n\r\n\r\n

from pyspark.sql import functions<\/p>\r\n\r\n\r\n\r\n

def loadMovieNames():<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0 movieNames = {}<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0 with open(“\/home\/<userid>\/10popularmovies\/u.item<\/strong>“, encoding=”ISO-8859-1”) as f:<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 for line in f:<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 fields = line.split(‘|’)<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 movieNames[int(fields[0])] = fields [1]<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0 return movieNames<\/p>\r\n\r\n\r\n\r\n

spark = SparkSession.builder.appName(“PopularMovies”).getOrCreate()<\/p>\r\n\r\n\r\n\r\n

nameDict = loadMovieNames()<\/p>\r\n\r\n\r\n\r\n

lines = spark.sparkContext.textFile(“sparkdata\/u.data”)<\/p>\r\n\r\n\r\n\r\n

movies = lines.map(lambda x: Row(movieID =int(x.split()[1])))<\/p>\r\n\r\n\r\n\r\n

movieDataset = spark.createDataFrame(movies)<\/p>\r\n\r\n\r\n\r\n

topMovieIDs = movieDataset.groupBy(“movieID”).count().orderBy(“count”,ascending = False).cache()<\/p>\r\n\r\n\r\n\r\n

topMovieIDs.show()<\/p>\r\n\r\n\r\n\r\n

top10 = topMovieIDs.take(10)<\/p>\r\n\r\n\r\n\r\n

print(“\\n”)<\/p>\r\n\r\n\r\n\r\n

for result in top10:<\/p>\r\n\r\n\r\n\r\n

\u00a0\u00a0\u00a0\u00a0 print(“%s: %d” % (nameDict[result[0]], result[1]))<\/p>\r\n\r\n\r\n\r\n

spark.stop()<\/p>\r\n\r\n\r\n\r\n

Change the highlighted area as your directory<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hadoop fs -put u.data sparkdata\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To copy u.data file into sparkdata<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hadoop fs -ls sparkdata\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To display the content<\/p>\r\n\r\n\r\n\r\n

$\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 spark-submit 10popular.py\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To execute the 10popular.py file<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

It will display the most popular 10 movies.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n","protected":false},"excerpt":{"rendered":"

Prerequisites GCP account Open Cloud Console. Open Menu > Dataproc > Clusters Click the Cluster. Click on VM Instances Click on SSH of master node Check whether the components is already installed. $\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pyspark\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #opens pyspark To exit press ctrl +d $\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hive\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To check hive is available or not To exit press ctrl +d […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1634,1],"tags":[1415,1412,1413,1414,1411,605,699,700,617,683,684,685,611,1400,692],"class_list":["post-8915","post","type-post","status-publish","format-standard","hentry","category-dataproc","category-google-cloud-platform","tag-dataproc","tag-dataproc-cluster","tag-dataproc-cluster-creation","tag-dataproc-cluster-properties","tag-dataproc-in-gcp","tag-gcp","tag-gcp-certification","tag-gcp-cloud-console","tag-google-cloud","tag-google-cloud-certification","tag-google-cloud-console","tag-google-cloud-courses","tag-google-cloud-platform","tag-google-cloud-platform-tutorial","tag-google-cloud-training"],"acf":[],"yoast_head":"\nWorking with Dataproc - Prwatech<\/title>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Working with Dataproc - Prwatech\" \/>\n<meta property=\"og:description\" content=\"Prerequisites GCP account Open Cloud Console. Open Menu > Dataproc > Clusters Click the Cluster. Click on VM Instances Click on SSH of master node Check whether the components is already installed. $\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pyspark\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #opens pyspark To exit press ctrl +d $\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hive\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To check hive is available or not To exit press ctrl +d […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/\" \/>\n<meta property=\"og:site_name\" content=\"Prwatech\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prwatech.in\/\" \/>\n<meta property=\"article:published_time\" content=\"2021-05-30T05:41:44+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-07-13T11:53:24+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-281.png\" \/>\n<meta name=\"author\" content=\"Prwatech\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:site\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Prwatech\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/\",\"url\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/\",\"name\":\"Working with Dataproc - Prwatech\",\"isPartOf\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-281.png\",\"datePublished\":\"2021-05-30T05:41:44+00:00\",\"dateModified\":\"2023-07-13T11:53:24+00:00\",\"author\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\"},\"breadcrumb\":{\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/#primaryimage\",\"url\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-281.png\",\"contentUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-281.png\",\"width\":456,\"height\":339},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/prwatech.in\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Working with Dataproc\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/prwatech.in\/blog\/#website\",\"url\":\"https:\/\/prwatech.in\/blog\/\",\"name\":\"Prwatech\",\"description\":\"Share Ideas, Start Something Good.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/prwatech.in\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\",\"name\":\"Prwatech\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"caption\":\"Prwatech\"},\"url\":\"https:\/\/prwatech.in\/blog\/author\/prwatech123\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Working with Dataproc - Prwatech","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"en_US","og_type":"article","og_title":"Working with Dataproc - Prwatech","og_description":"Prerequisites GCP account Open Cloud Console. Open Menu > Dataproc > Clusters Click the Cluster. Click on VM Instances Click on SSH of master node Check whether the components is already installed. $\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pyspark\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #opens pyspark To exit press ctrl +d $\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 hive\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #To check hive is available or not To exit press ctrl +d […]","og_url":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/","og_site_name":"Prwatech","article_publisher":"https:\/\/www.facebook.com\/prwatech.in\/","article_published_time":"2021-05-30T05:41:44+00:00","article_modified_time":"2023-07-13T11:53:24+00:00","og_image":[{"url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-281.png","type":"","width":"","height":""}],"author":"Prwatech","twitter_card":"summary_large_image","twitter_creator":"@Eduprwatech","twitter_site":"@Eduprwatech","twitter_misc":{"Written by":"Prwatech","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/","url":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/","name":"Working with Dataproc - Prwatech","isPartOf":{"@id":"https:\/\/prwatech.in\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/#primaryimage"},"image":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/#primaryimage"},"thumbnailUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-281.png","datePublished":"2021-05-30T05:41:44+00:00","dateModified":"2023-07-13T11:53:24+00:00","author":{"@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3"},"breadcrumb":{"@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/#primaryimage","url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-281.png","contentUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-281.png","width":456,"height":339},{"@type":"BreadcrumbList","@id":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/prwatech.in\/blog\/"},{"@type":"ListItem","position":2,"name":"Working with Dataproc"}]},{"@type":"WebSite","@id":"https:\/\/prwatech.in\/blog\/#website","url":"https:\/\/prwatech.in\/blog\/","name":"Prwatech","description":"Share Ideas, Start Something Good.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/prwatech.in\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3","name":"Prwatech","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","caption":"Prwatech"},"url":"https:\/\/prwatech.in\/blog\/author\/prwatech123\/"}]}},"_links":{"self":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/8915","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/comments?post=8915"}],"version-history":[{"count":2,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/8915\/revisions"}],"predecessor-version":[{"id":10084,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/8915\/revisions\/10084"}],"wp:attachment":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/media?parent=8915"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/categories?post=8915"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/tags?post=8915"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}