Prerequisites

GCP account

Open Notepad/Text editor in your machine.

Paste the below code.

from pyspark import SparkConf, SparkContext

import collections

conf = SparkConf().setMaster(“local”).setAppName(“Ratings”)

sc = SparkContext(conf = conf)

lines = sc.textFile(“/user/<userid>/sparkdata/u.data”)

ratings = lines.map(lambda x: x.split( )[2])

result = ratings.countByValue()

sortedResults = collections.OrderedDict(sorted(result.items()))

for key, value in sortedResults.items():

    print(“%s %i” % (key, value))

Give your user ID of console.

To get your user ID, Open cloud shell.

$          pwd                #It will show path with your user ID

Save file as ratingscounter.py

Open Cloud Console

Open Cloud Storage > Browser

Upload the created file into bucket.

Click on the file.

Copy the URI of file.

Open Menu > Dataproc > Jobs

Click on Submit Job

Give the Job ID.

Region will be automatically selected.

Choose the cluster.

Choose the job type as PySpark

Paste the URI of python file.

Click Submit.

It will execute the job and give the result.