{"id":8956,"date":"2026-09-21T06:09:23","date_gmt":"2026-09-21T06:09:23","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=8956"},"modified":"2026-09-22T16:09:18","modified_gmt":"2026-09-22T16:09:18","slug":"working-with-dataproc-in-console","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/google-cloud-platform\/dataproc\/working-with-dataproc-in-console\/","title":{"rendered":"Working with dataproc in console"},"content":{"rendered":"\r\n

Submitting a PySpark Job Through the Dataproc Console<\/h1>\r\n

Once a script works, you have two ways to actually run it against a Dataproc cluster: from the command line with spark-submit, or by submitting it as a job through the console. This covers the console route, writing a small PySpark script that counts movie ratings, uploading it to Cloud Storage, then submitting it as a job against an existing cluster and watching it run.<\/p>\r\n

This assumes you already have a Dataproc cluster running and the sample data already loaded into HDFS on that cluster, covered in the Working with Dataproc guide in this series.<\/p>\r\n

Before You Start: Getting the Sample Data Onto Your Cluster<\/h2>\r\n

If you have not already done this, SSH into your cluster and run the following to create a folder and load the sample ratings file into it:<\/p>\r\n

hadoop fs -mkdir \/user\/YOUR_USER_ID\/sparkdata
hadoop fs -put u.data sparkdata
hadoop fs -ls sparkdata<\/p>\r\n

The last command confirms the file actually landed where the script below expects to find it.<\/p>\r\n

A complete Step by Step Process of Submitting a PySpark Job Through the Dataproc Console<\/h2>\r\n

Step One: Write the Script<\/h3>\r\n

In a plain text editor on your own machine, paste the following:<\/p>\r\n

from pyspark import SparkConf, SparkContext
import collections

conf = SparkConf().setMaster(“local”).setAppName(“Ratings”)
sc = SparkContext(conf = conf)

lines = sc.textFile(“\/user\/YOUR_USER_ID\/sparkdata\/u.data”)
ratings = lines.map(lambda x: x.split()[2])
result = ratings.countByValue()

sortedResults = collections.OrderedDict(sorted(result.items()))
for key, value in sortedResults.items():
\u00a0\u00a0\u00a0\u00a0print(“%s %i” % (key, value))<\/p>\r\n

This counts how many ratings fall into each score, from the sample data you loaded into HDFS.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Two: Find Your User ID<\/h3>\r\n

Open Cloud Shell and run:<\/p>\r\n

whoami<\/p>\r\n

This gives you your username directly. Running pwd instead also reveals it, since Cloud Shell’s home directory path includes your username, but whoami is the more direct way to get it.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Three: Save the File<\/h3>\r\n

Save it as ratingscounter.py, replacing YOUR_USER_ID in the script with the actual ID from the previous step.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Four: Upload the File to a Bucket<\/h3>\r\n

Open the console, then<\/p>\r\n\r\n\r\n\r\n

Open Cloud Storage > Browser<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Upload the script into a bucket, then click it.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Five: Copy the File’s URI<\/h3>\r\n\r\n\r\n\r\n
\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Six: Open Dataproc Jobs<\/h3>\r\n

Open the menu, then Dataproc, then Jobs.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Seven: Submit a Job<\/h3>\r\n

Click Submit Job.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Eight: Configure the Job<\/h3>\r\n

Give the job an ID. The region fills in automatically, and you choose which cluster to run it against.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Set the job type to PySpark.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Paste the script’s URI into the main Python file field.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Step Nine: Submit and Review<\/h3>\r\n

Click Submit.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

The job runs and returns its result.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Consider Cloud Storage Instead of HDFS for New Work<\/h2>\r\n

This example reads from HDFS, which lives on the cluster itself, and disappears the moment you delete that cluster, the same tradeoff covered in the Dataproc Metastore guide in this series. For anything beyond a quick exercise, reading source data directly from a Cloud Storage path instead, using a gs colon slash slash URI in place of an HDFS path, means the data survives independently of any specific cluster, and multiple clusters can read the same source without copying it around first.<\/p>\r\n

Common Mistakes to Avoid<\/h2>\r\n