Scala Commands for Spark

This is a working reference to the core Scala language commands every Spark developer runs into early on, expressions, values, functions, classes, loops, and collections, each with a short explanation and example.

Expressions

An expression is simply a computable statement. Typed into the Scala shell, it evaluates immediately and returns a result.

scala> 12 * 10
res0: Int = 120

Printing Output With println

You can print the result of an expression directly using println.

scala> println(“Hello, ” + “Prwatech”)

Values

The val keyword names the result of an expression. Referencing a value later does not recompute it, it stays fixed.

Here such as  ‘x’ is value “Referencing a value doesn’t re-compute it”

Variables

Variables behave like values, except you can reassign them. Define one with the var keyword.

var x = 6 + 4
x = 10

Blocks

Surrounding a group of expressions with curly braces combines them into a block. The result of the last expression inside the block becomes the result of the block as a whole.

Functions

Functions

Functions are expressions that take parameters. You can define an anonymous function, one with no name, using arrow syntax, where the parameter list sits to the left of the arrow and the expression using it sits to the right.

On the left of =,> is a list of parameters. On the right is an expression involving the parameters.

You can also name functions.

Functions can also be given a name.

A function may take multiple parameters.

Or it can take no parameters at all.

Methods

Methods look and behave much like functions, with a few key differences. A method is defined with the def keyword, and its return type is declared after the parameter list, following a colon.

A method can also take multiple parameter lists, one set of parentheses after another.

Classes

You can define classes with the class keyword followed by its name and constructor parameters.

Objects

Objects are single instances of their own definitions. You can think of them as singletons of their own classes.

You can define objects with the object keyword.

Singleton Objects

Scala classes do not have a static keyword, so a singleton object fills that role instead. Its constructor runs the first time the object is used, it carries all the features of a class, and its constructor cannot take parameters. A singleton object can be imported from anywhere in the program.

Singleton objects are commonly used in a few situations:

• When a single instance is needed to coordinate a service.
• When one immutable instance can be shared across the program for efficiency.
• When an immutable instance is needed for utility functions or constants.

If Else Conditions

var a = 8
val s = if (a > 1 && a < 9) 1 else 0

var a = 9
val s = if (a > 1 && a < 9) 1 else 0

 

 

While Loop

var args = “PRWATECH”
var i = 0
while (i < args.length) {
println(args(i))
i += 1
}

 

 

Foreach Loop

var args = “SCALA”
args.foreach(println)

For Loop

for (i <- 1 to 5) print(i)

for (i <- 11 to 25) print(i)

Nested For Loop

for (i <- 1 to 5; j = i) println(20 * i + j)

How to write a function

Define the function first, then call it with a value.

def area(radius: Int): Double = { 7.20 * radius * radius }

Argument to function

def concatstr(arg1: String, arg2: String = “Institute”, arg3: String = “Prwatech”) = println(arg1 + arg2 + arg3)

concatstr(“BigData”)
concatstr(“BigData”, arg3 = “spark”)
concatstr(“BigData”)

Parameters can be given default values, so a caller only needs to supply the ones they want to override, such as arg3 in the second call above.

   

Procedures: Functions Without a Return Value

Scala has a special kind of function that returns no value. If a function is written without a preceding equals sign, its return type becomes Unit, and a function like this is called a procedure. Procedures never return a value in Scala.

def rect_area(length: Float, breadth: Float) { val area = length * breadth; println(area) }

Scala Collections

Scala has a rich collections library. The most commonly used are Array, ArrayBuffer, Map, Tuple, and List.

Array

An array is a collection of elements or records of the same type.

var i = Array(“Hello”, “Prwatech”)
print(i(0))
print(i(1))

 

Array Buffers

An ArrayBuffer is like an array, but it can grow or shrink in size, unlike a fixed size Array. You bring it in with an import statement before using it.

import scala.collection.mutable.ArrayBuffer

var buffer = ArrayBuffer(“Hello”, “Prwatech”)
buffer += “Spark”
print(buffer)

Maps

A Map is a collection of key and value pairs, similar to a Java HashMap or a Python dictionary. You look up a value by its key rather than by position.

var colors = Map(“red” -> “#FF0000”, “yellow” -> “#FFFF00”)
print(colors(“yellow”))

Tuples

A Tuple holds a fixed group of values that can be of different types in the same tuple, unlike an Array, and it is immutable. You access each element with an underscore followed by its position, starting from 1.

var t = (15, “Prwatech”, true)
print(t._1)
print(t._2)

Lists

A List is an immutable, ordered collection of elements of the same type, similar to an Array, but a List cannot be resized once created.

var l = List(“Hello”, “Prwatech”, “Spark”)
print(l(0))

Common Mistakes to Avoid

• Forgetting that a procedure has no equals sign. A method written without an equals sign before its braces returns Unit, meaning no value, even if the body computes something. Add the equals sign back if you actually need the result returned.
• Reassigning a val. A val is meant to stay fixed. If your logic genuinely needs to change a value later, use var instead of working around an immutable val.
• Confusing a function with a method. They look similar, but a method is defined with def and can carry multiple parameter lists, while a function is more often used as a value you can pass around, such as an anonymous function.
• Reaching for a mutable ArrayBuffer out of habit. If the size of your collection will not change, a plain List or Array is simpler and safer than an ArrayBuffer.

That covers the core Scala commands used in Spark development, from expressions and functions through classes, loops, and collections. To go further, explore Prwatech’s Apache Spark training program, which includes placement assistance.