{"id":1754,"date":"2019-05-24T05:16:46","date_gmt":"2019-05-24T05:16:46","guid":{"rendered":"https:\/\/prwatech.in\/blog\/?p=1754"},"modified":"2025-06-13T06:27:10","modified_gmt":"2025-06-13T06:27:10","slug":"apache-spark-sql-commands","status":"publish","type":"post","link":"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/","title":{"rendered":"Apache Spark SQL Commands"},"content":{"rendered":"

Apache Spark SQL Commands<\/strong><\/h1>\n

 <\/p>\n

Apache Spark SQL Commands<\/strong>, welcome to the world of Apache Spark Basic SQL commands. Are you the one who is looking forward to knowing the Apache Spark SQL commands List which comes under Apache Spark<\/a>? Or the one who is very keen to explore the list of all the SQL commands in Apache Spark with examples that are available? Then you\u2019ve landed on the Right path which provides the standard and Basic Apache Spark SQL Commands.<\/p>\n

If you are the one who is keen to learn the technology then learn the advanced certification course from the best Apache Spark training institute<\/a> who can help guide you about the course from the 0 Level to Advanced level. So don\u2019t just dream to become the certified Pro Developer Achieve it by choosing the best World classes Apache Spark Training Institute in Bangalore<\/a> which consists of World-class Trainers.<\/p>\n

We, Prwatech<\/a> listed some of the Top Apache Spark SQL Commands which Every Spark Developer should know about. So follow the Below Mentioned Apache Spark Basic SQL Commands and Learn the Advanced Apache Spark course<\/a> from the best Spark Trainer like a Pro.<\/p>\n

Spark context(sc) :<\/strong> To initialize the functionalities of Spark SQL<\/p>\n

To create a spark context<\/p>\n

\"Apache<\/p>\n

Check the context created<\/strong><\/h3>\n

\"spark<\/p>\n

Data set:<\/strong> Optimized version of RDD which uses interpreter and optimizer for processing.<\/p>\n

Data Frame:<\/strong> A data frame is a Data set organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R\/Python.<\/p>\n

Data Frame<\/strong><\/h2>\n

\"spark<\/p>\n

\"spark<\/p>\n

\"Apache<\/p>\n

\"scala<\/p>\n

Schema of a table:<\/strong> Using the below-mentioned command we find out the schema of Data set<\/h3>\n

\"readvalue,printschema\"<\/p>\n

Show data:<\/strong> Using below-mentioned command we display the values inside our data set<\/h3>\n

\"readvalue<\/p>\n

\"course<\/p>\n

Reading files using spark session:<\/strong> Using the below-mentioned command we can read data from an external source by providing their path of existence.<\/h3>\n

\"spark.read.json\"<\/p>\n

Show value:<\/strong> To Display the data<\/h3>\n

\"scala<\/p>\n

Selecting a particular column:<\/strong> Using below-mentioned command we can display all the data from a particular column<\/h3>\n

\"df<\/p>\n

Selecting more than one column:<\/strong> Using below-mentioned command we can display all the data from two selected column<\/h3>\n

\"scala<\/strong><\/p>\n

\u00a0<\/strong><\/p>\n

Incrementing the value of column: <\/strong>Using below-mentioned command we can increment the data with the given value<\/h3>\n

\"spark<\/p>\n

Alias:<\/strong> Using the below-mentioned command we can display columns as other names.<\/h3>\n

\"scala<\/p>\n

Filter:<\/strong> Using below-mentioned command we can filter out the value from using different parameters<\/h3>\n

\"scala<\/p>\n

Data frames are also transformational in nature and they are immutable<\/p>\n

Group by<\/strong><\/h2>\n

Various Functions in Group By<\/strong><\/h3>\n

\"<\/p>\n

Count:<\/strong> Using this Group by function to count the given data set.<\/h3>\n

\"df<\/p>\n

\"scala<\/p>\n

Max:<\/strong> Using this Group by function to find out the maximum values for given datasets.<\/h3>\n

\"apache<\/p>\n

Min:<\/strong> Using this Group by function to find out the minimum values for given datasets.<\/h3>\n

\"course<\/p>\n

Average:<\/strong> Using this Group by function to find out the average values for given datasets.<\/h3>\n

\"course<\/p>\n

Write:<\/strong> Using the below-mentioned command we can write and store data on the mentioned location as per user requirement.<\/h3>\n

\"de<\/p>\n

\"course<\/p>\n

To perform query we create a template view<\/strong><\/h3>\n

\"val<\/p>\n

\"spark<\/p>\n

Now we can use command per operations using SQL queries\u00a0<\/strong><\/p>\n

\"val<\/p>\n

Data Set<\/strong><\/h2>\n

Create dataset:<\/strong> Dataset is only created using (seq) object<\/p>\n

Data set with metadata info<\/strong><\/h3>\n

\"class<\/p>\n

\"caseipl<\/p>\n

\"scala<\/p>\n

\"<\/p>\n

Now we can perform SQL query<\/strong><\/h3>\n

\"spark<\/p>\n

UDF Functions<\/strong><\/h3>\n

UDF allows us to register custom functions to call within SQL. These are a very popular way to expose advanced functionality to SQL users in an organization so that these users can call into it without writing code<\/p>\n

Test Case 1: <\/strong>Converting Celsius into Fahrenheit<\/h3>\n

\"val<\/p>\n

\"temperature<\/p>\n

<\/h3>\n

Register UDF:<\/strong> Using the below-mentioned command we are registering new functions as per user need.<\/h3>\n

\"spark<\/p>\n

\"temperture<\/p>\n

\"spark<\/p>\n

\"res65<\/p>\n

Test case 2:<\/strong> Lower to Upper case<\/h3>\n

\"val<\/p>\n

\"val \"import<\/p>\n

\"val<\/p>\n

\"data<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Apache Spark SQL Commands   Apache Spark SQL Commands, welcome to the world of Apache Spark Basic SQL commands. Are you the one who is looking forward to knowing the Apache Spark SQL commands List which comes under Apache Spark? Or the one who is very keen to explore the list of all the SQL […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21,1691],"tags":[295,882,886,881,887,884,885,883,880],"class_list":["post-1754","post","type-post","status-publish","format-standard","hentry","category-apache-spark","category-apache-spark-sql","tag-apache-spark-sql-commands","tag-queries-in-scala","tag-scala-functions","tag-scala-queries","tag-scala-query","tag-scala-sql-query","tag-scala-table-creation","tag-spark-scala-dataset","tag-spark-sql-commands"],"acf":[],"yoast_head":"\nApache Spark SQL Commands | Basic SQL Commands for Apache Spark<\/title>\n<meta name=\"description\" content=\"Here is the list of Apache Spark SQL Commands which are available, Get the best Basic SQL Commands for Apache Spark from Prwatech\" \/>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Spark SQL Commands | Basic SQL Commands for Apache Spark\" \/>\n<meta property=\"og:description\" content=\"Here is the list of Apache Spark SQL Commands which are available, Get the best Basic SQL Commands for Apache Spark from Prwatech\" \/>\n<meta property=\"og:url\" content=\"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/\" \/>\n<meta property=\"og:site_name\" content=\"Prwatech\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prwatech.in\/\" \/>\n<meta property=\"article:published_time\" content=\"2019-05-24T05:16:46+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-06-13T06:27:10+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/05\/1-7.png\" \/>\n<meta name=\"author\" content=\"Prwatech\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:site\" content=\"@Eduprwatech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Prwatech\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"12 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/\",\"url\":\"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/\",\"name\":\"Apache Spark SQL Commands | Basic SQL Commands for Apache Spark\",\"isPartOf\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/05\/1-7.png\",\"datePublished\":\"2019-05-24T05:16:46+00:00\",\"dateModified\":\"2025-06-13T06:27:10+00:00\",\"author\":{\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\"},\"description\":\"Here is the list of Apache Spark SQL Commands which are available, Get the best Basic SQL Commands for Apache Spark from Prwatech\",\"breadcrumb\":{\"@id\":\"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/#primaryimage\",\"url\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/05\/1-7.png\",\"contentUrl\":\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/05\/1-7.png\",\"width\":623,\"height\":61},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/prwatech.in\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Apache Spark SQL Commands\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/prwatech.in\/blog\/#website\",\"url\":\"https:\/\/prwatech.in\/blog\/\",\"name\":\"Prwatech\",\"description\":\"Share Ideas, Start Something Good.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/prwatech.in\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3\",\"name\":\"Prwatech\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g\",\"caption\":\"Prwatech\"},\"url\":\"https:\/\/prwatech.in\/blog\/author\/prwatech123\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Spark SQL Commands | Basic SQL Commands for Apache Spark","description":"Here is the list of Apache Spark SQL Commands which are available, Get the best Basic SQL Commands for Apache Spark from Prwatech","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"en_US","og_type":"article","og_title":"Apache Spark SQL Commands | Basic SQL Commands for Apache Spark","og_description":"Here is the list of Apache Spark SQL Commands which are available, Get the best Basic SQL Commands for Apache Spark from Prwatech","og_url":"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/","og_site_name":"Prwatech","article_publisher":"https:\/\/www.facebook.com\/prwatech.in\/","article_published_time":"2019-05-24T05:16:46+00:00","article_modified_time":"2025-06-13T06:27:10+00:00","og_image":[{"url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/05\/1-7.png","type":"","width":"","height":""}],"author":"Prwatech","twitter_card":"summary_large_image","twitter_creator":"@Eduprwatech","twitter_site":"@Eduprwatech","twitter_misc":{"Written by":"Prwatech","Est. reading time":"12 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/","url":"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/","name":"Apache Spark SQL Commands | Basic SQL Commands for Apache Spark","isPartOf":{"@id":"https:\/\/prwatech.in\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/#primaryimage"},"image":{"@id":"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/#primaryimage"},"thumbnailUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/05\/1-7.png","datePublished":"2019-05-24T05:16:46+00:00","dateModified":"2025-06-13T06:27:10+00:00","author":{"@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3"},"description":"Here is the list of Apache Spark SQL Commands which are available, Get the best Basic SQL Commands for Apache Spark from Prwatech","breadcrumb":{"@id":"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/#primaryimage","url":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/05\/1-7.png","contentUrl":"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2019\/05\/1-7.png","width":623,"height":61},{"@type":"BreadcrumbList","@id":"https:\/\/prwatech.in\/blog\/apache-spark\/apache-spark-sql\/apache-spark-sql-commands\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/prwatech.in\/blog\/"},{"@type":"ListItem","position":2,"name":"Apache Spark SQL Commands"}]},{"@type":"WebSite","@id":"https:\/\/prwatech.in\/blog\/#website","url":"https:\/\/prwatech.in\/blog\/","name":"Prwatech","description":"Share Ideas, Start Something Good.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/prwatech.in\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/db90baff7744090b2288bbc98fea87f3","name":"Prwatech","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/prwatech.in\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c00bafc1b04045f31eda917de39891456c44fa47c092b9bb6be0f860a3a30a2f?s=96&d=mm&r=g","caption":"Prwatech"},"url":"https:\/\/prwatech.in\/blog\/author\/prwatech123\/"}]}},"_links":{"self":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/1754","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/comments?post=1754"}],"version-history":[{"count":15,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/1754\/revisions"}],"predecessor-version":[{"id":11621,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/posts\/1754\/revisions\/11621"}],"wp:attachment":[{"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/media?parent=1754"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/categories?post=1754"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prwatech.in\/blog\/wp-json\/wp\/v2\/tags?post=1754"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}