Blog

GoLang– Break

How to break out of nested loops in Go

 

Breaking out of nested loops in Go involves using control flow mechanisms to exit from multiple nested loop levels prematurely based on certain conditions. Go does not have built-in support for labeled break statements like some other languages, but there are alternative approaches to achieve this.

One common technique is to use a boolean flag variable that controls the outer loop execution based on an inner loop condition. By setting the flag when the desired condition is met in the inner loop, the outer loop can check this flag and break out accordingly.

Another approach involves restructuring the nested loops into separate functions and using return statements to exit from the entire loop structure based on specific conditions.

Use Case 1:

package main  

import “fmt”  

func main() {  

   var  a int = 1  

   for a < 10{  

      fmt.Print(“Value of a is “,a,”\n”)  

      a++;  

      if a > 5{  

         /* terminate the loop using break statement */  

         break;  

 

Output:

Use Case 2:

package main  

import “fmt”  

func main() {  

   var a int  

   var b int  

   for a = 1; a <= 3; a++ {  

      for b = 1; b <= 3; b++ {  

fmt.Println(“Prwatech”)

         if (a == 2 && b == 2) {  

            break;  

 

         fmt.Print(a, ” “, b, “\n”)  

     

Output:

Use Case 3:

package main  

import “fmt”  

func main() {  

   var a int  

   var b int  

   for a = 1; a <= 3; a++ {  

      for b = 1; b <= 3; b++ { 

        if (a % 2 == 0) {  

            break;  

 

         fmt.Print(a, ” “, b, “\n”)  

 

Output:

0
0

GoLang Map

Creating Maps in GoLang: Step-by-Step Tutorial

 

A Map is a type of data structure, it is a collection of unorder pairs of key-value. With the help of map we can easily do retrieve, update and insert operations.

Syntax:

var mymap map[keytype]valuetype 

Use Case 1:

package main

import “fmt”

func main() {

            var mymap map[int]int

            if mymap == nil {

                        fmt.Println(“true”)

            } else {

                        fmt.Println(“False”)

            }

}

Output:

Use Case 2:

package main

import “fmt”

func main() {

            var mymap map[int]int

            if mymap == nil {

                        fmt.Println(“true”)

            } else {

                        fmt.Println(“False”)

            }

            map2 := map[int]string{

                        10: “Prwatech”,11: “Bangalore”,

            }

            fmt.Println(“Map 2 value”, map2)

}

Output:

Use Case 3: Create a map using make() function.

package main

import “fmt”

func main() {

            // Creating a map

            // Using make() function

            var map1 = make(map[int]string)

            fmt.Println(map1)

            // As we already know that make() function

            / always returns a map which is initialized

            // So, we can add values in it

            map1[1] = “Prwatech”

            map1[2] = “Bangalore”

            fmt.Println(map1)

}

Output:

Use Case 4: Adding new key elements and update operation:

package main

import “fmt”

// Main function

func main() {

            // Creating and initializing a map

            m_a_p := map[int]string{

                        90: “Prwatech”,

                        91: “Bangalore”,

                        92: “Big Data”,

            }

            fmt.Println(“Original map: “, m_a_p)

            // Adding new key-value pairs in the map

            m_a_p[93] = “Python”

            m_a_p[94] = “Golang”

            fmt.Println(“Map after adding new key-value pair:\n”, m_a_p)

            // Updating values of the map

            m_a_p[93] = “Gcp”

            fmt.Println(“\nMap after updating values of the map:\n”, m_a_p)

}

Output:

Use Case 5:Retrieving values with the help of keys:

package main

import “fmt”

// Main function

func main() {

            // Creating and initializing a map

            m_a_p := map[int]string{

                        90: “Red”,

                        91: “Yellow”,

                        92: “Orange”,

                        93: “Blue”,

                        94: “White”,

            }

            fmt.Println(“Original map: “, m_a_p)

            // Retrieving values with the help of keys

            value_1 := m_a_p[90]

            value_2 := m_a_p[93]

            fmt.Println(“Value of key[90]: “, value_1)

            fmt.Println(“Value of key[93]: “, value_2)

}

Output:

Use Case 6:Deleting keys

package main

import “fmt”

// Main function

func main() {

            // Creating and initializing a map

            m_a_p := map[int]string{

                        90: “Red”,

                        91: “Yellow”,

                        92: “Orange”,

                        93: “Blue”,

                        94: “White”,

            }

            fmt.Println(“Original map: “, m_a_p)

            // Deleting keys

            // Using delete function

            delete(m_a_p, 90)

            delete(m_a_p, 93)

            fmt.Println(“Map after deletion: “, m_a_p)

}

Output:

 

0
0

Creating Scheduled Query in BigQuery

Automate analysis with BigQuery scheduled queries

 

Scheduled queries in BigQuery represent a valuable feature for automating and optimizing recurring data processing tasks within Google’s cloud-based data warehouse platform. This introduction will delve into the fundamentals of creating scheduled queries in BigQuery, empowering users to leverage automation for seamless execution of data transformations, reports generation, and other periodic operations.

Scheduled queries enable users to define SQL queries or Data Definition Language (DDL) statements and schedule them to run at predefined intervals, ranging from minutes to months. This automation eliminates the need for manual intervention, ensuring timely execution of critical data tasks without the hassle of manual execution.

The process of creating a scheduled query involves specifying the SQL code to be executed, setting the desired execution schedule, and configuring optional parameters such as destination table for query results, notification preferences, and error handling mechanisms. This streamlined workflow enables users to define complex data processing pipelines with ease, streamlining operations and reducing overhead.

Scheduled queries in BigQuery offer several benefits, including improved operational efficiency, enhanced data timeliness, and simplified management of recurring tasks. Whether it’s generating daily reports, updating aggregations, or performing data validations, scheduled queries provide a robust mechanism for automating repetitive data workflows, freeing up valuable time and resources for more strategic endeavors. Join us as we explore the capabilities of scheduled queries in BigQuery and unlock new possibilities for automated data processing.

Prerequisites.

GCP account

Open the console.

Open Menu > BigQuery

Type the Query in the query editor.

Click on Schedule > Create a new Scheduled Query.

Give the name for query.

Choose the repeats.

Choose the Dataset.

Give the table name. Click Schedule.

The Query will be scheduled.

Click on Big Query > Scheduled Queries

It will list the Scheduled Queries.

 

Automate analysis with BigQuery scheduled queries

0
0

Displaying error message in BigQuery

Viewing detailed error messages in BigQuery

Accessing detailed error messages in BigQuery is essential for diagnosing and resolving issues encountered during query execution. This introduction elucidates the importance and methods of viewing detailed error messages in BigQuery, empowering users to troubleshoot effectively and optimize query performance.

BigQuery provides comprehensive error messages that offer insights into the root causes of query failures, syntax errors, and data processing issues. These messages often include specific error codes, contextual information, and suggested resolutions, facilitating efficient troubleshooting and resolution of query-related issues.

Users can access detailed error messages in several ways within the BigQuery interface. When a query encounters an error, BigQuery displays an error message directly in the query editor, highlighting the problematic section of the query and providing relevant details to aid in diagnosis. Additionally, users can review error messages in the query history or job logs, accessing a detailed record of query executions and associated errors.

Prerequisites

GCP account

You may face various error codes and error messages when working with BigQuery, including HTTP error codes, job errors, and Google Cloud Console error messages. You can give your own error messages.

Open Cloud Console

Open Menu > Bigquery

Click 3 dots on project Click Create Dataset

Give Dataset ID.

Click Create Dataset.

3 dots in dataset. Click Open

Click on  the Create table

Select  Source as empty table.

Give Table name.

 create

Click on 3 dots in table. Click Open

Compose new query.

Paste the Query in query editor. Click run

CREATE OR REPLACE TABLE business.customer AS

SELECT

     123 AS cust_id,

‘Evan’ AS name;

SELECE * FROM business.customer;

Paste the below query in query editor.

DELETE FROM business.customer WHERE true;

Click run.

It will remove 1 row.

Paste the below code in query editor.

SELECT

COUNT(*) AS row_count

FROM business.customer

HAVING

     IF(row_count > 0,true,

ERROR(FORMAT(‘Error : row_count must be positive but it is %t ‘,row_count)

)

);

Click run.

It will display error message.

 

This is all about Viewing detailed error messages in BigQuery

0
0

Data Fusion with BigQuery

Cloud Data Fusion and BigQuery for data integration

 

Prerequisites

GCP account

Open Console.

Click on Activate cloud shell

$          echo $GOOGLE_CLOUD_PROJECT                   #Display the Project ID

$          gsutil cp gs://cloud-training/OCBL017/ny-taxi-2018-sample.csv gs://$GOOGLE_CLOUD_PROJECT                                

#Copying file from cloud training public bucket into our bucket

$          gsutil mb gs://$GOOGLE_CLOUD_PROJECT-temp   #Creating bucket

Open  Menu > Cloud Storage > Browser

The temporary bucket is created and the file is copied in main bucket.

Open Menu > Data Fusion

Click on View Instance. It will open the Data Fusion page.

Click No thanks.

Click on Wrangler

In Default Cloud Storage > Open the bucket which we copied the file.

Select the copied file.

Click on drop down arrow.

Click Parse > CSV

Select comma and tick set first row as header.

Click Apply

In drop down of body column, Click delete column

In Trip distance drop down > change data type to Float

 open Filter > Custom condition

Give Condition as >0.0

Click Apply

In right side of table, we can see the transformations which we did in our table

Then Click on Create a pipeline

 

Click on Batch Pipeline.

It will create a Pipeline.

properties in Wrangler

The modifications done in table will be already mentioned in here

In right side, output schema is visible. In that delete column named extra.

Validate.

Click on close if no errors found.

Open  Console > BigQuery

Click Create dataset.

Give dataset name. Click Create Dataset.

Click on More > Query Settings

Select Set destination table for query results.

Give Project name,Dataset name, and table name.

Select Write if empty or Overwrite table and press save.

Click save

In Query Editor, Paste the below code

SELECT

  zone_id,

  zone_name,

  borough

FROM

  bigquery-public-data.new_york_taxi_trips.taxi_zone_geom

Click Run

The output will be saved in the created table.

Go back to Data Fusion

 Source > BigQuery

Properties in BigQuery.

Reference name.

Give the dataset name and table name which we created.

Click Get schema

The Output Schema will be displayed in Right side.

Temporary bucket name.

Click on Validate.

if no errors found.

Click Analytics > Joiner

Drag the arrow into joiner from both BigQuery and Wrangler.

Click on properties in joiner

Jointype : inner

Join condition

Wrangler : pickup_location_id

BigQuery : zone_id

Click on Get Schema.

In output schema remove pickup_location_id & zone_id

remove zone_id.

Validate.

close if no errors found.

Sink > bigQuery

properties in second big query.

Give Reference name, dataset, table name.

Give the temporary bucket name which we created.

Click on Validate.

close if no errors found

Now drag the arrow from Joiner

rename the pipeline. Click save

Save.

Deploy

It will deploy pipeline

Click run.

It will Start Execution. This execution will take arround 10-15 minutes.

Execution is Provisioning

 Starting

Running

 Finished Successfully.

Go to BigQuery. In the dataset, new table will be created.

In the data Fusion, Click Logs.You can see the logs of Execution.

 

Cloud Data Fusion and BigQuery for data integration

 

0
0

Creating Data Fusion Instance

Create Cloud Data Fusion instance

 

Creating a Cloud Data Fusion instance in Google Cloud Platform (GCP) enables users to build, deploy, and manage data integration pipelines for extracting, transforming, and loading (ETL) data from various sources into their data warehouses or analytics platforms. Cloud Data Fusion provides a visual interface and a managed service for building ETL pipelines without requiring users to manage infrastructure or write code.

To create a Cloud Data Fusion instance, users can navigate to the Cloud Data Fusion section of the GCP Console and click on the “Create instance” button. From there, users can specify configuration details such as the instance name, location, and version of Cloud Data Fusion. Users can also choose the machine type, network settings, and other advanced options as needed.

 

Prerequisites

GCP account

Open Console.

Open Menu > API services > Library

Search Data Fusion API

Select Cloud Data Fusion API

Click Enable API

Open Menu > Data Fusion

Click on create an instance

Give name

Choose Edition as Basic

In Authorization, click on Grant Permission

Click Create.

The instance will be created. It will create 10-15 minutes.

Click on Instance.

Copy the service account.

Go to IAM & Admin to add permission.

Click Add

Paste the service account.

Give role as Cloud Data Fusion API service Agent.

NB : Sometimes you may have to give Cloud Data Fusion Admin access also.

Click Save.

The Service account will be saved with the given role.

0
0

Adding Drive File into BigQuery

BigQuery external tables documentation

 

Incorporating files from Google Drive into BigQuery offers a streamlined pathway to leverage diverse data sources within Google’s comprehensive cloud ecosystem. This introduction elucidates the process of seamlessly integrating Drive files into BigQuery, highlighting the methods, considerations, and advantages inherent in this operation.

Google Drive serves as a robust cloud storage platform, facilitating collaboration and centralizing document management. BigQuery, on the other hand, is a powerful data analytics platform renowned for its scalability and performance. By bridging these platforms, users can harness the rich data stored in Drive for advanced analytics and insights generation.

The process of adding Drive files into BigQuery typically involves exporting the desired files from Drive to Google Cloud Storage in a compatible format, such as CSV or JSON. Subsequently, users can leverage BigQuery’s native functionality to ingest the data from Cloud Storage into BigQuery tables, enabling seamless integration with existing datasets and analytical workflows.

Prerequisites

GCP account

https://drive.google.com/drive/

Click the above link. It will open google drive of yours.

Click New.

Google Sheets > Blank Spreadsheet.

Type the spreadsheet like below.

In cell B3 type

=GOOGLETRANSLATE(‘source cell’, ”written language”, ‘translation language’)

Give the cell ID’s for source message and translation language.

Click outside of that cell

Open the console.

Go to Menu > Big Query > SQL Instances

Select the project.

Click on Create Dataset.

Give the Dataset ID.

Click on Create Dataset.

It will listed in under the project.

Click on Create Table

Select the Source as Drive.

Open the Drive spreadsheet. Copy the Link.

Paste it in the URI. Choose File format as Google Drive.

Give the table name.

Select the edit as text.

Type the below code.

Message:STRING

Code:STRING

Translation:STRING

Click Create Table.

Type the Query in Query Editor. And press run.

The query will be executed and it will show the results in Query results.

 

BigQuery external tables documentation

0
0

Adding BigQuery data through cloud shell

Prerequisites

GCP account

Open Cloud shell.

Paste the Below Code in Console.

bq load \

–source_format=CSV \

–autodetect \

–noreplace \

nyc_taxi.2018_trips \

gs://cloud-training/OCBL013/nyc_tlc_yellow_trips_2018_subset_1.csv

It will add the file into the Bigquery.

Open the Table and check the details.

Paste the below code.

CREATE TABLE

     nyc_taxi.january_trips AS

SELECT

     *

FROM

     nyc_taxi.2018_trips

WHERE

     EXTRACT(Month

     FROM

          pickup_datetime)=1;

Click Run.

 It will create a new table named January_trips.

It will create a new table named January_trips.

0
0

Adding Local File into Big Query

Uploading local files to BigQuery alternatives

Integrating local files into BigQuery represents a vital aspect of expanding data accessibility and leveraging the platform’s analytical capabilities. This introduction will elucidate the process of seamlessly incorporating local files into BigQuery, elucidating the methods, considerations, and advantages inherent in this operation.

While BigQuery excels in processing vast datasets stored in Google Cloud Storage or other cloud-based repositories, there are scenarios where local files, residing within an organization’s infrastructure or on individual workstations, contain valuable data that warrants inclusion in analytical workflows.

The process of adding local files into BigQuery involves a few essential steps, such as preparing the data in a compatible format, uploading it to Google Cloud Storage, and then utilizing BigQuery’s native capabilities to ingest the data into its tables. Supported file formats include CSV, JSON, Avro, Parquet, and more, ensuring flexibility and compatibility with various data sources.

Prerequisites

GCP account

Open Console.

Open Menu > Bigquery > SQL Workspace

In Resources, select your project.

Click on Create  Dataset

Give the name for Dataset. Rest of it no need to change.

Click Create Dataset.

The dataset will be created.

Download one file by pressing the below link.

https://storage.googleapis.com/cloud-training/OCBL013/nyc_tlc_yellow_trips_2018_subset_1.csv

In the created dataset, click create Table.

Choose the source as upload.

Click Browse and choose the downloaded file.

It will automatically detect the file format. If not, select it as CSV

Give the Project name and Dataset name. Usually it will be auto filled.

Give the table name.

Tick the Auto Detect Schema. Click Create table.

It will create the table with the downloaded file content.

Paste the below query in query editor.

SELECT

     *

FROM

     nyc_taxi.2018_trips

ORDER BY

     fare_amount DESC

LIMIT

     10

And Click run. It will give the output in query results.

 

Uploading local files to BigQuery alternatives

0
0

Adding GCS file into BigQuery

BigQuery data loading from Google Cloud Storage

 

Loading data from Google Cloud Storage (GCS) into BigQuery is a fundamental process that forms the backbone of many data ingestion pipelines within Google’s comprehensive cloud ecosystem. This introduction will explore the intricacies of data loading from GCS to BigQuery, providing insights into the mechanisms, best practices, and benefits associated with this essential operation.

Google Cloud Storage serves as a scalable and durable object storage solution, ideal for storing a wide array of data types, including structured, semi-structured, and unstructured data. BigQuery, on the other hand, offers a powerful, fully managed data warehouse platform with lightning-fast SQL queries and built-in machine learning capabilities.

The process of loading data from GCS into BigQuery involves several steps, including defining the data schema, configuring the load job settings, and initiating the data transfer. Users can choose from a variety of file formats supported by both GCS and BigQuery, such as CSV, JSON, Avro, Parquet, and more, ensuring compatibility and flexibility in data loading operations.

Prerequisites

GCP account

Download file by pressing the below link.

https://storage.googleapis.com/cloud-training/OCBL013/nyc_tlc_yellow_trips_2018_subset_1.csv

This is a google cloud training data.

Open Console.

Add the downloaded file into Bucket.

Open Menu > BigQuery.

Open the Project.

Click on Create Dataset.

Give dataset ID. Then click Create Dataset.

Click on Create Table.

Select Source as Google Cloud Storage. Click Browse.

Choose the file from bucket. Click Select

The File will be selected with file format.

Choose the Destination table.

Give project name, dataset name and table name.

Click on auto detect. Click create table.

The table will be added.

Paste your query in query editor.

SELECT

*

FROM

     gcs_bq.nyc_trips

ORDER BY

     Fare_amount DESC

LIMIT

10;

Click Run.The Query will be executed.

Paste your query in query editor.

SELECT

*

FROM

     gcs_bq.nyc_trips

WHERE

     EXTRACT(Month

          FROM

          Pickup_datetime)=1

ORDER BY

     fare_amount DESC

LIMIT

10;

Click Run.The Query will be executed.

 

BigQuery data loading from Google Cloud Storage

0
0

Quick Support

image image