Creating an instance takes a handful of choices, but two of them are worth slowing down for. The edition decides the hourly price and cannot be changed once the instance exists. And the instance bills for every hour it exists, so a forgotten practice instance is an expensive mistake.<\/p>\r\n
This walks through enabling the API, creating the instance, and giving its service account the role pipelines need, with the edition and cost explained at the steps where you meet them.<\/p>\r\n
Prerequisites<\/strong><\/p>\r\n Open the console, then the menu, then the APIs and Services section, then Library.<\/p>\r\n\r\n\r\n\r\n Search for Data Fusion, then select Cloud Data Fusion API.<\/p>\r\n\r\n\r\n\r\n Click Enable.<\/p>\r\n\r\n\r\n\r\n Open the menu, then Data Fusion.<\/p>\r\n\r\n\r\n\r\n Click Create an instance.<\/p>\r\n\r\n\r\n\r\n Enter an instance name. Google’s form also asks for a description and the region to create the instance in, so choose a region close to your data.<\/p>\r\n\r\n\r\n\r\n The original page says to choose Basic. That is a reasonable choice for learning, and the table below shows why, but make the choice on purpose. Google’s documentation states that you cannot change an instance’s edition after creating it.<\/p>\r\n\r\n\r\n\r\n These prices are per instance hour, from the time the instance is created to the time it is deleted, billed by the minute. The figures above are from Google’s pricing page, and the monthly amounts are Google’s own approximations. Basic is the edition with 120 free hours a month per account, which is why most learning labs use it, as long as you delete the instance before those hours run out.<\/p>\r\n The original page tells you to click Grant Permission under Authorization. Google’s current creation form asks you to choose the service account that runs your pipelines in Dataproc. The default is the Compute Engine service account, and Google says to make sure whichever account you use has the right IAM roles for your needs. If your form still shows a Grant Permission option, use it. The screenshot may show the older form.<\/p>\r\n\r\n\r\n\r\n Click Create. Google says it takes up to 30 minutes for the instance to finish being created, not the 10 to 15 minutes the original page gave. A progress indicator shows next to the instance while it is being built.<\/p>\r\n\r\n\r\n\r\n When it is ready, click the instance name to see its details.<\/p>\r\n\r\n\r\n\r\n On the details page, copy the service account listed for running pipelines in Dataproc. This is the account that pipelines run as.<\/p>\r\n\r\n\r\n\r\n Go to IAM and Admin, then IAM, and click Grant access, shown as Add in older versions.<\/p>\r\n\r\n\r\n\r\n Paste the service account into the New principals field and choose the role Cloud Data Fusion API Service Agent.<\/p>\r\n\r\n\r\n\r\n Pipelines run on Dataproc clusters that start up using this service account, so the account needs permission to work with Data Fusion. The original page adds that sometimes you may have to give Cloud Data Fusion Admin access as well. Admin is a much broader role than this step needs, so before granting it, read the exact permission error you are getting and check Google’s guidance on service account permissions. The GCP Service Account and IAM Roles guides in this series explain why narrow roles are safer.<\/p>\r\n Click Save.<\/p>\r\n\r\n\r\n\r\n The service account is saved with the role you gave it.<\/p>\r\n\r\n\r\n\r\n There are two parts to the bill. The first is the instance itself, charged per hour from creation to deletion at the edition’s rate, whether or not any pipeline is running. The second is processing, which is the cost of the Dataproc clusters that run your pipelines, charged on the vCPUs they use and how long they run.<\/p>\r\n As an example, a Basic instance left running after its free hours are used costs 1.80 US dollars every hour, which would use up a 300 dollar free trial credit in roughly a week. Delete practice instances as soon as you finish. The Creating and Deleting a Budget guide in this series explains why a budget alert on its own will not stop that spending.<\/p>\r\n Once the instance is ready, you open its web interface from the instances list and start designing pipelines. The Data Fusion With BigQuery guide in this series builds one.<\/p>\r\n\r\n
Step by Step Process of Creating Cloud Data Fusion Instance<\/h2>\r\n
Step 1: Open the API Library<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-141.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 2: Find the Data Fusion API<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-142.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 3: Enable the API<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-143.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 4: Open Data Fusion<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-144.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 5: Create an Instance<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-145.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 6: Name It<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-146.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 7: Choose the Edition<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-147.png\")
<\/figure>\r\n\r\n\r\n\r\nChoosing an Edition<\/h3>\r\n
\r\n \r\nEdition<\/th>\r\n Price per instance hour<\/th>\r\n About per month<\/th>\r\n Meant for<\/th>\r\n<\/tr>\r\n<\/thead>\r\n \r\n Developer<\/th>\r\n 0.35 US dollars<\/td>\r\n 250 US dollars<\/td>\r\n Development and exploration. Two users and two simultaneous pipelines are recommended.<\/td>\r\n<\/tr>\r\n \r\n Basic<\/th>\r\n 1.80 US dollars, after 120 free hours a month per account<\/td>\r\n 1,100 US dollars<\/td>\r\n Testing, sandboxes, and proof of concept work. Unlimited simultaneous pipelines.<\/td>\r\n<\/tr>\r\n \r\n Enterprise<\/th>\r\n 4.20 US dollars<\/td>\r\n 3,000 US dollars<\/td>\r\n Production workloads. Adds role based access control and regional high availability.<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/div>\r\n Step 8: Authorization and the Service Account<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-148.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 9: Create the Instance<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-149.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 10: Open the Instance<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-150.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 11: Copy the Service Account<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-151.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 12: Open IAM and Add the Account<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-152.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 13: Paste the Account and Give It the Role<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-153.png\")
<\/figure>\r\n\r\n\r\n\r\nWhat This Role Step is For?<\/h3>\r\n
Step 14: Save<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-154.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 15: Confirm the Account Has the Role<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-155.png\")
<\/figure>\r\nWhat Data Fusion Costs?<\/h2>\r\n
What to do Next?<\/h2>\r\n
Common Mistakes to Avoid<\/h2>\r\n