{"id":131,"date":"2023-10-11T10:13:00","date_gmt":"2023-10-11T10:13:00","guid":{"rendered":"https:\/\/cloudtechner.com\/blog\/?p=131"},"modified":"2024-06-13T11:20:54","modified_gmt":"2024-06-13T11:20:54","slug":"terraformer-write-back-your-infrastructure-to-code","status":"publish","type":"post","link":"https:\/\/cloudtechner.com\/blog\/terraformer-write-back-your-infrastructure-to-code\/","title":{"rendered":"Terraformer \u2014 Write-back your Infrastructure to code"},"content":{"rendered":"\n<p>Author : <a href=\"https:\/\/www.linkedin.com\/in\/nishtha-malhotra-557bb8167\">Nishtha Malhotra<\/a>, Associate Engineer &#8211; CloudDevOps<\/p>\n\n\n\n<p>Original Published Date: October 11, 2023<\/p>\n\n\n<div class=\"taxonomy-post_tag wp-block-post-terms\"><a href=\"https:\/\/cloudtechner.com\/blog\/tag\/automation\/\" rel=\"tag\">Automation<\/a><span class=\"wp-block-post-terms__separator\">, <\/span><a href=\"https:\/\/cloudtechner.com\/blog\/tag\/devops\/\" rel=\"tag\">DevOps<\/a><span class=\"wp-block-post-terms__separator\">, <\/span><a href=\"https:\/\/cloudtechner.com\/blog\/tag\/terraform\/\" rel=\"tag\">Terraform<\/a><span class=\"wp-block-post-terms__separator\">, <\/span><a href=\"https:\/\/cloudtechner.com\/blog\/tag\/terraformer\/\" rel=\"tag\">Terraformer<\/a><\/div>\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"81bc\">Problem Statement<\/h3>\n\n\n\n<p id=\"ac6f\"><a href=\"https:\/\/www.simplilearn.com\/what-is-infrastructure-as-code-article\" rel=\"noreferrer noopener\" target=\"_blank\">IAC or Infrastructure as Code<\/a>&nbsp;allows you to&nbsp;<strong>build, change, and manage<\/strong>&nbsp;your infrastructure through coding instead of manual processes. The configuration files are created according to your infrastructure specifications and these configurations can be edited and distributed securely within an organization.<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter is-resized\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/1*MctX_ekrSoGXHMygb-eUzg.gif\" alt=\"\" style=\"width:654px;height:auto\"\/><figcaption class=\"wp-element-caption\">Infrastructure as a Code<\/figcaption><\/figure><\/div>\n\n\n<p id=\"ff33\">We all know Terraform is a popular infrastructure as code tool that&nbsp;<strong>lets you build, change, and version cloud and on-prem resources safely and efficiently<\/strong>. Now, many organisations have large footprints of Infrastructure on Cloud which was created manually. As a part of organization-wide modernization strategy, organization might want to bring those Infrastructure pieces also to be maintained using Infrastructure as a Code tool. Now if the number of resources are less, in that case you can simply write the code and import the infrastructure state into statefile. But if the number of resources are really high then it would become really challenging to write the code of that existing infrastructure with all the required dependencies.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"0e97\">Solution<\/h3>\n\n\n\n<p id=\"be3a\"><a href=\"https:\/\/github.com\/GoogleCloudPlatform\/terraformer\" rel=\"noreferrer noopener\" target=\"_blank\">Terraformer<\/a>&nbsp;can address this challenge by automatically generating terraform files from the existing infrastructure with all the dependencies. In this blog I am going to explain how to setup and use Terraformer utility to automatically generate Terraform files for AWS Cloud.<\/p>\n\n\n\n<p id=\"6c18\">Before we move into setup, let us look at some more details around Terraformer utility and its capabilities.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"4911\">What is Terraformer?<\/h4>\n\n\n\n<p id=\"b591\">Terraformer is an open-source CLI (Command Line Interface) tool from Google Cloud Platform to generate files from existing infrastructure. In short performing the reverse of what Terraform is designed to do\u200b that involves creating Terraform files from pre-existing infrastructure.<\/p>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"c037\">Capabilities<\/h5>\n\n\n\n<ul>\n<li>17 Cloud and more than 25+ Providers support like Kubernetes, Pagerduty, Okta<\/li>\n\n\n\n<li>Read-only permissions to just fetch the code.\u200b<\/li>\n\n\n\n<li>Generate tf\/Json + tfstate files from existing infrastructure for all supported objects by resource.\u200b<\/li>\n\n\n\n<li>Save tf\/Json files using a custom folder tree pattern which means it saves your code in the directory generated \u2192 name of the provider e.g aws \u2192 name of the resource for which you are fetching the code \u2192 tf files \u200b<\/li>\n\n\n\n<li>Import by resource name and type.\u200b<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"e7a0\">Setup and Usage of the Terraformer Utility<\/h3>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"2b5c\">Pre-requisites<\/h5>\n\n\n\n<ul>\n<li>Support terraform 0.13 (for terraform 0.11 use v0.7.9).\u200b<\/li>\n\n\n\n<li>Go (If you are installing from source)<\/li>\n<\/ul>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"4dcf\">Setup Instructions<\/h5>\n\n\n\n<p id=\"6a0d\">For installation follow the instructions mentioned&nbsp;<a href=\"https:\/\/github.com\/GoogleCloudPlatform\/terraformer\/tree\/master#installation\" rel=\"noreferrer noopener\" target=\"_blank\">GoogleCloudPlatform\/terraformer: CLI tool to generate terraform files from existing infrastructure (reverse Terraform). Infrastructure to Code (github.com<\/a><\/p>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"20bc\">Testing<\/h5>\n\n\n\n<p id=\"2461\">Now lets test few commands to import resources using terraformer.<\/p>\n\n\n\n<ul>\n<li>Import a VPC resource<\/li>\n<\/ul>\n\n\n\n<p id=\"4b0a\"><code>terraformer import aws --resources=vpc --regions=us-east-1<\/code><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/0*6F4KLcVh1I2T4E3N\" alt=\"\"\/><\/figure>\n\n\n\n<ul>\n<li>This will create a directory with name&nbsp;<strong>generated \u2192 aws \u2192 vpc&nbsp;<\/strong>in the same directory you run the command.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1216\/0*HuR3FI88ipON5kM-\" alt=\"\"\/><\/figure>\n\n\n\n<ul>\n<li>Under vpc all the files with code will be created:<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/0*v7P3kJ1VGxNBn-5G\" alt=\"\"\/><\/figure>\n\n\n\n<p id=\"5693\">Similarly you can import all the resources for other cloud and services by running the command like below.<\/p>\n\n\n\n<p id=\"bc8f\"><code>terraformer import &lt;provider name&gt; --resources=&lt;name of the resource&gt; --regions=&lt;resources region&gt;<\/code><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"be37\">Terraform Import v\/s Terraformer Import<\/h3>\n\n\n\n<p id=\"f8b2\">Terraform also provides&nbsp;<code>terraform import<\/code>&nbsp;command out of the box which you can use to import the state of your infrastructure. But the&nbsp;<code>terraform import<\/code>&nbsp;only imports the state of the infrastructure into statefile i.e. terraform.tfstate, it doesn\u2019t generate the code files automatically and you still have to write your terraform code manually. However with&nbsp;<code>terraformer import<\/code>, it generates both code files as well the stat of the infrastructure in statefile.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"7cbf\">Challenges and Solutions<\/h3>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"f7a9\">Multiple statefiles<\/h5>\n\n\n\n<p id=\"67da\">Though the terraformer is a great utility to generate terraform code files from existing infrastructure. But the challenge with this utility is it creates a subfolder for each resource and inside subfolder it maintains the terraform code files and the terraform.tfstate for that particular resource. We might need to merge all the states into one single statefile in order to easly operate and maintain the infrastructure code.<\/p>\n\n\n\n<p id=\"4680\">For this we can use a small shell script written here<\/p>\n\n\n\n<p id=\"249e\"><a href=\"https:\/\/github.com\/cloudtechner\/ct-terraformer-automation\/blob\/main\/tfstate-merge.sh\" rel=\"noreferrer noopener\" target=\"_blank\">ct-terraformer-automation\/tfstate-merge.sh at main \u00b7 cloudtechner\/ct-terraformer-automation (github.com)<\/a>&nbsp;Connect your Github account to merge multiple terraform.tfstate files into one single terraform.tfstate file.<\/p>\n\n\n\n<p id=\"bfe9\"><strong>Script Usage<\/strong><\/p>\n\n\n\n<p id=\"4ce6\"><code>sh tfstate-merge.sh \u201cname of the resource in which you want to merge\u201c \u201cname of the resource which you want to merge\u201d<\/code><\/p>\n\n\n\n<p id=\"f430\">Lets see how the script works to merge two resources ec2 and vpc . Run the script from parent directory with ec2 as source and vpc as target<\/p>\n\n\n\n<p id=\"a2b0\"><code>sh tfstate-merge.sh ec2_instance vpc<\/code><\/p>\n\n\n\n<ol>\n<li>It will initialize the terraform in the parent directory directory, pull the terraform.tfstate file from the source directory i.e ec2_instance.<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/0*9kyW9JStmmDAYhWI\" alt=\"\"\/><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/0*UhvTUYonx_vxOfIx\" alt=\"\"\/><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1252\/0*kTjgVIYecwn_Pcc7\" alt=\"\"\/><\/figure>\n\n\n\n<p id=\"192c\">2. Similarly, it will go to the destination directory , initialize the terraform and pull the terraform.tfstate file from the destination directory i.e vpc .<\/p>\n\n\n\n<p id=\"7748\">3.After that it will take a backup of the terraform.tfstate file from the source and merge it to the destination and store the new terraform.tfstate in the destination directory .<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/0*9GNmIbrnf9aO1siX\" alt=\"\"\/><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/0*SjJE-4slMotLW2eh\" alt=\"\"\/><\/figure>\n\n\n\n<p id=\"b144\">4. You will see the merged terraform.tfstate file and backup files in the destination directory.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1400\/0*zd5sxdBe9dpJAKdK\" alt=\"\"\/><\/figure>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"b75b\">Variablization<\/h5>\n\n\n\n<p id=\"2736\">Another challenge with terraformer utility is it creates all the terraform code files with hard-coded variable values. Now this could be challenging where have multiple resources\/multiple environments, we might need to variablize the hard-coded values and managed those values tfvars file.<\/p>\n\n\n\n<p id=\"51c0\">For this we have written another small python based utility&nbsp;<a href=\"https:\/\/github.com\/cloudtechner\/ct-terraformer-automation\/blob\/main\/terraformer-variables-automation.py\" rel=\"noreferrer noopener\" target=\"_blank\">ct-terraformer-automation\/terraformer-variables-automation.py at main \u00b7 cloudtechner\/ct-terraformer-automation (github.com)<\/a>&nbsp;which takes care of creating tfvars files too automatically for AWS resources. As of now script supports for ec2, vpc, rds, ebs and cloudwatch resources for AWS. You can extend the same script as per your requirement.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"0ad9\">Conclusion<\/h3>\n\n\n\n<p id=\"2378\"><strong>Terraformer<\/strong>&nbsp;is a powerful infrastructure as code (IaC) tool that allows you to generate the configuration from any existing infrastructure . It define, provision, and manage your cloud infrastructure resources in a declarative way. However, like any other tool, it has its learning curve and considerations, and it\u2019s important to familiarize yourself with its concepts, syntax, and best practices to make the most out of it.<\/p>\n\n\n\t<div id=\"respond\" class=\"comment-respond wp-block-post-comments-form\">\n\t\t<h3 id=\"reply-title\" class=\"comment-reply-title\">Leave a Reply <small><a rel=\"nofollow\" id=\"cancel-comment-reply-link\" href=\"\/blog\/wp-json\/wp\/v2\/posts\/131#respond\" style=\"display:none;\">Cancel reply<\/a><\/small><\/h3><form action=\"https:\/\/cloudtechner.com\/blog\/wp-comments-post.php\" method=\"post\" id=\"commentform\" class=\"comment-form\"><p class=\"comment-notes\"><span id=\"email-notes\">Your email address will not be published.<\/span> <span class=\"required-field-message\">Required fields are marked <span class=\"required\">*<\/span><\/span><\/p><p class=\"comment-form-comment\"><label for=\"comment\">Comment <span class=\"required\">*<\/span><\/label> <textarea id=\"comment\" name=\"comment\" cols=\"45\" rows=\"8\" maxlength=\"65525\" required=\"required\"><\/textarea><\/p><p class=\"comment-form-author\"><label for=\"author\">Name <span class=\"required\">*<\/span><\/label> <input id=\"author\" name=\"author\" type=\"text\" value=\"\" size=\"30\" maxlength=\"245\" autocomplete=\"name\" required=\"required\" \/><\/p>\n<p class=\"comment-form-email\"><label for=\"email\">Email <span class=\"required\">*<\/span><\/label> <input id=\"email\" name=\"email\" type=\"text\" value=\"\" size=\"30\" maxlength=\"100\" aria-describedby=\"email-notes\" autocomplete=\"email\" required=\"required\" \/><\/p>\n<p class=\"comment-form-url\"><label for=\"url\">Website<\/label> <input id=\"url\" name=\"url\" type=\"text\" value=\"\" size=\"30\" maxlength=\"200\" autocomplete=\"url\" \/><\/p>\n<p class=\"comment-form-cookies-consent\"><input id=\"wp-comment-cookies-consent\" name=\"wp-comment-cookies-consent\" type=\"checkbox\" value=\"yes\" \/> <label for=\"wp-comment-cookies-consent\">Save my name, email, and website in this browser for the next time I comment.<\/label><\/p>\n<p class=\"form-submit\"><span class=\"bloghash-submit-form-button\"><input name=\"submit\" type=\"submit\" id=\"submit\" class=\"submit\" value=\"Post Comment\" \/><\/span> <input type='hidden' name='comment_post_ID' value='131' id='comment_post_ID' \/>\n<input type='hidden' name='comment_parent' id='comment_parent' value='0' \/>\n<\/p><\/form>\t<\/div><!-- #respond -->\n\t\n\n\n<p><a href=\"https:\/\/medium.com\/@nishtha.malhotra?source=post_page-----9b86ca930457--------------------------------\"><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Author : Nishtha Malhotra, Associate Engineer &#8211; CloudDevOps Original Published Date: October 11, 2023 Problem Statement IAC or Infrastructure as Code&nbsp;allows you to&nbsp;build, change, and manage&nbsp;your infrastructure through coding instead&hellip;<\/p>\n","protected":false},"author":1,"featured_media":167,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[69,23,70,71],"tags":[66,39,68,89],"_links":{"self":[{"href":"https:\/\/cloudtechner.com\/blog\/wp-json\/wp\/v2\/posts\/131"}],"collection":[{"href":"https:\/\/cloudtechner.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cloudtechner.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cloudtechner.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cloudtechner.com\/blog\/wp-json\/wp\/v2\/comments?post=131"}],"version-history":[{"count":3,"href":"https:\/\/cloudtechner.com\/blog\/wp-json\/wp\/v2\/posts\/131\/revisions"}],"predecessor-version":[{"id":168,"href":"https:\/\/cloudtechner.com\/blog\/wp-json\/wp\/v2\/posts\/131\/revisions\/168"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cloudtechner.com\/blog\/wp-json\/wp\/v2\/media\/167"}],"wp:attachment":[{"href":"https:\/\/cloudtechner.com\/blog\/wp-json\/wp\/v2\/media?parent=131"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cloudtechner.com\/blog\/wp-json\/wp\/v2\/categories?post=131"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cloudtechner.com\/blog\/wp-json\/wp\/v2\/tags?post=131"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}