Terraformer — Write-back your Infrastructure to code

Author : Nishtha Malhotra, Associate Engineer – CloudDevOps

Original Published Date: October 11, 2023

Automation, DevOps, Terraform, Terraformer

Problem Statement

IAC or Infrastructure as Code allows you to build, change, and manage your infrastructure through coding instead of manual processes. The configuration files are created according to your infrastructure specifications and these configurations can be edited and distributed securely within an organization.

We all know Terraform is a popular infrastructure as code tool that lets you build, change, and version cloud and on-prem resources safely and efficiently. Now, many organisations have large footprints of Infrastructure on Cloud which was created manually. As a part of organization-wide modernization strategy, organization might want to bring those Infrastructure pieces also to be maintained using Infrastructure as a Code tool. Now if the number of resources are less, in that case you can simply write the code and import the infrastructure state into statefile. But if the number of resources are really high then it would become really challenging to write the code of that existing infrastructure with all the required dependencies.

Solution

Terraformer can address this challenge by automatically generating terraform files from the existing infrastructure with all the dependencies. In this blog I am going to explain how to setup and use Terraformer utility to automatically generate Terraform files for AWS Cloud.

Before we move into setup, let us look at some more details around Terraformer utility and its capabilities.

What is Terraformer?

Terraformer is an open-source CLI (Command Line Interface) tool from Google Cloud Platform to generate files from existing infrastructure. In short performing the reverse of what Terraform is designed to do that involves creating Terraform files from pre-existing infrastructure.

Capabilities

17 Cloud and more than 25+ Providers support like Kubernetes, Pagerduty, Okta
Read-only permissions to just fetch the code.
Generate tf/Json + tfstate files from existing infrastructure for all supported objects by resource.
Save tf/Json files using a custom folder tree pattern which means it saves your code in the directory generated → name of the provider e.g aws → name of the resource for which you are fetching the code → tf files
Import by resource name and type.

Setup and Usage of the Terraformer Utility

Pre-requisites

Support terraform 0.13 (for terraform 0.11 use v0.7.9).
Go (If you are installing from source)

Setup Instructions

For installation follow the instructions mentioned GoogleCloudPlatform/terraformer: CLI tool to generate terraform files from existing infrastructure (reverse Terraform). Infrastructure to Code (github.com

Testing

Now lets test few commands to import resources using terraformer.

Import a VPC resource

terraformer import aws --resources=vpc --regions=us-east-1

This will create a directory with name generated → aws → vpc in the same directory you run the command.

Under vpc all the files with code will be created:

Similarly you can import all the resources for other cloud and services by running the command like below.

terraformer import <provider name> --resources=<name of the resource> --regions=<resources region>

Terraform Import v/s Terraformer Import

Terraform also provides terraform import command out of the box which you can use to import the state of your infrastructure. But the terraform import only imports the state of the infrastructure into statefile i.e. terraform.tfstate, it doesn’t generate the code files automatically and you still have to write your terraform code manually. However with terraformer import, it generates both code files as well the stat of the infrastructure in statefile.

Challenges and Solutions

Multiple statefiles

Though the terraformer is a great utility to generate terraform code files from existing infrastructure. But the challenge with this utility is it creates a subfolder for each resource and inside subfolder it maintains the terraform code files and the terraform.tfstate for that particular resource. We might need to merge all the states into one single statefile in order to easly operate and maintain the infrastructure code.

For this we can use a small shell script written here

ct-terraformer-automation/tfstate-merge.sh at main · cloudtechner/ct-terraformer-automation (github.com) Connect your Github account to merge multiple terraform.tfstate files into one single terraform.tfstate file.

Script Usage

sh tfstate-merge.sh “name of the resource in which you want to merge“ “name of the resource which you want to merge”

Lets see how the script works to merge two resources ec2 and vpc . Run the script from parent directory with ec2 as source and vpc as target

sh tfstate-merge.sh ec2_instance vpc

It will initialize the terraform in the parent directory directory, pull the terraform.tfstate file from the source directory i.e ec2_instance.

2. Similarly, it will go to the destination directory , initialize the terraform and pull the terraform.tfstate file from the destination directory i.e vpc .

3.After that it will take a backup of the terraform.tfstate file from the source and merge it to the destination and store the new terraform.tfstate in the destination directory .

4. You will see the merged terraform.tfstate file and backup files in the destination directory.

Variablization

Another challenge with terraformer utility is it creates all the terraform code files with hard-coded variable values. Now this could be challenging where have multiple resources/multiple environments, we might need to variablize the hard-coded values and managed those values tfvars file.

For this we have written another small python based utility ct-terraformer-automation/terraformer-variables-automation.py at main · cloudtechner/ct-terraformer-automation (github.com) which takes care of creating tfvars files too automatically for AWS resources. As of now script supports for ec2, vpc, rds, ebs and cloudwatch resources for AWS. You can extend the same script as per your requirement.

Conclusion

Terraformer is a powerful infrastructure as code (IaC) tool that allows you to generate the configuration from any existing infrastructure . It define, provision, and manage your cloud infrastructure resources in a declarative way. However, like any other tool, it has its learning curve and considerations, and it’s important to familiarize yourself with its concepts, syntax, and best practices to make the most out of it.