How I Cleaned up 148 Terraform Resource Deprecations in 10 Seconds

Recently, I found myself with the rather unenviable task of tidying up a handful of Terraform warnings in a relatively large codebase. Why so "unenviable", you ask? Indeed, I normally quite enjoy a spring clean. Until I noticed that a particular deprecated resource argument appeared 148 times.

Ah, the accursed inline_policy block. For the best part of about 30 seconds, I genuinely considered manually trawling through and replacing them with the recommended aws_iam_role_policy resource.

The idea would be to go from the image on the left to the image on the right:

But by the end of said 30 seconds, the idea of starting a new life abroad in some wildly different profession had suddenly become comparatively quite appealing. So, I looked to alternatives.

My mind wandered to clever uses of sed; some eldritch Bash or Python script that I wouldn’t be so fondly writing an article about, perhaps. I’d happened upon HCL parsers in the past, but I’d never wandered into the realm of programmatically writing HCL. Luckily for me, I stumbled upon hclwrite.

Using hclwrite

First, hclwrite is a nice little Go library that helps with, as you might expect, writing HCL configurations. It does also come equipped with the ability to parse HCL (there is also hclparse, which just handles the latter), so you can not only write from scratch, but you can also modify existing code. Much like an XML/HTML DOM, HCL is interpreted as nodes within an abstract syntax tree (AST), enabling you to use the API to work with blocks, such as a Terraform resource, without actually unmarshalling those constructs into Go structs.

Thus, the solution to my 148 deprecations crisis, I decided, was to write a Go script (it ended up being 124 lines long; you can find it on my GitHub, here), which I’ll now walk you through the thinking behind...

If you’re unfamiliar with Go (if so, I would quite recommend you pop over to go.dev/learn; this’ll get semi-advanced at times), we’ll create a folder for our script, call it something like migrate-inline-policies, run a go mod init, fetch the hclwrite package with go get hclwrite, and create ourselves a main.go file.

Our entrypoint, our main function, looks as so:

func main() {
  root := "my/terraform/root/dir" // Change this to your Terraform repository root directory

  // Walk through the repository recursively
  err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error {
    if err != nil {
      return err
    }

    // Process the file if it's a Terraform file
    if !info.IsDir() && strings.HasSuffix(path, ".tf") {
      fmt.Printf("Processing: %s\n", path)
      processFile(path)
    }

    return nil
  })

  if err != nil {
    log.Fatalf("Error walking the path: %v\n", err)
  }
}

Here’s where everything else takes place from. We’ll just walk through the repository recursively, from the root directory specified in root. Specifically, we skip everything that isn’t a Terraform file (suffixed with .tf), and process those that are. processFile is admittedly a rather hefty function, so you’re free to break it down a bit if you’re looking to make production use of this script. But for now, I’ll just explain it step by step.

func processFile(path string) {
  src, err := ioutil.ReadFile(path)
  if err != nil {
    log.Printf("Error reading %s: %v\n", path, err)
    return
  }

  // Parse Terraform
  file, diag := hclwrite.ParseConfig(src, path, hcl.Pos{Line: 1, Column: 1})
  if diag.HasErrors() {
    log.Printf("Parse errors in %s: %v\n", path, diag.Errs())
    return
  }
  body := file.Body()

  blocks := body.Blocks()
  var newPolicyBlocks []*hclwrite.Block

Here, we’re reading the file, having hclwrite parse our HCL (starting from the beginning of the file; you can choose where to start by using the Pos object from the hcl package). Provided our Diagnostics object contains no errors, we’ll keep the body in memory. hclwrite’s File object contains Blocks, which are a representation of - you guessed it — HCL blocks, such as Terraform resources (remember the mention of nodes at the beginning of this section?). This is where the fun comes in, so you’ll have to forgive the rather mammoth loop below:

// Search through all blocks within the file being processed
  for _, block := range blocks {

    // Search for `aws_iam_role` resources
    if block.Type() == "resource" && len(block.Labels()) == 2 && block.Labels()[0] == "aws_iam_role" {
      roleName := block.Labels()[1]
      roleBody := block.Body()

      var blocksToRemove []*hclwrite.Block

      // Look for `inline_policy`` blocks within the `aws_iam_role` resource
      for _, subBlock := range roleBody.Blocks() {
        if subBlock.Type() == "inline_policy" {
          policyNameAttr := subBlock.Body().GetAttribute("name")
          policyBodyAttr := subBlock.Body().GetAttribute("policy")

          // This should never be the case, since validation wouldn't pass, but useful to have anyway
          if policyNameAttr == nil || policyBodyAttr == nil {
            fmt.Printf("Skipping inline_policy in %s due to missing name or policy\n", path)
            continue
          }

          // Build `aws_iam_role_policy` block
          policyName := strings.Trim(strings.ReplaceAll(string(policyNameAttr.Expr().BuildTokens(nil).Bytes()[:]), "\"", ""), " ")
          resourceName := strings.ReplaceAll(roleName+"_"+policyName, "-", "_")

          newBlock := hclwrite.NewBlock("resource", []string{"aws_iam_role_policy", resourceName})
          newBody := newBlock.Body()

          newBody.SetAttributeRaw("name", policyNameAttr.Expr().BuildTokens(nil))
          traversal := hcl.Traversal{
            hcl.TraverseRoot{Name: "aws_iam_role"},
            hcl.TraverseAttr{Name: roleName},
            hcl.TraverseAttr{Name: "name"},
          }
          newBody.SetAttributeTraversal("role", traversal)
          newBody.SetAttributeRaw("policy", policyBodyAttr.Expr().BuildTokens(nil))

          newPolicyBlocks = append(newPolicyBlocks, newBlock)
          blocksToRemove = append(blocksToRemove, subBlock)
        }
      }

      // Remove the `inline_policy` blocks
      for _, b := range blocksToRemove {
        roleBody.RemoveBlock(b)
      }
    }
  }

Akin to how we walked through each of the files within the repository, we’re now walking through each of the blocks within the file, and filtering for aws_iam_role resources, which is what our inline policy is (problematically) declared within.

Now, pay close attention, because this is where you could, theoretically, adapt this script to your own needs. Not necessarily the specific case of the inline_policy argument being deprecated, but any kind of resource query, and any swapping around of their internals.

In this filter, we query the Type of the block (this could be a resource, a data source, a provider; whatever prepends the labels of the
block), and the Labels object, which contains, in this case, the name of the resource as defined by the provider (i.e., aws_iam_role), and the consumer-defined name given to the specific resource instance (i.e., my_iam_role).

We now have all of the information we need. So we can store the name of our block (since we’ll name our new aws_iam_role_policy resource based on it), and the body itself, for later.

We can then go a layer deeper, iterating through each of the blocks within those aws_iam_role resources to find any inline_policy arguments, bearing in mind there can be multiple within a single resource. We can then use the GetAttribute method to query the sub-block for the name and the policy itself, so we can extract those out later on.

---

On to building the aws_iam_role_policy itself. I’ve left comments throughout the code block earlier as “bookmarks”, in case my rambling has you lost 😅.

Now, since we’re creating a new resource, decoupled from the role resource itself, I realised we needed a decent way of naming said new resource. A portmanteau of the name of the role (which we stored in the roleName variable earlier) and the name of the inline policy (which we stored in the policyNameAttr variable) would make sense — though it may be a bit verbose, it captures the relationship between role and policy quite nicely. We’ll strip any hyphens and replace with underscores, in line with Terraform naming conventions.

Now, to actually create a new block, you use the intuitively named NewBlock method exposed by hclwrite, into which you can pass a string representation of the block Type (i.e., resource), and your labels, as a list of strings (so, an aws_iam_role_policy, and the resourceName we created above). At this stage we have a new, named resource, with no attributes. Bear in mind that this creation of blocks and such is all happening in memory; it hasn’t been committed back to the file yet.

In the body of our new block, we can set our attributes. There’s a few variations of this method, based on the three musketeers of assigning/manipulating stuff with hclwrite. You’ve got:

• SetAttributeRaw allows you to set the value for an attribute verbatim, i.e., based upon raw tokens. This doesn’t do any kind of validation or use a type system, so SetAttributeValue is often recommended instead, but this works for when you just want to rip the raw value from one place and stick it somewhere else within the same context (as we do).

• SetAttributeTraversal allows you to set the value for an attribute based upon a reference. It takes a Traversal object, which I’ll get on to in a second. But essentially, you can reference an entity within HCL, such as var.name – not just a raw, literal value.

• SetAttributeValue handles Value objects (literals), doing the token conversion for you with more safety than SetAttributeRaw.

In our case, we’re fine to use SetAttributeRaw by grabbing the raw tokens from variables like our policy name, that we stored earlier. We can do the same for the policy block. That might cause us a problem if we were moving to a different context (if you have references to variables/locals/etc. that you’re copying literally, it might not work if you move it to another module), but we’re not – this all happens within the same file.

Now, Traversals. They’re constructed of information that allows you to query HCL constructs for a particular entity. In this case, we want to get the name attribute of the original aws_iam_role resource (since it’s an attribute exported from a resource, we can’t just reference it literally), so we build a Traversal object to find it. So, we find the root by its name, aws_iam_role, using a TraverseRoot object, and a TraverseAttr object to query for a label that matched the roleName we saved earlier, and retrieve the name attribute of it. Then we can use SetAttributeTraversal to set the role argument of the new, decoupled aws_iam_role_policy resource to reference the name of the aws_iam_role resource that it’s to be associated with.

Then all we need to do is append that to our list of blocks to add to the file, and the inline_policy sub-block from earlier appended to our list of blocks to remove from the file.

// Append the new `aws_iam_role_policy` resources at the end
  for _, b := range newPolicyBlocks {
    body.AppendNewline()
    body.AppendBlock(b)
  }

  // Write back to the same file
  err = ioutil.WriteFile(path, file.Bytes(), 0644)
  if err != nil {
    log.Printf("Error writing %s: %v\n", path, err)
    return
  }

  fmt.Printf("Updated: %s\n", path)
}

Finally, we can write all of those changes, based upon those lists, to the file. Then carry on iterating through the other files, until the file traversal completes.

Now you can just run your script (go run main.go) and watch with a smug grin as it blitzes through a job that otherwise would have taken you 8 hours and 18 head bangs against the desk 🥳🎉

Caveats

There are a couple gripes that I have with hclwrite (though my opinion overall is that it’s a pretty useful tool).

• There doesn’t seem to be a way, as it stands, to specify where new blocks are added into the file, because hclwrite doesn’t care about lines, it cares about HCL constructs. It just sticks new blocks at the end of the file. That can be a bit annoying if you want a logical structure (e.g., your policy resources follow immediately after your role resource). I had a play with getting it to work, but it was a bit of a nightmare that I eventually gave up on.
• In a similar vein, there doesn’t seem to be way to handle adding newlines for padding, or scrapping trailing lines left behind after removals. For example, I had to use a bit of a gross find + replace to get rid of the empty lines at the end of my aws_iam_role resources, above where the inline_policy blocks once were. This isn’t something that your terraform fmt does for you (and I don’t believe tflint offers something for it either), so it leaves you a bit stuck.

Then again, both of the above complaints are largely around formatting/linting-related things, which could reasonably be argued is kind of out of hclwrite’s intended scope. However, its functionality does enter the territory where it probably should be considering code style a little bit.

---

And that about wraps it up. So if you ever find yourself having a bunch of repetitive refactoring to do around your Terraform (or OpenTofu) codebase, think hclwrite. And please don’t think about using sed or something. That way lies suffering.