Creating custom ansible filters

Often when writing ansible playbooks you might find yourself needing to perform some text transformation. When doing simple things like direct data type conversion or running an algorithm against the string this can usually be easily performed by the built in Jinja2 filters in ansible.

# For human readable output, you can use:
as_json: "{{ some_variable | to_nice_json }}"
as_yaml: "{{ some_variable | to_nice_yaml }}"

# Get the min and max of lists
min_num: "{{ [3, 4, 2] | min }}"
max_num: "{{ [3, 4, 2] | max }}"

# Perform complex operations on strings
sha1_hash: "{{ 'test1' | hash('sha1') }}"
checksum: "{{ 'test1' | checksum }}"

However, when we being to have complex text transformation needs using just the built-in ansible filters you will probably end up in the same spot each time, an unreadable string of chained filters. I'll go over how you can make these complex transformations easier by using custom filters in your ansible playbooks so that they are readable and reusable.

Creating a custom filter

Take this example for a good candidate for a custom ansible filter. We have an S3 bucket with each top-level folder representing a package. We want to create a playbook to audit the bucket and find each top-level folder present in the bucket.

However, if we were to perform a list on the s3 bucket we would get the following output.

- folder1/file1.js
- folder1/sub-folder/file1.js
- folder2/
- folder3/file1.js
- folder3/file2.js
- folder3/file3.js

This list shows the key of each object in the bucket as the concept of folders is emulated by using paths in keys. Now to get each top-level folder in the bucket we can simply get the first folder in the path and filter these folders to there unique items. And hopefully get something like below:

folder1
folder2
folder3

To create a new filter module just create a .py file in a directory named filter_plugins (You can name this file anything, all .py files are automatically picked up from the filter_plugins folder) that is adjacent to our playbook in the file tree. We will create a python module that defines a FilterModule class. This class must have a method called filters that returns a map of names and callable implementations for the filter.

To create a custom filter to gather the unique root folders of our bucket we could write something like:

#!/usr/bin/python
class FilterModule(object):
    def filters(self):
        return {'unique_folders': self.unique_base_paths}

    def unique_base_paths(self, keys):
        # A Set allows for no duplication of keys
        base_paths = set()
        for key in keys:
            # A path contains a directory if it has a "/"
            if "/" in key:
                # Get the first folder in each path
                base_folder = key.split("/")[0]
                base_paths.add(base_folder)

        # Sets aren't handled by ansible so convert to list
        return list(base_paths)

Our custom ansible functions can now be called in our playbooks by calling our filter named unique_folders just like any other filter.

- hosts: localhost

  vars:
    deployment_bucket:
      name: "some-bucket-with-packages"
      region: ap-southeast-2

  tasks:
    - name: List keys
      aws_s3:
        bucket: "{{ deployment_bucket.name }}"
        region: "{{ deployment_bucket.region }}"
        mode: list
      register: bucket_content

    - name: All bucket paths
      debug:
        var: bucket_content.s3_keys

    - name: Get all top level folders in bucket
      set_fact:
        deployed_packages: "{{ bucket_content | unique_folders }}"

    - name: Unique top level folders
      debug:
        var: deployed_packages

And once we run our playbook the final debug output should show just the top level folders in the bucket:

TASK [Unique top level folders] *********************
ok: [localhost] => {
    "deployed_packages": [
        "folder1",
        "folder2",
        "folder3"
    ]
}

Final thoughts

Custom filters are a powerful tool to move some of your complex text transformations to some more readable python code rather than the alternative of long chaining of the built-in ansible filters. These filters as they are just python code are reusable across projects allowing you to build useful filters for validating your custom data types and structures easily in your ansible playbooks.

Further reading


Cover photo by Anders Jildén on Unsplash

If you enjoyed this post or any of the content on Code Punnet, please consider subscribing to our weekly newsletter and following our RSS feed or twitter.

Aaron Berry

I'm a big fan of backend programming, dev-ops automation and all things in the cloud. I like to write about things I'm currently working on and new things I've learnt / learning.

Read more posts by this author.