Github Actions That Commit to Github

I’m enjoying experimenting with GitHub Actions, and have now used the new beta for a number of projects and tasks. I’ve mainly been interested in Actions for ad-hoc automation, either based on a schedule or on an event like pushing to master. Several of those projects have involved having an Action generate some files and commit those back to GitHub. At the moment doing so requires some jumping through hoops, so I thought it would be worthwhile documenting.

In my case I’m using the following in a few projects:

  • Updating a Homebrew Formula when a new version of upstream software is released
  • Generating a set of GitHub Actions and documentation from some input data

The following snippet commits back to GitHub, correctly authenticating with the repository and setting the user details for the commit.

- name: Commit to repository
    GITHUB_TOKEN: ${{ secrets.github_token }}
      <a commit message>
      skip-checks: true
  run: |
    # Hard-code user configuration
    git config "<an email address>"
    git config "<a name>"
    # Update origin with token
    git remote set-url origin https://x-access-token:${GITHUB_TOKEN}${GITHUB_REPOSITORY}.git
    # Checkout the branch so we can push back to it
    git checkout master
    git add .
    # Only commit and push if we have changes
    git diff --quiet && git diff --staged --quiet || (git commit -m "${COMMIT_MSG}"; git push origin master)

It would be nice to have some sort of abstraction here, given that’s quite a bit of code to achieve what we’re wanting to do. It would be nice for this to be build-in to GitHub Actions, as I can see this being something that’s useful in lots of places. But for the moment it’s nice to know it’s possible. It might even be nice to encapsulate this into an Action of it’s own.

Thanks to Steve Winton for the original pointers that helped me get this working. Hopefully it comes in useful to others too.

Running Tekton in a Github Action

I really like Tekton. The project is at a fairly early stage, but is building the primitives (around Tasks and Pipelines) that other higher-level (and user facing) tools can be built on. It’s plumbing, with lots of potential for sharing and abstraction. So my kind of project.

Now Tekton’s abstractions are similar to GitHub Actions. Tasks broadly map to Actions and Pipelines to Workflows. I’d love to see this broken out of the separate implementations and a general standard emerge. Portable abstractions would allow for more innovation on top and less make work for integrators like me. For instance see the overlap in the Kubeval Action and the Kubeval Task. But that’s a separate tangent to the one in this post.

In this post I’m doing something questionable. I’m going to use GitHub Actions to:

  1. Spin up an ephemeral Kubernetes cluster for each change
  2. Install Tekton and some tasks on the cluster
  3. Run a task
  4. Grab the results

Now while you could use this to run Tekton tasks on GitHub instead of Actions, you would be paying quite a large performance cost and wasting lots of compute cycles to do it. This is however useful in a few ways. It’s handy if you’re building and testing Tekton tasks or pipelines. It’s also useful as an example of the terrible things you can do with a platform as flexible as GitHub Actions.

Show me how already

Save the following at .github/workflows/push.yml:

name: "Demonstrate using Tekton Pipelines in GitHub Actions"
on: [pull_request, push]

    runs-on: ubuntu-latest
    - uses: actions/checkout@master
    - uses: engineerd/setup-kind@v0.1.0
    - name: Install jq
      run: |
        sudo apt-get install jq
    - name: Install Tekton
      run: |
        export KUBECONFIG="$(kind get kubeconfig-path)"
        kubectl apply -f
    - name: Install Kubeval Tekton Task
      run: |
        export KUBECONFIG="$(kind get kubeconfig-path)"
        kubectl apply -f
    - name: Run Kubeval Task
      run: |
        export KUBECONFIG="$(kind get kubeconfig-path)"
        kubectl apply -f taskrun.yaml
        STATUS=$(kubectl get taskrun kubeval-example -o json | jq -rc .status.conditions[0].status)
        while [ "${STATUS}" != "Unknown" ]; do
          if [ $SECONDS -gt $LIMIT ]
            echo "Timeout waiting for taskrun to complete"
            exit 2
          sleep 10
          echo "Waiting for taskrun to complete"
          STATUS=$(kubectl get taskrun kubeval-example -o json | jq -rc .status.conditions[0].status)
    - name: Install Tekton CLI
      run: |
        curl -LO
        sudo tar xvzf tkn_0.2.2_Linux_x86_64.tar.gz -C /usr/local/bin/ tkn
    - name: Get TaskRun Logs
      run: |
        export KUBECONFIG="$(kind get kubeconfig-path)"
        tkn taskrun logs kubeval-example -a -f
    - name: Result
      run: |
        export KUBECONFIG="$(kind get kubeconfig-path)"
        REASON=$(kubectl get taskrun kubeval-example -o json | jq -rc .status.conditions[0].reason)
        echo "The job ${REASON}"
        test ${REASON} != "Failed"

You can see this running in garethr/tekton-in-github-actions.

This is terrible isn’t it?

It is. Mixing bash and YAML is however how the internet works now. The above is also really just a proof-of-concept. There are a number of things we can do to make the above nicer for general usage. To begin with we can abstract it away behind an action! I think you could probably get the above down to something like the following:

- uses: actions/checkout@master
- uses: tekton/setup-tekton@v0.1.0
- uses: tekton/run-tekton-task@v0.1.0
    task: taskrun.yaml

If you don’t like the idea of hiding all of that YAML and bash behind some more YAML and a Docker image then please don’t look at how most software works :)

The above example also has at least one more flaw. If you take a look at the TaskRun soec, it’s actually testing code from a different repository rather than this one. Now Actions happily provides the relevant details as environment variables, namely GITHUB_REPOSITORY and GITHUB_SHA. With a bit of envsubst or sed you could swap in the relevant local values into the TaskRun template.

Happily, you can also now implement Actions in TypeScript, so you could write an action in a real programming language using the Kubernetes client libraries. That would be the best approach for any sense of maintainability if someone wanted to make this a real thing.

In summary

Thanks to Radu for the Kind Action, he also has a nice post up on writing it. This post mainly highlights how far you can stretch GitHub Actions without it breaking, which is always a useful property in my book. It does mean folks are going to do terrible things. All those things people did with Jenkins? They are now in the cloud and powered by more events.

More usefully, this also demonstrates how useful GitHub Actions could be in testing anything that integrates with Kubernetes. Kind was build specifically for this usecase, and being able to run it close to the code and suitably abstracted away is powerful for anyone building clients or tools which use the API.

Running Falco on Docker Desktop

Falco is a handy open source project for intrusion and abnormality detection. It nicely integrates with Kubernetes as well as other linux platforms. But trying it out with Docker Desktop (for Mac or Windows), either using Docker, the built-in Kubernetes cluster, or with Kind requires a new kernel module.

This is slighly complicated by the fact the VM used by Docker Desktop is really not intended for direct management by the end user. It’s an implementation detail of how Docker Desktop works. The VM is also based on LinuxKit which is a toolkit for building small limited purpose operating systems.

Luckily, it is possible to work around the above. But I couldn’t find this documented outside a few GitHub issues, and even then without much context. Hence this post.

Building the Kernel module

In order to build and install the Kernel module we’re going to use Docker itself. Kernel modules need to align with the running Kernel, so the following Dockerfile takes several build args.

  • ALPINE_VERSION - the version of the Alpine operating system, this should match the Docker Desktop version you are using
  • KERNEL_VERSION - again, this needs to match the version in use by Docker Desktop
  • FALCO_VERSION - the version of Falco you plan on using
  • SYSDIG_VERSION - the version of Sysdig used by Falco

For the Alpine and Kernel versions you can check the release notes of the Docker Desktop version you are running. You can check this online for Mac and for Windows or you can access them from the About menu where you should find a Release notes link.

The following Dockerfile has default values for the latest versions of everything, including Docker Desktop (37199) which uses Alpine 3.10 and a 4.9.184 Kernel.


FROM alpine:${ALPINE_VERSION} AS alpine

FROM linuxkit/kernel:${KERNEL_VERSION} AS kernel

FROM alpine

COPY --from=kernel /kernel-dev.tar /

RUN apk add --no-cache --update wget ca-certificates \
    build-base gcc abuild binutils \
    bc \
    cmake \
    git \
    autoconf && \
  export KERNEL_VERSION=`uname -r  | cut -d '-' -f 1`  && \
  export KERNEL_DIR=/usr/src/linux-headers-${KERNEL_VERSION}-linuxkit/ && \
  tar xf /kernel-dev.tar && \
  cd $KERNEL_DIR && \
  zcat /proc/1/root/proc/config.gz > .config && \
  make olddefconfig && \
  mkdir -p /falco/build && \
  mkdir /src && \
  cd /src && \
  wget$FALCO_VERSION.tar.gz && \
  tar zxf $FALCO_VERION.tar.gz && \
  wget$SYSDIG_VERSION.tar.gz && \
  tar zxf $SYSDIG_VERSION.tar.gz && \
  mv sysdig-$SYSDIG_VERSION sysdig && \
  cd /falco/build && \
  cmake /src/falco-$FALCO_VERSION && \
  make driver && \
  rm -rf /src && \
  apk del wget ca-certificates \
    build-base gcc abuild binutils \
    bc \
    cmake \
    git \

CMD ["insmod","/falco/build/driver/falco-probe.ko"]

Building an image from the above Dockerfile will compile the kernel module, and then running the resulting image will install it. Note that we need to run the container as --privileged in order to install the module on the host VM.

docker build -t falco-docker-desktop .
docker run -it --rm --privileged falco-docker-desktop

If you’re using a newer version of DOcker Desktop, or want to use a specific version of Falco, you can use the same Dockerfile and pass in build arguments. For instance if you are running the Edge version of Docker Desktop ( this has a newer kernel. Currently you would want to build the image like so:

docker build --build-arg KERNEL_VERSION=4.14.131 -t falco-docker-desktop .

Running Falco

With the module installed we can now try out Falco locally. The following is a very simple demonstration, using the built-in rules and just using the Docker runtime.

$ docker run -e "SYSDIG_SKIP_LOAD=1" -it --rm --name falco --privileged -v /var/run/docker.sock:/host/var/run/docker
.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro falcosecurity/falco

2019-08-23T14:32:52+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2019-08-23T14:32:52+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2019-08-23T14:32:52+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2019-08-23T14:32:52+0000: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
2019-08-23T14:32:53+0000: Starting internal webserver, listening on port 8765
2019-08-23T14:32:53.499871000+0000: Notice Container with sensitive mount started (user=<NA> command=container:814be8a1ef88 k8s_snyk-monitor_snyk-monitor-68d5f8d85f-vxqbn_snyk-monitor_643e8eb1-c33d-11e9-b290-025000000001_28 (id=814be8a1ef88) image=snyk/kubernetes-monitor:latest mounts=/var/lib/kubelet/pods/643e8eb1-c33d-11e9-b290-025000000001/volumes/,/var/lib/kubelet/pods/643e8eb1-c33d-11e9-b290-025000000001/etc-hosts:/etc/hosts::true:rprivate,/var/lib/kubelet/pods/643e8eb1-c33d-11e9-b290-025000000001/containers/snyk-monitor/41e63105:/dev/termination-log::true:rprivate,/var/run/docker.sock:/var/run/docker.sock::true:rprivate,/var/lib/kubelet/pods/643e8eb1-c33d-11e9-b290-025000000001/volumes/
2019-08-23T14:32:53.514614000+0000: Notice Privileged container started (user=<NA> command=container:7603e8ff28a7 falco (id=7603e8ff28a7) image=falcosecurity/falco:latest)

Note Falco noticed a container starting which was mounting sensitive information from the host, and also noticed a privileged container starting. In this case that container was Falco itself :)

Using Conftest and Kubeval With Helm

I maintain a few open source projects that help with testing configuration, namely Kubeval and Conftest. Recently I’ve been hacking on various integrations for these tools, the first of which are plugins for Helm.

Validate Helm Charts with Kubeval

Kubeval validates Kubernetes manifests against the upstream Kubernetes schemas. It’s useful for catching invalid configs, especially in CI environments or when testing against multiple different versions of Kubernetes. Lots of folks have been using Kubeval with Helm for a while, mainly using helm template and piping to kubeval on stdin. The Helm Kubeval plugin makes that easier to do just from Helm.

You can install the Helm plugin using the Helm plugin manager:

helm plugin install

With that installed, let’s grab a sample chart and run helm kubeval <path>:

$ git clone
$ helm kubeval charts/stable/nginx-ingress
The file nginx-ingress/templates/serviceaccount.yaml contains a valid ServiceAccount
The file nginx-ingress/templates/clusterrole.yaml contains a valid ClusterRole
The file nginx-ingress/templates/clusterrolebinding.yaml contains a valid ClusterRoleBinding
The file nginx-ingress/templates/role.yaml contains a valid Role
The file nginx-ingress/templates/rolebinding.yaml contains a valid RoleBinding
The file nginx-ingress/templates/controller-service.yaml contains a valid Service
The file nginx-ingress/templates/default-backend-service.yaml contains a valid Service
The file nginx-ingress/templates/controller-deployment.yaml contains a valid Deployment
The file nginx-ingress/templates/default-backend-deployment.yaml contains a valid Deployment
The file nginx-ingress/templates/controller-configmap.yaml contains an empty YAML document
The file nginx-ingress/templates/controller-daemonset.yaml contains an empty YAML document
The file nginx-ingress/templates/controller-hpa.yaml contains an empty YAML document
The file nginx-ingress/templates/controller-metrics-service.yaml contains an empty YAML document
The file nginx-ingress/templates/controller-poddisruptionbudget.yaml contains an empty YAML document
The file nginx-ingress/templates/controller-servicemonitor.yaml contains an empty YAML document
The file nginx-ingress/templates/controller-stats-service.yaml contains an empty YAML document
The file nginx-ingress/templates/default-backend-poddisruptionbudget.yaml contains an empty YAML document
The file nginx-ingress/templates/headers-configmap.yaml contains an empty YAML document
The file nginx-ingress/templates/podsecuritypolicy.yaml contains an empty YAML document
The file nginx-ingress/templates/tcp-configmap.yaml contains an empty YAML document
The file nginx-ingress/templates/udp-configmap.yaml contains an empty YAML document

Here we can see the various parts of the template are picked up and, in this case, contain valid Kubernetes resources.

The plugin also supports the various flags on Kubeval, so you can for example test charts against other versions of Kubernetes.

helm kubeval . -v 1.9.0

You can also test variations on the chart, for instance by setting particular values before validating.

helm kubeval charts/stable/nginx-ingress --set controller.image.tag=latest

Check Charts for security issues with Conftest

Conftest is a little more general purpose than Kubeval. Where as Kubeval is specific to Kubernetes, Conftest is used for testing all kinds of configuration. That means you need to bring your own tests or policy, which is written using Rego and Open Policy Agent.

The Helm Conftest plugin makes it easy to use Conftest with Helm, in a similar way to the above Kubeval plugin. Installation is easy with the Helm plugin manager.

helm plugin install

Conftest needs you to write or otherwise aquire some policies. For the purposes of this example we’ll use an existing policy, written to test various security properties of Kubernetes configurations.

Create a file called conftest.toml in the same directory as the chart and set it to download our sample policy.

repository = ""
tag = "latest"

With that in place we can run helm conftest. The --update flag asks the plugin to download any policies from the config file before running the tests.

$ helm conftest . --update
FAIL - release-name-test in the Pod release-name-mysql-test does not have a memory limit set
FAIL - test-framework in the Pod release-name-mysql-test does not have a memory limit set
FAIL - release-name-test in the Pod release-name-mysql-test does not have a CPU limit set
FAIL - test-framework in the Pod release-name-mysql-test does not have a CPU limit set
FAIL - release-name-test in the Pod release-name-mysql-test doesn't drop all capabilities
FAIL - test-framework in the Pod release-name-mysql-test doesn't drop all capabilities
FAIL - release-name-test in the Pod release-name-mysql-test is not using a read only root filesystem
FAIL - test-framework in the Pod release-name-mysql-test is not using a read only root filesystem
FAIL - release-name-test in the Pod release-name-mysql-test is running as root
FAIL - test-framework in the Pod release-name-mysql-test is running as root
FAIL - The Pod release-name-mysql-test is mounting the Docker socket
FAIL - release-name-mysql in the Deployment release-name-mysql does not have a memory limit set
FAIL - remove-lost-found in the Deployment release-name-mysql does not have a memory limit set
FAIL - release-name-mysql in the Deployment release-name-mysql does not have a CPU limit set
FAIL - remove-lost-found in the Deployment release-name-mysql does not have a CPU limit set
FAIL - release-name-mysql in the Deployment release-name-mysql doesn't drop all capabilities
FAIL - remove-lost-found in the Deployment release-name-mysql doesn't drop all capabilities
FAIL - release-name-mysql in the Deployment release-name-mysql is not using a read only root filesystem
FAIL - remove-lost-found in the Deployment release-name-mysql is not using a read only root filesystem
FAIL - release-name-mysql in the Deployment release-name-mysql is running as root
FAIL - remove-lost-found in the Deployment release-name-mysql is running as root
FAIL - The Deployment release-name-mysql is mounting the Docker socket
FAIL - release-name-mysql must include Kubernetes recommended labels:

Here we can see various failures, mainly related to not setting CPU and memory limits, running as root and not providing the expected labels.

You can write your own polices to cover any aspect of the configuration, usin the powerful Rego language from Open Policy Agent. You can find more example in the Conftest repository.

Introducing Conftest

For the past few months I’ve been hacking on Conftest, a tool for writing tests for configuration, using Open Policy Agent. I spoke about Conftest at KubeCon but am only now getting round to writing up a quick introducion.

The problem

We’re all busy writing lots of configuration. There are already more than 1 million Kubernetes config files public on GitHub, and about 70 million YAML files too. We’re using a range of tools, from CUE to Kustomize to templates to DSLs to general purpose programming languages to writing YAML by hand. But we’re rarely automatically enforcing policy against any of that configuration. That leaves two approaches to spread best practices; manually via code review, or not at all. Conftest is aiming to be a handy tool to help with that problem.

Open Policy Agent

Conftest builds on Open Policy Agent.

Open Policy Agent (OPA) is a general-purpose policy engine with uses ranging from authorization and admission control to data filtering. OPA provides greater flexibility and expressiveness than hard-coded service logic or ad-hoc domain-specific languages. And it comes with powerful tooling to help you get started.

OPA introduces the Rego data assertion language

Rego was inspired by Datalog, which is a well understood, decades old query language. Rego extends Datalog to support structured document models such as JSON.

Most of the usecases OPA has been applied to so far have been on the server side. Authentication systems, proxies, Kubernetes admission controllers, storage system policies. That says more about the flexibility of OPA more than anything else though. OPA is a very general purpose tool with lots of potential uses. Conftest simply runs with that in the direction of a nice local user interface, and usage in continuous integration pipelines.

Conftest for Kubernetes

Conftest is available from Homebrew for macOS and from Scoop for Windows, Docker images, as well as executables being you can download directly. You can find installation instructions in the README. With Conftest installed, let’s write a simple test for a Kubernetes configuration file.

Save the following as policy/base.rego. policy is just the default directory where Conftest will look for .rego policy files, you can override that if desired.

package main

deny[msg] {
  input.kind = "Deployment"
  not input.spec.template.spec.securityContext.runAsNonRoot = true
  msg = "Containers must not run as root"

deny[msg] {
  input.kind = "Deployment"
  msg = "Containers must provide app label for pod selectors"

The tests here are:

  1. Checking for Deployments with containers running as root
  2. Checking for Deployments with containers without a specific label

Both realistic, if simple, examples of the type of thing you might want to enforce in your cluster. Assuming we have our Deployment described in a file called deployment.yaml we would run Conftest like so:

$ conftest test deployment.yaml
   Containers must not run as root
   Deployments are not allowed

Here we see our Deployment failed both tests and Conftest is returning the message we defined in the policy above. Conftest will return a non-zero status code when tests fail, and can also take input via stdin rather than reading files.

$ cat deployment.yaml | conftest test -
   Containers must not run as root
   Deployments are not allowed

The above example tests a single configuration via against a single policy file. However you can split your policies over multiple files in the policy directory, and also point Conftest at multiple files at the same time as well as multi-document YAML files.

Conftest for other structured data

Conftest isn’t just for Kubernetes configs. You can use it with any structured data, starting with JSON and YAML. More input formats are likely to be supported in the future.

The repository has examples using Conftest to test CUE, Typescript, Docker Compose, Serverless configs and Terraform state. Let’s take a look at a Terraform example.

package main

blacklist = [

deny[msg] {
  check_resources(input.resource_changes, blacklist)
  banned := concat(", ", blacklist)
  msg = sprintf("Terraform plan will change prohibited resources in the following namespaces: %v", [banned])

# Checks whether the plan will cause resources with certain prefixes to change
check_resources(resources, disallowed_prefixes) {
  startswith(resources[_].type, disallowed_prefixes[_])

Here the policy is a little more complicated than the examples above, showing some of the power of Rego. Here we check the list of resource changes for any resources on a blacklist.

Conclusion and next steps

This post is just a quick introduction to Conftest. The tool already has a few features I’ll try and cover in future posts, including:

  • Store OPA bundles in OCI registries, making reusing policies easier
  • Debugging and tracing the Rego assertions
  • Usage in a CI pipeline
  • A kubectl plug for making assertions against a running cluster
  • Workflows for using Conftest locally and then loading the policies into Gatekeeper

It also needs a bit of tidying up and better testing now I have the basic UI down.

Conftest is looking for other contributors as well. If you have ideas for features, or fancy hacking on a small but useful Go tool, a few of the current issues are marked as good for test time contributors.

Emphemeral Clusters for Helm Charts and Operators

Building on the previous post, I found myself wanting to grab quick clusters for experimenting with Helm Charts and with Kubernetes Operators.


helm-%: cluster-%
        kubectl -n kube-system create sa tiller
        kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
        helm init --service-account tiller --kubeconfig=$$(kind get kubeconfig-path --name $(NAME))
        helm repo --kubeconfig=$$(kind get kubeconfig-path --name $(NAME)) add incubator

The above target for our Makefile makes it easy to grab a new Kind cluster and instantiate Helm. Here we’re setting up a new service account for the Tiller component and ensuring Helm has the right permissions to launch things on the cluster. We’re not attempting to secure the cluster in any way here, this is intended purely for throwaway clusters for testing after all. The following will launch a new cluster named clustername with Helm already installed:

make helm-clustername

The cluster won’t have any Helm Charts installed yet, but you should be able to run helm install once you point your KUBECONFIG environment variable at the new cluster as described when you run the command.

export KUBECONFIG=(kind get kubeconfig-path --name="clustername")


With the new Operator Hub serving as a repository for finding and installing Kubernetes Operators, it was simple enough to add support for bootstrapping and installing operators into a new cluster.

operator-%: cluster-%

The above snippet added to our Makefile allows us to run commands like the following:

make operator-etcdoperator

This will create a brand new Kind cluster, install the Operator Lifecycle Manager, and the install the operator specified in the target name, in this case etcdoperator

Skopeo as a Docker Image

Skopeo is a handy tool for interogating OCI registries. You can inspect the image manifests and copy images between various stores. I found myself wantting to use Skopeo in the context of a container, and having searched on Hub mainly found either out-of-date images or images designed for a slightly different purpose. Maintaining images is non-trivial, and sometimes it’s better to just let folks know how to build there own. So for anyone else in need of such a thing, here is a Skopeo image based on Alpine linux.

FROM golang:1.12-alpine AS builder

RUN apk add --no-cache \
    git \
    make \
    gcc \
    musl-dev \
    btrfs-progs-dev \
    lvm2-dev \
    gpgme-dev \
    glib-dev || apk update && apk upgrade

WORKDIR /go/src/
RUN git clone .
RUN make binary-local-static DISABLE_CGO=1

FROM alpine:3.7
run apk add --no-cache ca-certificates
COPY --from=builder /go/src/ /usr/local/bin/skopeo
COPY --from=builder /go/src/ /etc/containers/policy.json
ENTRYPOINT ["/usr/local/bin/skopeo"]
CMD ["--help"]

Using the image is straightforward if you’re familiar with Skopeo and Docker. You can inspect an image like so:

$ docker run -it --rm garethr/skopeo inspect docker://
    "Name": "",
    "Digest": "sha256:2a60898a6dd7da9964b0c59fedcf652e24bfff04142e5488f793c9e8156afd33",
    "RepoTags": [
    "Created": "2019-03-12T00:20:38.300667849Z",
    "DockerVersion": "18.06.1-ce",
    "Labels": {
        "maintainer": "Clement Verna <>"
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [

Copying is a little more complicated, assuming you want to save the image locally you’ll need to mount an empty directory, or keep the image around and use docker cp. For instance:

$ docker run -it --rm -v $PWD/data:/data garethr/skopeo copy docker://alpine:latest oci:data/alpine:latest                                                                   Sun  2 Jun 12:24:39 2019
Getting image source signatures
Copying blob e7c96db7181b done
Copying config 8c79fc7093 done
Writing manifest to image destination
Storing signatures

That should have saved the image like so:

$ tree data/                                                                                                                                                                 Sun  2 Jun 12:27:14 2019
└── alpine
    ├── blobs
    │   └── sha256
    │       ├── 63aec9aa7e327bf2359eca3a8345b1678b6a592241916427dbeb1d884eb3cda2
    │       ├── 8c79fc709348f9fdb30cc2dc10999b30e095fc7cad8aa9320e8834aca05f7740
    │       └── e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10
    ├── index.json
    └── oci-layout

3 directories, 5 file

Ephemeral Kubernetes Clusters With Kind and Make

There are a number of options for persistent local Kubernetes clusters, but when you’re developing tools against the Kubernetes APIs it’s often best to be throwing things away fairly regularly. Enter Kind. Originally designed as a tool for testing Kubernetes itself, Kind runs a working Kubernetes cluster on top of Docker.

At its simplest we can create a new Kubernetes cluster like so:

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.14.2) 🖼
 ✓ Preparing nodes 📦
 ✓ Creating kubeadm config 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Cluster creation complete. You can now use the cluster with:

export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
kubectl cluster-info

The instructions from running the command show how to connect to the new cluster. Running the Docker commands will show the running container.

$ docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED              STATUS              PORTS                                  NAMES
0e39e49feaed        kindest/node:v1.14.2   "/usr/local/bin/entr…"   About a minute ago   Up About a minute   61227/tcp,>6443/tcp   kind-control-plan

Automating Kind

You can use the Kind CLI tool directly, but it’s also a handy utility for simple automation tasks. The Roadmap promises better support for Kind as a Go library in the future, but for the moment I’ve started using make.

WAIT := 200s
APPLY = kubectl apply --kubeconfig=$$(kind get kubeconfig-path --name $@) --validate=false --filename
NAME = $$(echo $@ | cut -d "-" -f 2- | sed "s/%*$$//")

tekton tekton%: create-tekton%

gatekeeper gatekeeper%: create-gatekeeper%

        -@kind create cluster --name $(NAME) --wait $(WAIT)

        @kind delete cluster --name $(NAME)

        @kind get kubeconfig-path --name $(NAME)

        @kind get clusters | xargs -L1 -I% kind delete cluster --name %

        @kind get clusters

.PHONY: tekton gatekeeper tekton% gatekeeper% create-% delete-% env-% clean list

The above Makefile provides a few handy shortcuts for Kind commands. Not strictly necessary but useful for consistency. So I can now run commands like:

$ make create-hello
# Create a new cluster called hello
$ make delete-hello
# Delete the hello cluster
$ make list
# Provide a list of all cluster
$ make clean
# Delete all of my clusters

More useful are the higher-level commands, in the example above make tekton and make gatekeeper. Tekton is a project aiming to provide Kubernetes-style resources for declaring CI/CD pipelines. Gatekeeper provides a policy controller for Kubernetes, using Open Policy Agent. Both projects add new custom resources to Kubernetes. With the Makefile above I can run one command and spin up an ephemeral Kubernetes cluster pre-provisioned with Tekton, or Gatekeeper. Because I got carried away the Makefile also supports grabbing multiple Tekton or Gatekeeper clusters with different names, so the following works too:

$ make tekton1
$ make tekton2
$ make tekton-loves-make


I can imagine Kind, or a sub-project, growing a Kindfile in the future. And an improved CLI which moves beyond the initial cluster testing scenarios it was originally designed for. But for now, and for me at least, Make makes a great prototyping tool. I can grab arbitrary ephemeral Kubernetss clusters until I run out of memory, and adding new projects I’m interested in is trivial. Just as importantly thowing them away is easy too.

Make is a great tool to have in your toolbox in my experience, and learning just enough to be dangerous goes a long way in terms of expressiveness and power. But as mentioned before, I have a bit of a thing for DSLs.

Setting Up a New Mac For Development

With a new job (at Snyk) comes the opportunity to setup a new machine from scatch. I’ve always taken a perverse pleasure in building development machines for myself. It’s also the first time I’ve been back on a Mac (a MacBook Pro 13” to be precise) for a while so always new things to play with.


The reality is I don’t use much beyond a web browser and a terminal (partly a concious decision, it makes it easier to move between different operating systems and computers). On Mac Homebrew is my go-to starting point. Here’s a list of the various packages I have installed to start with.

$ brew list
adns                    libassuan               pcre2
asdf                    libevent                pinentry
autoconf                libffi                  pkg-config
automake                libgcrypt               python
bats                    libgpg-error            readline
conftest                libidn2                 rlwrap
coreutils               libksba                 skaffold
curl                    libtasn1                snyk
fish                    libtool                 sqlite
gdbm                    libunistring            terraform
gettext                 libusb                  tilt
git                     libxml2                 tmux
git-credential-netlify  libxslt                 tree
git-lfs                 libyaml                 unbound
gmp                     ncurses                 unixodbc
gnupg                   nettle                  unzip
gnutls                  npth                    wget
goreleaser              opa                     xz
hugo                    openssl                 zlib
kubeval                 p11-kit

Some of these are dependencies of other packages, or small system tools. The others of interest are:

  • fish - I’m a convert to the Fish shell, mainly for the excellent defaults which keep configuration to an absolute minimum
  • tmux - TMUX is my go-to shell environment
  • bats - I still turn to bats regularly for high-level acceptance tests
  • terraform - I’m not a heavy Terraform user, but I do have an interest in Terraform tooling
  • snyk - :)
  • conftest - My latest open source project, using Open Policy Agent to test structured data
  • kubeval - Another of my projects, Kubeval helps validate Kubernetes configurations using the upstream schemas
  • hugo - I use Hugo for building this site, and a few other small sites I maintain
  • goreleaser - GoReleaser is a fantastic tool for releasing Go projects
  • opa - I’m experimenting with Open Policy Agent for a few things at the moment, including conftest above
  • tilt - A very handy tool for local development against Kubernetes


For installing various programming language environments I took a go at using asdf and I’ve been very impressed. So far I’ve installed the following.

$ asdf list

The asdf version manager provides a similar interface to all of those platform specific tools (like rbenv, pyenv, gvm, nvm, etc.) but has a plugin system, and has plugins for most language environments you might be interested in.

$ asdf plugin-add lua
# installs the plugin for managing lua
$ asdf list-all lua
# lists all available versions on lua that can be installed
$ asdf install lua 5.3.5
# installs a specific version
$ asdf global lua 5.3.5
# sets the version to be used everywhere that doesn't have a local override


On top of the command line tools and language toolchains I installed Docker Desktop (obviously) to provide a nice Docker environment. I also installed Spectacle as a simple keyboard-powered windows manager.

I installed a few things globally within the above development environments. Where I can avoid it I prefer not to do so, I’d much rather have standalone tools, but some things are new enough that they haven’t release packages outside a development toolchain yet.

  • Krew - Krew provides a plugin manager for kubectl (custom installer)
  • Kind - Running ephemeral Kubernetes clusters on top of Docker is fantastic for testing (Go module)
  • CUE - A data constraint language well-suited to generating configuration (Go module)
  • Netlify - This and other sites I maintain run on netlify, so having the CLI installed is handy for management (NPM module)
  • TypeScript - The compiler and language toolchain builds atop the NodeJS tools (NPM module)

I’m sure I’ll install more things as I go along, and I’m sure I’ll have missed something that I’ll remember the moment I publish this post, but all-in-all I’m up and running with a nice development machine quickly and fairly painlessly thanks to good package management tools and the work of lots of folks to package up and maintain packages for the wide various of software I use.

Automating the CUE workflow with Tilt

We now have a nice configuration language in which to author our configs (CUE), and a way of validating and testing that configuration using Kubeval and Conftest.

Next we want to wrap that in a little automation. When writing our configuration we probably want to be running it against a Kubernetes cluster as we make changes. For that I’m going to use Tilt.

Before we jump into the Tilt configuration I’ll add another useful CUE commannd. Tilt doesn’t yet support CUE directly (not unexpected given the age of both projects) but we can integrate them manually. To do so we need a command which will dump out the multi-file YAML document to stdout for our CUE configuration. Save the following as dump_tool.cue:

package kubernetes

import "encoding/yaml"

command dump: {
  task print: {
    kind: "print"
    text: yaml.MarshalStream(objects)

With that in place let’s create a Tiltfile. Tiltfile uses another DSL, written in Skylark which is a dialect of Python. I’m showing a very simple example here but you can do a lot more if you check out the API docs. Save the following as Tiltfile:

local("cue dump | kubeval")
local("cue dump | conftest -")
config = local("cue dump")

This should be reasonably simple to understand, but for clarify:

  1. We use read_file to make sure Tilt knows to re-run whenever it sees a change to deployment.cue
  2. We run both cue validate and cue test, though note that without this issue being resolved this won’t actual fail
  3. We then run cue dump to get the YAML representation and store it in a variable called config
  4. Finally we pass that configuration to the Kubernetes API to create the relevant objects

With that in place we can now run tilt up to run everything.

$ tilt up

Tilt provides a handy CLI user interface for interrogating logs and generally seeing what’s happening. This makes debugging your configuration easy and fast. Now whenever you change the deployment.cue file, or the OPA tests, it will re-run the validation, tests and redeploy to your Kubernetes cluster.

You like DSLs then?

I mentioned in the first post of this series a certain love of DSLs. If you’ve followed along with each post you might have spotted:

  • CUE - for defining the configuration itself
  • Rego - for writing the assertions used for testing the configuration
  • Skylark - for writing the Tiltfile

I appreciate not everyone likes a good DSL, and that three of them combined together in this way is going to cause some folks to get angry. I’m not saying this is the ideal workflow for all (or any) teams to adopt. I don’t think that ideal exists howeer, people and teams are different and have different constrains and sensibilities. It’s also why we won’t all just end up writing Lisp.

But if someone wants to say things about “not proper programming languages” or complain about configuration vs code or pretend configuration can’t be programmatically tested or validated then feel free to point them to these posts.

Testing Cue Configuration with Open Policy Agent

In the first two posts we have written some configuration using CUE and validated it against the Kubernetes schemas using Kubeval. In this post we’re going to expand testing to include custom assertions.

As mentioned before, I’m using Kubernetes configuration as an example here. The same approach is just as valid for other structured data configs like CloudFormation, Azure Resource Manager templates, Circle CI configs, etc.

Syntactically valid configuration doesn’t make it correct. It might for instance breach some internal policy or other, for example:

  • Disallow containers running as root
  • Mandate certain labels are set of management or auditing purposes
  • Ban images using the latest tag or without a tag at all
  • Restrict which resources can be used

Let’s demonstrate this by starting with a slightly modified version of our deployment written in CUE.

package kubernetes

deployment <Name>: {
  apiVersion: string
  kind:       "Deployment"
  metadata name: Name
  spec: {
    replicas: 1 | int
    template: {
      metadata labels app: Name
      spec containers: [{name: Name}]

deployment "hello-kubernetes": {
  apiVersion: "apps/v1"
  spec: {
    replicas: 3
    template spec containers: [{
      image: "paulbouwer/hello-kubernetes:1.5"
      ports: [{
        containerPort: 8080

Introducing conftest

Conftest is a new project I’ve been working on. It’s intended for writing tests against structured data, using the Rego language from Open Policy Agent. With conftest installed we can wire it up to CUE as we did with Kubeval.

package kubernetes

import "encoding/yaml"

command test: {
  task conftest: {
    kind:   "exec"
    cmd:    "conftest -"
    stdin:  yaml.MarshalStream(objects)
    stdout: string
  task display: {
    kind: "print"
    text: task.conftest.stdout

Open Policy Agent

Open Policy Agent, or OPA for short, is a super interesting project which has a wide range of usecases. It’s described as a general purpose policy engine and already has several more specific subprojects, for instance gatekeeper for Kubernetes and an Istio plugin. The documentation has examples of enforcing policy around AWS IAM and Terraform as well.

Open Policy Agent uses a language called Rego to define policies. You can find more information on Rego and how to write policies in the official documentation. Conftest simply provides a nice user interface to using Rego in a local testing context.

Let’s write some tests for our deployment config. Save the following as policy/base.rego:

package main

deny[msg] {
  input.kind = "Deployment"
  not input.spec.template.spec.securityContext.runAsNonRoot = true
  msg = "Containers must not run as root"

deny[msg] {
  input.kind = "Deployment"
  msg = "Containers must provide app label for pod selectors"

We’ve written two tests here:

  1. The first checks that all deployments are not set to run with root permissions
  2. The second test ensures that deployments have an app label seletor specified

Running the tests

If you check our deployment you’ll notice both of these policies are breached by our configuration. Let’s run conftest (via our CUE command):

$ cue test
  Containers must not run as root
  Containers must provide app label for pod selectors

Here we see our expected failures.

As mentioned previously, cue currently eats the exit code so although this should exit with a non-zero status it doesn’t do so currently.

Let’s go about fixing our deployment configuration:

package kubernetes

deployment <Name>: {
  apiVersion: string
  kind:       "Deployment"
  metadata name: Name
  spec: {
    replicas: 1 | int
    selector matchLabels app: Name
    template: {
      metadata labels app: Name
      spec containers: [{name: Name}]
      spec securityContext runAsNonRoot: true

deployment "hello-kubernetes": {
  apiVersion: "apps/v1"
  spec: {
    replicas: 3
    template spec containers: [{
      image: "paulbouwer/hello-kubernetes:1.5"
      ports: [{
        containerPort: 8080

With those changes we should expect the tests to pass:

$ cue test
$ echo $status


This is a simple example of the power of Open Policy Agent applied to static testing. With Open Policy Agent it’s possible to define quite powerful tests, incorporating risk scoring and more. Open Policy Agent is also intended to be used to protect and define policy within a cluster, which opens up some powerful workflows for reusing policies for local development, testing in CI and enforcement in production. The advantage of doing so is speeding up changes by making those policy violations part of the development process, rather than just part of the deployment process.

It’s the ability to reuse OPA policies in multiple contexts that I find interesting, and think that makes introducing Rego into the mix worthwhile, even if simply policies can actually be encoded in CUE itself.

Validating Cue Kubernetes Configuration with Kubeval

In the previous post I introduced using CUE for managing Kubernetes configuration. In this post we’ll start building a simple workflow.

One of the features of CUE is a declarative scripting language. This can be used to add your own commands to the cue CLI. Script files are named *_tool.cue and are automatically loaded by the CUE tooling.

First lets lay some of the ground work. This is taken from the Kubernetes tutorial and converts our map of Kubernetes objects to a list.

Save the following as kubernetes_tool.cue:

package kubernetes

objects: [ x for x in deployment  ]

We’ve saved the above in a separate file so it can be reused by other tools more easily. Next we define our validate command. For this we’re using Kubeval. Save the following as validate_tool.cue:

package kubernetes

import "encoding/yaml"

command validate: {
  task kubeval: {
    kind:   "exec"
    cmd:    "kubeval --filename \"cue\""
    stdin:  yaml.MarshalStream(objects)
    stdout: string
  task display: {
    kind: "print"
    text: task.kubeval.stdout

Commands are quite powerful, allowing for defining flags and arguments, as well as providing inline help and usage examples. The only place this is documented is in the source and I’ve not been able to get it all working quite yet, but this should be familiar if you’ve build tools using a CLI framework like Cobra in Go before.

With our command defined, what can we now do? We can run it:

$ cue validate
The document "cue" contains a valid Deployment

A few things happened here:

  1. Our deployment defined in CUE was evaluated
  2. The map data structure was flatted and converted to a list
  3. The list of objects was conerted into a multi-file YAML document
  4. That document was piped into kubeval

Their is one caveat with the above, the exit code isn’t passed through. So if Kubeval finds an error it will return a non-zero exit code. But CUE doesn’t yet support passing that along, although I have now opened an issue.

The nice thing about defining aspects of the workflow in the authoring tool is consistency and discoverability. Early adopters might like remembering a bewildering number of discreet tools but it’s nice to build up a considered user interface. CUE doesn’t support everything needed to make this happen as yet, but I did mention that it’s very new.

Future ideas

Kubeval is really a thin wrapper around validation using the Kubernetes JSON Schema. It should be possible to convert JSON Schema to valid CUE templates. That would remove the need for this step completely as evaluating the CUE definitions would catch any issues. I think this should have generic utility for formats where you already have a JSON Schema handy as well as being specifically useful for Kubernetes. I think CUE has some code in the repository looking at generating CUE templates from Go types. That sounds useful, but I think CUE has potential outside just Go (and ask me to rant about the Kubernetes Go client versus the OpenAPI definitions anytime.)

Configuring Kubernetes with CUE

My interest in Kubernetes has always been around the API. It’s the potential of a unified API and set of objects that keeps me coming back to hacking on Kubernetes and building tools around it. If I think about it, I find that appealing because I’ve also spend time automating the management of operating systems which lack anything like a good API for doing so.

I’m also a little obsessed with domain specific languages and general configuration management topics. I’ll happily talk theory, but I’m also an observer of real world practice. So while somewhere in my head I’m wondering what a configuration language built on prolog would look like, I also can’t help but appreciate people would probably still prefer to write YAML by hand.

One (very) recent new tool that piqued my interest was the new language CUE.

CUE is an open source data constraint language which aims to simplify tasks involving defining and using data. It is a superset of JSON, allowing users familiar with JSON to get started quickly.

The official documentation suggests a few usecases for CUE:

  • define a detailed validation schema for your data (manually or automatically from data)
  • reduce boilerplate in your data (manually or automatically from schema)
  • extract a schema from code
  • generate type definitions and validation code
  • merge JSON in a principled way
  • define and run declarative scripts

Before we get started, a few caveats. CUE is very new, with the first public commits in the Git repository coming in November last year. I can find zero CUE code (with a .cue extension) anywhere public on GitHub. Installation requires knowledge of Go, there are no official releases or packages. I’ve also not contributed to the project, so the following is based on a bit of experimenting rather than any intimate knowledge. CUE appears to mainly be the work of Marcel van Lohiuzen from Google and the Go team.

CUE for Kubernetes

With that out of the way, how about an example?

The documentation already has a Kubernetes tutorial which covers some of the same ground as this post, but I wanted to explain a few things differently, and to then build on the example with a few new tricks. If you find this post interesting then definitely read the official docs too.

Note as well CUE isn’t specific to Kubernetes. It can be used for authoring any structured configuration. CloudBuild or Circle CI configs, Cloud Init, CloudFormation, Azure Resource Manager templates, etc. CUE is a tool for authoring configuration, not another serialisation wire format.

Let’s start with a simple deployment configuration in YAML.

apiVersion: apps/v1
kind: Deployment
  name: hello-kubernetes
  replicas: 3
      app: hello-kubernetes
        app: hello-kubernetes
      - name: hello-kubernetes
        image: paulbouwer/hello-kubernetes:1.5
        - containerPort: 8080

CUE comes with tools to help you get started if you already have existing configs. Let’s run one here.

$ cue import deployment.yaml
$ ls
deployment.cue    deployment.yaml

cue import converted our configuration into CUE. Let’s have a look at what that looks like:

apiVersion: "apps/v1"
kind:       "Deployment"
metadata name: "hello-kubernetes"
spec: {
  replicas: 3
  selector matchLabels app: "hello-kubernetes"
  template: {
    metadata labels app: "hello-kubernetes"
    spec containers: [{
      name:  "hello-kubernetes"
      image: "paulbouwer/hello-kubernetes:1.5"
      ports: [{
        containerPort: 8080

As mentioned above, CUE is a superset of JSON. But note a few differences focused on usability:

  • No outer braces
  • No trailing commas
  • Single line format for nested statements, eg. selector matchLabels app: "hello-kubernetes"

CUE templates

YAML is just a serialisation format. To cut down on repetition when managing multiple configuration files people often apply a templating tool on top. CUE makes data templating a first class part of the language. I’ll only scratch the surface of what’s possible here but let’s see a simple example by building on the above automatically converted configuration.

package kubernetes

deployment <Name>: {
	apiVersion: string
	kind:       "Deployment"
	metadata name: Name
	spec: {
		replicas: *1 | int
		selector matchLabels app: Name
		template: {
			metadata labels app: Name
			spec containers: [{name: Name}]

deployment "hello-kubernetes": {
	apiVersion: "apps/v1"
	spec: {
		replicas: 3
		template spec containers: [{
			image: "paulbouwer/hello-kubernetes:1.5"
			ports: [{
				containerPort: 8080

Here we have created a deployment template which we can reuse, and then used it to define a concrete deployment called hello-kubernetes. A few things to note:

  • apiersion is specified as a string in the template. If it is omitted, or isn’t a string, evaluation of the CUE configuration will fail
  • replicas is specified as defaulting to 1 or taking an int value
  • The name of the deployment is placed in a variable called Name which is then used to populate the metadata and labels

In this simple case, where we have only a single deployment, this may seen a little over the top. But remember we can reuse the template for lots of deployments. That makes it easy to inject attributes into all types, or make some attributes required or limited and more. When authoring YAML the language just sees arbitrary data, with CUE you can introduce semantics, which means you can reason about your configuration in the language you’re writing.

We can evaluation our slightly more abstract configuration, and check our template is working:

$ cue eval
    deployment "hello-kubernetes": {
        apiVersion: "apps/v1"
        kind:       "Deployment"
        metadata name: "hello-kubernetes"
        spec: {
            replicas: 3
            selector matchLabels app: "hello-kubernetes"
            template: {
                metadata labels app: "hello-kubernetes"
                spec containers: [{
                    name:  "hello-kubernetes"
                    image: "paulbouwer/hello-kubernetes:1.5"
                    ports: [{
                        containerPort: 8080

Exporting to JSON

As noted, CUE is an authoring tool. It’s not intended to replace JSON or YAML or other serialisation formats directly. It supports the concept of exporting to those formats for use in other tools. Currently CUE only supports exporting to JSON, but we’ll look at some ways around that in a following post.

$ cue export
  "deployment": {
    "hello-kubernetes": {
      "apiVersion": "apps/v1",
      "kind": "Deployment",
      "metadata": {
        "name": "hello-kubernetes"
      "spec": {
        "replicas": 3,
          "selector": {
          "matchLabels": {
            "app": "hello-kubernetes"
       "template": {
         "metadata": {
           "labels": {
             "app": "hello-kubernetes"
        "spec": {
          "containers": [
              "name": "hello-kubernetes",
              "image": "paulbouwer/hello-kubernetes:1.5",
              "ports": [
                  "containerPort": 8080


I’ve shown the basics of CUE here using a simple Kubernetes configuration file. The full CUE tutorial covers more of the language features too.

For me CUE appears a powerful mix of data-centric authoring with built-in validation and templating. It’s pleasant enough even for single examples but it’s features are particularly powerful when managing large amounts of configuration, where introducing powerful abstractions can drastically cut down the amount of configuration needing to be managed.

Next I’ll look at integrating other tools with CUE, to build a workflow supporting further testing and validation.

About is the new blog of Gareth Rushgrove. I’m a professional technologist, mainly specialising in instrastructure, automation and information security.

I’m currently Director of Product Management at Snyk, working on application security tools for developers.

I have previously worked as:

  • A group Product Manager for Docker, responsible for a range of the developer-facing tools like Docker Desktop, and work on the Cloud Native Application Bundles (CNAB) specification.
  • A Principal Engineer at Puppet, working on configuration management tooling for cloud infrastructure, Kubernetes and various source code analytics projects.
  • A Technical Architect for the UK Cabinet Office, working as one of the early members of what became the Government Digital Service, on building the infrastructure that underpins GO.UK, and assisting various government departments with devops and digital transformation

You can generally find me somewhere on the internet if you need to do so:

This blog is managed with the following tools:

  • Hugo for generating the site
  • Netlify for serving up the content
  • GitHub for storing the source