The Yambler

This is a simple yaml stitcher program, which operates at the YAML event level, rather than the character stream level. This has the advantage that all input files are themselves valid YAML, making it easy to edit and understand them.

Run like this:

yambler -i <input file> -o <output file> -s <snippet files ...>

Or like this:

yambler -i <input dir> -o <output dir> -s <snippet dir>

This replaces placeholder strings in the input file(s) with YAML objects defined in the snippet files, writing the resultant document to the output file(s).

Getting Started

Install the Latest release for your platform, and start yambling some yamls.

For GitHub Actions

You've got to know when to code them,
know when to load them,
know when to "uses" tag,
and know when to "run".

It's difficult to reuse common logic in the various workflows that you build for GitHub Actions. You're either stuck publishing custom actions (which themselves are limited in what they can reuse), hacking some shell scripts together, or doing some big copy-and-paste and hoping you remember where all the copies are when you need to make a change.

The Yambler was written to deal with this problem: it's not an ideal solution (GitHub is working on more elegant solutions), but this lets you at least keep your workflows relatively DRY.

Example

The Yambler is used in the CI/CD pipelines of the Versio release manager, another handy developer tool. My .github directory there looks something like this:

.github
├─ workflows-src
│  ├─ pr.yml
│  └─ release.yml
├─ snippets
│  ├─ check-versio.yml
│  ├─ common-env.yml
│  ├─ job-premerge-checks.yml
│  └─ <other snippet files ...>
└─ workflows
   ├─ pr.yml
   └─ release.yml

I don't touch anything in workflows directly: everything there is generated. Instead, workflows-src is where I do my top-level editing. For example, .github/workflows-src/pr.yml looks something like this:

---
name: pr
on:
  - pull_request
env: SNIPPET_common-env

jobs:
  create-matrixes: SNIPPET_job-create-matrixes
  premerge-checks: SNIPPET_job-premerge-checks

I then keep my snippets, one per file, in .github/snippets. Here's common-env.yml:

key: common-env
value:
  RUSTFLAGS: '-D warnings'
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  GITHUB_USER: ${{ github.actor }}

Before I push my repo, I generate the actual workflows by calling yambler:

yambler \
    -i .github/workflows-src \
    -o .github/workflows \
    -s .github/snippets

Of course there's a script to do this automatically. There's also a companion script to check that your workflows are up-to-date; copy it to a file named .git/hooks/pre-push in your local repo and never push out-of-date workflows again.

You'd normally want to .gitignore generated files, but you can't with workflow files, or you defeat their whole purpose. I'm sure it's theoretically possible to keep these up-to-date automatically, but I find it easier just to yamble + commit them manually whenever I change the inputs or snippets.

Operation

Options

The --snippet (-s) argument can be a list of files, or a single directory with a bunch of YAML files in it. Likewise, the --input (-i) argument can either be a single file, or a directory that contains a bunch of YAML input files. If the input is a directory, the output (--output / -o) must also be a directory, in which case it generates a single output file for each input file.

Algorithm

This is how the Yambler works: First, all documents of each input file are read, and wherever a placeholder string of the form "SNIPPET_<snippet name>" is encountered, it is replaced by the YAML snippet value defined in the snippet files. This process happens recursively, so snippets can contain other snippets, etc. Infinite loops are detected at runtime and cause the program to terminate without writing anything. After all placeholders are replaced, the resulting YAML is written to the output file.

Because processing is done at the YAML level, the generated output doesn't respect the formatting or style decisions of the input files, preferring the style of the Yambler's own internal YAML emitter. Any comments in the inputs could be discarded; and bare, block or continue-style strings could be replaced by simple quotes, etc. This is usually not a problem, because you rarely want to look at generated files anyway, and the generated output is semantically identical (with respect to the YAML specification).

Using Yambler is roughly analogous to using a macro language such as C/C++ macros, VBA, or ML/1; with many of the same benefits and pitfalls. There is no stacked parameter passing or templating: snippets are basically inserted verbatim into the text, so keep that in mind. On the other hand, this makes it very easy to judge what your final output is going to be.

Snippets

A snippet file can have multiple YAML documents, and each is considered its own snippet: each snippet must be a hash with at least the two keys "key" and "value". (You can have other keys: they're just ignored.) The "key" must be a string that defines the snippet key (which is identified in the placeholder string); the "value" is the YAML value itself, which can have any YAML type.

The names of the snippet files are largely irrelevant, but it's good practice to have at least some association between the file name and the snippets contained within, so that it's easy to quickly find a particular snippet.

If multiple snippets are defined with the same key, the behavior is undefined, although what probably happens is that the last defined snippet is the one that "wins" that key. Don't do this!

Splicing

One exception to the replacement described above is the splice rule: if the placeholder string is a direct array element, and the replacement snippet is also an array, then the snippet array is spliced in directly, rather than replacing the single element. This makes it easy to place a snippet directly inside a array, or to concatenate multiple snippets to form a longer list.

Examples

• Simple string replacement

  Input:

  first_name: "John"
  last_name: SNIPPET_family
  

  Snippet:

  key: family
  value: Smith
  

  Output:

  first_name: John
  last_name: Smith
  

• Object replacement

  Input:

  job_1: SNIPPET_job1
  

  Snippet:

  key: job1
  value:
    name: 'complex job'
    run: |
      Something something
      is strange
  

  Output:

  job_1:
    name: complex job
    run: |
      Something something
      is strange
  

• Splicing

  Input:

  steps:
    - SNIPPET_setup
    - run: echo custom
    - SNIPPET_teardown
  

  Snippet:

  ---
  key: setup
  value:
    - run: curl http://setmeup.com/now
    - name: finish setup
      use: my-actions/finish-setup@v1
  ---
  key: teardown
  value:
    - name: start teardown
      use: my-actions/start-teardown@v1
    - run: curl http://tearmedown.com/now
  

  Output:

  steps:
    - run: "curl http://setmeup.com/now"
    - name: finish setup
      use: my-actions/finish-setup@v1
    - run: echo custom
    - name: start teardown
      use: my-actions/start-teardown@v1
    - run: "curl http://tearmedown.com/now"