turboinstall

⚠️ Warning: This tool is not fully finished and there are some bugs.

A quick and simple tool that overlays directory trees.

Table of contents

• turboinstall
  • Table of contents
  • What does this mean?
  • Who even needs this?
  • Features
    • Platform specific
      • Unix
  • Installation
  • Usage
    • The ignore file
    • Profiles and path expansion
      • Example profiles
        • JSON
        • TOML
        • YAML
        • ENV
    • Hooks
      • Hook environment
      • Pre-install
      • Post-install

What does this mean?

It means you can effortlessly and easily install files to the right places without writing any custom install scripts. Just replicate the structure you need inside your source tree and everything else will be handled by the tool.

Who even needs this?

Ever needed to create some sort of directory layering for packaging applications? In reality this tool was made to serve a very specific need: the runtime system for my zeus project and more specifically how the packaging for that works.

If you do decide to try out this tool, please be aware that there probably are many bugs (especially in path traversal), use it with care.

Features

• 🌲 Overlay multiple sources trees on top of each other
• ✂ In-path variable expansion (basically path substitution)
• 🪪 4 different profile formats (json, toml, yaml, env)
• 🪝 Hooks for custom actions
• 🌈 Pretty colors
• 📏 Ability to define regex rules to ignore paths (like .gitignore)
• 🔒 Preserve file permissions
• 🐚 Shell completions

Platform specific

Unix

• ⏰ Preserve ownership & timestamps of files
• 🐮 Make CoW filesystem copies (requires support from the filesystem (btrfs, xfs, ...))

Installation

If you are the kind of person who needs this, then there is a high chance that you have rust and cargo installed. In that case:

cargo install turboinstall

Usage

A simple tool for overlaying directory trees on top of each other

Usage: turboinstall [OPTIONS] <dir> [dir]...

Arguments:
  <dir>     Destination directory
  [dir]...  Overlay source(s)

Options:
  -p, --profile </path/to/profile>  Path to the file with the profile definition [default: .turboinstall.json]
  -f, --format <fmt>                Specify which format the profile uses [possible values: json, toml, yaml, env]
  -l, --link                        Hard link files instead of copying
  -n, --no-clobber                  Do not overwrite existing files
  -u, --update                      Overwrite only when the source path is newer
  -q, --quiet                       Don't print anything to the console
      --ignore <path,path,...>      Paths to extra ignore files
      --no-abort                    Don't exit on error
      --dry-run                     Do not perform any filesystem operations (implies --no-hooks)
      --no-hooks                    Do not run any hooks
      --hooks <type,type,...>       Only run these types of hooks [possible values: pre-install, post-install]
      --porcelain                   Use machine readable output
      --preserve <attr,attr,...>    Preserve the specified attributes [possible values: ownership, timestamps]
      --reflink <when>              Create clone/CoW copies [default: auto] [possible values: never, always, auto]
  -h, --help                        Print help information
  -V, --version                     Print version information```

In short this tool does 1 thing. It takes a directory tree (src) like this:

 src/
├──  dir1/
│   ├──  dir2/
│   │   └──  file2
│   └──  file1
└──  file0

And it copies it to another directory (dst) like this:

 dst/
├──  dir1/
│   ├──  dir2/
│   │   └──  file2
│   └──  file1
└──  file0

The command to achieve this is:

turboinstall ./dst ./src

The ignore file

The ignore file is a simple text file at .turboinstall/ignore that contains everyone's favorite regular expressions 🎉. Each line of the file contains a regex pattern that will be matched on each path of the overlay. In other words, just like .gitignore files. Other ignore files can be specified on the command line with --ignore, relative paths will be resolved from the overlay root, while absolute paths will resolve normally.

Let's suppose we have a source tree:

 src/
├──  .turboinstall/
│   └──  ignore
├──  dir0/
│   ├──  dir1/
│   │   ├──  file1
│   │   └──  file2
│   └──  file0
└──  file0

and src/.turboinstall/ignore contains:

# This is a comment
# Empty lines are also ignored

/file0

This would mean that when we run turboinstall ./dst ./src we would get:

 dst/
└──  dir0/
    └──  dir1/
        ├──  file1
        └──  file2

Notice how both src/file0 and src/dir0/file0 are missing. This is because unlike gitignore files, these files match with pure regex on paths. The pattern /file0 from the ignore file matches both:

• /file0
• /dir0/file0

Ok, so how can we only match /file0? This is very simple as long as you know basic regex. Just prepend the pattern with ^, which means: only match the following if it is at the start of the path, so our ignore file becomes:

# This is a comment
# Empty lines are also ignored

^/file0

In this example the paths that will be tested are:

• /dir0
• /dir0/dir1
• /dir0/dir1/file1
• /dir0/dir1/file2
• /dir0/file0
• /file0

NOTE: Anything inside the /.turboinstall folder is always automatically ignored, there is no way to change this.

Profiles and path expansion

The profile is a fancy way of saying configuration file or variable store. It is a file in one of the supported formats (see Features) that holds the variables for the path expansion.

NOTE: Path expansion is not fully completed but it is functional

Profiles are only used for path expansion and nothing else, they are not needed if don't plan to use this feature at all.

The following examples act in the same way, they are just expressed in different formats. This makes the tool easy to integrate with other custom tooling. The env format is especially useful because you can source it directly from shell scripts.

You can specify a custom profile with -p.

The following profiles can be used to do path expansion. So instead of the previous example, it would turn this:

 src/
├──  file
└──  {DIR}/
    ├──  test/
    │   └──  file
    └──  file_{VARIABLE_1}

Into this:

 dst/
├──  file
└──  usr/
    └──  local/
        ├──  test/
        │   └──  file
        └──  file_VALUE_1

This way you can create different outputs from one easily understood source tree by just specifying another profile on the command like. For example, this could allow you to build packages for, let's say, different systems with different filesystem hierarchies from one source tree and a bunch of configuration files.

No more pesky install scripts.

The command to do this is:

turboinstall ./dst ./src -p example_profile.json

Where example_profile.json is the file with one of the example profiles bellow. It does not need to be named like that, just a normal file with the corresponding extension, otherwise you will need to specify the format with -f.

Example profiles

JSON

{
  "VARIABLE_1": "VALUE_1",
  "DIR": "/usr/local"
}

TOML

VARIABLE_1 = "VALUE_1"
DIR = "/usr/local"

YAML

VARIABLE_1: "VALUE_1"
DIR: "/usr/local"

ENV

# This is a comment

VARIABLE_1=VALUE_1
DIR="/usr/local"

Hooks

Hooks are just executables placed in a special location that are executed in wildcard order (alphanumerical) with 2 arguments:

1. The source tree
2. The destination tree

The special location for these files is inside the root of the source tree in a folder called .turboinstall, in other words this is how your source tree should look like:

 src/
├──  .turboinstall/
│   ├──  post-install/
│   │   └──  some_hook.sh
│   └──  pre-install/
│       ├──  00-hook.sh
│       └──  10-another_hook.sh

The hooks are executed in the following order:

pre-install hooks:

1. 00-hook.sh
2. 10-another_hook.sh

post-install hooks:

1. some_hook.sh

It is not strictly necessary to follow the naming convention shown here in the pre-install hooks, but it gives a clear indication of the order the hooks are executed in.

Hook environment

The hooks are invoked with 2 arguments:

1. The path of the source tree they reside in
2. The path of the destination tree

Their working directory is left untouched and is the same as the working directory where turboinstall was ran. This allows the hooks to access any other files that might be relevant and are not present in the source tree.

Pre-install

The executables inside .turboinstall/pre-install, like the name suggests are ran before any of the actual source tree has been copied.

Post-install

The executables inside .turboinstall/post-install, like the name suggests are ran after the source tree has been copied.