Skip to content

Simulate a run of a CI system using local containers

[Home | sourcehut | crates.io]

Overview

The sim-ci utility starts a local container, installs packages within it, and then runs some commands as a "real" CI system would. Or at least that is the plan.

Limitations

The sim-ci utility currently has the following limitations:

  • only Alpine Linux is fully supported; on that, apk is used for configuring additional repositories and installing packages
  • Debian GNU/Linux and Ubuntu are partially supported; additional repositories may not be configured yet
  • only the current directory is used, no other Git repositories are checked out

Building sim-ci

With a recent enough Rust build toolchain, cargo build --release should be enough to build a target/release/sim-ci executable file. You may then copy that file to a location of your choice and run it.

Configuring sim-ci

Configuration files

The sim-ci tool follows the drop-in search pattern for configuration files: - several directories are searched for *.toml files: - /usr/local/share/sim-ci/config.d/ - /usr/share/sim-ci/config.d/ - /etc/xdg/sim-ci/config.d/ - ~/.config/sim-ci/config.d/ - files found in later directories are processed instead of files with the same name found in earlier directories - a symlink to /dev/null means files with the same name in earlier directories will not be considered at all - after determining which files will be considered at all, they are processed in alphabetical order of their file names (disregarding the directory part)

So given the following set of files: - /usr/share/sim-ci/config.d/host.toml - /etc/xdg/sim-ci/config.d/host.toml - /etc/xdg/sim-ci/config.d/localhost.toml - ~/.config/sim-si/config.d/00-init.toml - ~/.config/sim-ci/config.d/localhost.toml -> /dev/null

The sim-ci utility will process only two files in this order: - ~/.config/sim-si/config.d/00-init.toml - /etc/xdg/sim-ci/config.d/host.toml

The reasoning for that is as follows: - /etc/xdg/sim-ci/config.d/host.toml overrides /usr/share/sim-ci/config.d/host.toml - ~/.config/sim-ci/config.d/localhost.toml overrides /etc/xdg/sim-ci/config.d/localhost.toml and, since the former is a symlink to /dev/null, it will not be processed at all - finally the file names are sorted, and 00-init.toml comes before host.toml

The mediaType key - format version

Each sim-ci section (whether top-level in the system and per-user files, or within the tool section in the project files) must have a mediaType key. Its value specifies the version of the format of the configuration file itself.

For the present, mediaType must be set to this exact value:

[sim-ci]
mediaType = "vnd.ringlet.sysutils.sim-ci.config/file.v0.1+toml"

As more configuration file values are added, removed, or their type is very rarely changed, the 0.1 version may evolve.

Secrets for the sourcehut CI

Currently only plain text files are supported as secrets. They may either be configured globally, or in a per-project "inject" section. We should really describe the per-project injection mechanism in more detail.

The secrets may be defined in files in the user's home directory (in a .config/sim-ci/config.d/*.toml file) so that they do not have to be kept under version control with the projects themselves.

Configuring a secret globally

[sim-ci.secrets."unguessable-uuid"]
name = "frob-the-duckie-api-key"
contents = { line = "frob the duckie secret phrase" }
file = { path = ".config/secrets/frob-the-duckie.txt", mode = "0600" }

Configuring a secret so that sim-ci only shows it to a specific project

This method makes sure that the secret will not be mistakenly used by a different project. There is currently no way to reference the same secret in two projects.

[sim-ci.inject.project."sim-ci-test-project".secrets."unguessable-uuid"]
name = "neverland-ssh-key"
contents = { whole = """
This is an SSH key file.
It has a very specific format.
Right?
""" }
file = { path = ".ssh/id-neverland", mode = "0600" }

Build manifests for the sourcehut CI

The only configuration settings supported so far are include and exclude glob patterns for sourcehut CI build manifests. By default the sim-ci utility will attempt to parse and run all tasks defined in all files matching the .builds/*.yml pattern.

Each sourcehut.files section must contain a single include or exclude item - an array of strings representing glob patterns relative to the project's directory. More than one sourcehut.files section may be present in a single configuration file, or across several configuration files; they will be processed in order, and any files included or excluded in later sections will override settings from earlier sections.

Defining exclusions in a per-user configuration file

If the .config/sim-ci/config.d/ directory contains any *.toml files, they will be expected to contain a sim-ci section:

[sim-ci]
mediaType = "vnd.ringlet.sysutils.sim-ci.config/file.v0.1+toml"

[[sim-ci.sourcehut.files]]
exclude = [
  ".builds/*-webhook.yml",
]

Defining exclusions in a per-project pyproject.toml file

If a file named pyproject.toml in the project's directory contains a tool.sim-ci section, it will be honored:

[tool.sim-ci]
mediaType = "vnd.ringlet.sysutils.sim-ci.config/file.v0.1+toml"

[[tool.sim-ci.sourcehut.files]]
exclude = [
  ".builds/*-webhook.yml",
]

Using sim-ci

List the configuration settings

sim-ci show config parsed

Simulate a sourcehut CI run

sim-ci run sourcehut

Simulate a sourcehut CI run for one or more manifests only

sim-ci run sourcehut .builds/this.yml .builds/that.yml

Full configuration file example

System or per-user file:

[sim-ci]
# The mandatory format version field
mediaType = "vnd.ringlet.sysutils.sim-ci.config/file.v0.1+toml"

# Make the `.builds/*.toml` default more precise
[[sim-ci.sourcehut.files]]
exclude = [
  ".builds/*-webhook.yml",
  ".builds/deploy-*.yml",
]

# Add back some files that were excluded by the previous one
[[sim-ci.sourcehut.files]]
include = [
  ".builds/deploy-simulate.yml",
]

# A secret that will be visible to all `sim-ci` invocations in any directory
[sim-ci.secrets."42-616"]
name = "global-secret"

[sim-ci.secrets."42-616".contents]
whole = """
This is a file.
It is nothing but a file.
"""

[sim-ci.secrets."42-616".file]
path = ".config/global/secret.txt"
mode = "0600"

# A secret that will only be visible to a project that either:
# - resides in a directory called `sim-ci-sample-project/`, or
# - has a `tool.sim-ci.project.name = "sim-ci-sample-project"` setting
[sim-ci.inject.project."sim-ci-sample-project".secrets."7-11"]
name = "only-ours"
contents = { line = "A newline character will be added at the end of the file" }
file = { path = "not-really-a-secret.txt", mode = "0644" }

A file named pyproject.toml within the source directory where sim-ci is run:

[tool.sim-ci]
# The section is now `tool.sim-ci`, not just `sim-ci`
# The mandatory format version field
mediaType = "vnd.ringlet.sysutils.sim-ci.config/file.v0.1+toml"

[[tool.sim-ci.sourcehut.files]]
# Exclude a specific manifest that cannot be run locally for any reason
# (maybe it contains features that `sim-ci` does not support yet)
exclude = [
  ".builds/complicated.yml",
]

[tool.sim-ci.project]
# Override the default "base name of the directory" project name
name = "sim-ci-sample-project"

Contact

The sim-ci utility was written by Peter Pentchev. It is developed in a sourcehut repository. This documentation is hosted at Ringlet.