feat(arc): keep a profile's work volume in memory, and refuse measured profiles that share nodes

roles/github_runner_arc/README.md

@@ -18,14 +18,15 @@ Before touching the cluster the role then checks the secrets it is about to writ
   - `suffix`: appended to the namespace (`arc-runners-<org><suffix>`) and release (`<org>-runners<suffix>`) names. Default empty.
   - `namespace`, `release_name`: the profile's namespace and Helm release name, in place of the names derived from the suffix. The release name is also the scale set's name on GitHub, so two profiles of one scope (an organisation, or one repository) cannot share it, and Helm accepts at most 53 characters.
   - `values_file`: Helm values for the release, a Jinja template read from `github_runner_arc_values_dir` on the control node (or an absolute path). Without it the role's `templates/runner-scale-set-values.yaml.j2` is used, driven by the profile's own `max_runners`, `min_runners`, `container_mode` (`dind` for the chart's Docker-in-Docker mode) and `resources` (the runner container's requests and limits).
-  - `node_selector`: a nodeSelector for the runner pods, merged over the eligibility label below.
+  - `node_selector`: a nodeSelector for the runner pods, merged over the eligibility label below. Measured sizing counts only the nodes carrying it, so a profile pinned to some of the nodes is sized from those alone; two profiles whose ceilings are measured may not count the same node (see [Profiles on different nodes](#profiles-on-different-nodes)).
   - `max_runners`: the release's `maxRunners`. It takes precedence over the values file and over `sizing`; on the autoscaled profile it is the floor the autoscaler starts from.
   - `runs_on_label`: the extra `scaleSetLabels` entry the runners carry. Default `<org>-runners`, which pools every profile of the org under one `runs-on` label; set a different one to keep a profile out of that pool.
   - `scale_set_labels`: the whole `scaleSetLabels` list, in place of `runs_on_label`. An empty list sets no `scaleSetLabels`, so jobs target the scale set by its release name.
   - `tolerations`: Kubernetes tolerations for the profile's runner pods, for nodes that carry both the eligibility label and a `NoSchedule` or `NoExecute` taint that keeps other workloads off them, such as machines the controller and listener must not run on. Measured sizing counts a node whose only repelling taints these tolerations match; without them it leaves a tainted node out, because it takes runner pods to tolerate nothing. A taint Kubernetes adds for a passing condition (not ready, unreachable, cordoned) still counts as before. With `burst` they come before `burst.tolerations`.
   - `autoscale`: `true` makes the profile's scale set one of those the autoscaler manages, pooled against one memory budget; at most one autoscaled profile in a cluster may set `sizing`.
   - `sizing`: derive a runner ceiling from node capacity, given as uniform figures or measured from each eligible node (see Sizing). On an ordinary profile without `max_runners` it sets `maxRunners`; on the autoscaled profile it bounds the ceiling the autoscaler works out from the nodes, leaving `maxRunners` as the floor.
   - `capabilities`: names of entries in `github_runner_arc_capabilities` that the profile's runner pods carry, in order (see [Capabilities](#capabilities)).
+  - `work_volume_medium`, `work_volume_size_limit`: what the runner's work volume sits on: unset or `""` for the node's disk, or `Memory` for a tmpfs of `work_volume_size_limit` (a Kubernetes quantity such as `6Gi`, required with `Memory`), which counts against the pod's memory (see [The work volume in memory](#the-work-volume-in-memory)).
   - `capability_store`: `true` copies the capabilities' tool-cache and system-library payloads into the capability store, which the pods of a node share, instead of into every pod, and does the same for the dind sidecar's copy of the runner's externals when the profile's values write that sidecar out with an `init-dind-externals` init container (a values file, or the registry cache's wiring), but not when the chart builds the sidecar itself from `containerMode: dind` (see [The capability store](#the-capability-store)). Default `false`.
   - `burst`: let the runner pods overflow onto nodes a cluster autoscaler adds on demand, and count those nodes' runners into `maxRunners` (see [Burst nodes](#burst-nodes)). Needs `sizing`, and no `max_runners` or `autoscale`; with `exclusive: true` the pods run on burst nodes only and `burst.max_runners` is the whole ceiling, with no `sizing`.
 - `github_runner_arc_values_dir`: the control-node directory relative `values_file` paths are read from.
@@ -95,6 +96,29 @@ A profile's own `max_runners` still takes precedence, and on the autoscaled prof
 
 The filter behind it is `exadev.github_runner.arc_max_runners`.
 
+### Profiles on different nodes
+
+A measured ceiling counts the whole of the room on every node its profile's runner pods may use, leaving every scale set's runner pods out of what it subtracts. Two measured profiles that may both use a node would each count all of it, and together let more runner pods start than it holds, so the role refuses that before it installs anything (the `exadev.github_runner.arc_measured_overlap_errors` filter), naming the profiles and the nodes they share. How one node's room should be divided between scale sets whose pods differ in size is a choice the cluster's figures do not make, so the role does not guess one. Give each measured profile its own nodes: a `node_selector` for the label of the nodes it is pinned to, and, for the others, nodes it cannot use, for example by a `NoSchedule` taint on the pinned nodes that only the pinned profile's `tolerations` match. A profile with its own `max_runners` is not measured and not compared, which is how a fleet divides one set of nodes between scale sets on purpose. Every profile in the cluster is compared, including those another host installs.
+
+The autoscaled profiles share one pool, whose ceiling the autoscaler works out from the nodes every pooled scale set's runner pods can be placed on, so they must set the same `node_selector` and `tolerations`; validation refuses them otherwise.
+
+## The work volume in memory
+
+The runner keeps every job's checkout, dependency installs and cache archives (`actions/cache` writes them under the runner's temporary directory, inside the work directory) in its work directory, `/home/runner/_work`. On a node whose disk sustains far less write throughput than its memory could absorb, those writes can stall every task on the node on I/O while its CPU and memory sit idle. `work_volume_medium: Memory` puts the work directory on a tmpfs of `work_volume_size_limit` instead (an `emptyDir` with `medium: Memory` and `sizeLimit`), so a job writes nothing there to the disk.
+
+```yaml
+work_volume_medium: Memory
+work_volume_size_limit: 6Gi
+```
+
+The values' own volume named `work` is changed in place, and must then be an `emptyDir` that sets neither `medium` nor `sizeLimit`. Without one, the chart's Docker-in-Docker mode is given one (the chart keeps a values volume of that name in place of its own), and otherwise the volume is added and mounted on the runner container at the work directory. The `kubernetes` container modes run a job's steps in a separate pod whose work volume the chart claims, so the role refuses the option there. Unset, the values are left exactly as they were.
+
+The tmpfs's pages are memory. They are charged to the container that writes them (the runner, and a Docker-in-Docker sidecar that mounts the work directory for the containers it runs), and they cannot be reclaimed while the files exist, so a profile that sets the option raises its runner container's memory request and limit by `work_volume_size_limit` in its values, and its sizing's `pod_memory_request_gib` by the same, or the kubelet evicts its pods when the node runs short and measured sizing places more pods than the memory holds. With the higher request, measured sizing derives fewer pods per node by memory, which is the point: the option suits nodes whose disk, not their memory, bounds how many jobs they serve, pinned with `node_selector` (see [Profiles on different nodes](#profiles-on-different-nodes)) so that the memory is reserved only where the disk is the bound. A write past the limit fails with no space left on the device, as on a full disk.
+
+Size it from running pods, sampled while the profile's real jobs run: the largest work directory seen (`du -sm /home/runner/_work` in the runner container, or `du -sm` of the `work` volume under `/var/lib/kubelet/pods/<pod uid>/volumes/kubernetes.io~empty-dir/` on the node), plus the largest cache archive a job restores or saves beside it, rounded up.
+
+Only the work volume is offered. The tool cache and the sysroot hold the capabilities' copies, which the capability store already writes once per node rather than once per pod (see [The capability store](#the-capability-store)), and in memory they would be reserved in every pod whether a job used them or not. The Docker daemon's data root is large only in jobs that build images, so reserving it in every pod's memory would charge the size of a build to every job; a values file whose profile only builds images can mount its own `emptyDir` with `medium: Memory` at the dind container's `/var/lib/docker`.
+
 ## Burst nodes
 
 A fixed pool can overflow onto nodes that exist only while there is work for them, such as cloud instances a cluster autoscaler (Kubernetes' Cluster Autoscaler, for one) launches when pods are Pending and removes once they are idle. Such nodes usually carry a label and a matching `NoSchedule` taint so nothing else lands on them, and no eligibility label, so measured sizing never counts them. A profile's `burst` lets its runner pods onto them:

roles/github_runner_arc/defaults/main.yml

@@ -1,7 +1,7 @@
 ---
 # ARC (Actions Runner Controller) in an existing Kubernetes cluster: the controller, each org's runner scale sets, and the fleet-health platform (heartbeat and autoscaler). Each part is gated on this host's own github_runner_arc_orgs or github_runner_arc_heartbeat_gist_id, so any host that can reach the cluster's API can be the one Ansible installs from: a k3s server host brought up by github_runner_cluster, or localhost with a kubeconfig (playbooks/arc.yml). On a host github_runner_cluster made an agent, this role installs nothing.
 
-# Orgs whose runner scale sets this host installs. Each entry: name, app_id (needed only when the role writes the App Secret; otherwise read from the existing Secret, and checked against it when set), image, private_key (the App's PEM private key, from any source Ansible reads) or private_key_op_reference (an op:// reference the 1Password adapter resolves into private_key), optionally installation_id (resolved from the App when empty), optionally app_secret_name (the App Secret's name in each of the org's namespaces, default <org>-github-app), and scale_set_profiles, a list of {suffix, namespace?, release_name?, values_file?, max_runners?, node_selector?, runs_on_label?, scale_set_labels?, capabilities?, autoscale?, sizing?, burst?}. capabilities names entries of github_runner_arc_capabilities (below) that the profile's runner pods carry. sizing takes uniform node figures, or measured: true to read each eligible node's free capacity from the cluster at install time. burst ({node_label_key, node_label_values?, tolerations?, max_runners, exclusive?}) lets a sized profile's runner pods overflow onto nodes carrying node_label_key, which a cluster autoscaler adds when pods are Pending, and adds max_runners to the sized maxRunners; with exclusive: true the pods run on burst nodes only and max_runners is the whole maxRunners. Each profile becomes one namespace (namespace, default arc-runners-<org><suffix>), two Secrets in it, and one Helm release (release_name, default <org>-runners<suffix>); setting both overrides lets the role take over scale sets an existing install created under its own names. See the README for every profile key.
+# Orgs whose runner scale sets this host installs. Each entry: name, app_id (needed only when the role writes the App Secret; otherwise read from the existing Secret, and checked against it when set), image, private_key (the App's PEM private key, from any source Ansible reads) or private_key_op_reference (an op:// reference the 1Password adapter resolves into private_key), optionally installation_id (resolved from the App when empty), optionally app_secret_name (the App Secret's name in each of the org's namespaces, default <org>-github-app), and scale_set_profiles, a list of {suffix, namespace?, release_name?, values_file?, max_runners?, node_selector?, runs_on_label?, scale_set_labels?, capabilities?, work_volume_medium?, work_volume_size_limit?, autoscale?, sizing?, burst?}. work_volume_medium: Memory puts the runner's work volume on a tmpfs of work_volume_size_limit, which counts against the pod's memory. capabilities names entries of github_runner_arc_capabilities (below) that the profile's runner pods carry. sizing takes uniform node figures, or measured: true to read each eligible node's free capacity from the cluster at install time. burst ({node_label_key, node_label_values?, tolerations?, max_runners, exclusive?}) lets a sized profile's runner pods overflow onto nodes carrying node_label_key, which a cluster autoscaler adds when pods are Pending, and adds max_runners to the sized maxRunners; with exclusive: true the pods run on burst nodes only and max_runners is the whole maxRunners. Each profile becomes one namespace (namespace, default arc-runners-<org><suffix>), two Secrets in it, and one Helm release (release_name, default <org>-runners<suffix>); setting both overrides lets the role take over scale sets an existing install created under its own names. See the README for every profile key.
 github_runner_arc_orgs: []
 
 # Directory on the control node that a relative scale_set_profiles[].values_file is read from. Values files are Jinja templates rendered on the control node, so the target host needs no checkout. A profile with no values_file uses the role's own templates/runner-scale-set-values.yaml.j2.

roles/github_runner_arc/meta/argument_specs.yml

@@ -13,13 +13,15 @@ argument_specs:
         description:
           - Scopes whose runner scale sets this host installs. Each entry has name, app_id (needed only when the role writes the App Secret; otherwise the existing Secret supplies it), image, private_key or private_key_op_reference, and optionally repository, installation_id and app_secret_name (the App Secret's name, default <org>-github-app).
           - Without repository an entry registers organisation runners in the organisation name names (githubConfigUrl https://github.com/<name>). With repository it registers repository runners in that one repository (https://github.com/<name>/<repository>), where name may be a user or an organisation; its derived names use <name>--<repository> (lower-cased, with any character a DNS label does not allow as '-') in place of <org>, and its App needs the repository-scope permissions. The scope (name, or name/repository) identifies an entry, so one login may have an organisation entry and any number of repository entries.
-          - Each entry's scale_set_profiles is a non-empty list of profiles, each with suffix and optionally namespace (default arc-runners-<org><suffix>), release_name (default <org>-runners<suffix>, at most 53 characters, Helm's limit), values_file, max_runners, min_runners, container_mode, resources, node_selector, runs_on_label, scale_set_labels, capabilities, capability_store, tolerations, autoscale, sizing and burst.
+          - Each entry's scale_set_profiles is a non-empty list of profiles, each with suffix and optionally namespace (default arc-runners-<org><suffix>), release_name (default <org>-runners<suffix>, at most 53 characters, Helm's limit), values_file, max_runners, min_runners, container_mode, resources, node_selector, runs_on_label, scale_set_labels, capabilities, capability_store, work_volume_medium, work_volume_size_limit, tolerations, autoscale, sizing and burst.
           - capability_store, a boolean (default false), makes the profile's runner pods copy each tool-cache and system-library capability's payload, and the runner's externals for a Docker-in-Docker sidecar the values write out, into the capability store (github_runner_arc_capability_store_host_path), which the pods of a node share, instead of into every pod's own volumes, and mount the result read-only. Every image those copies come from must be pinned by digest, and the store needs the node cache pruner, so a profile that sets it is refused while github_runner_arc_node_cache_size_limit_gib is 0 or github_runner_arc_node_label_key is empty, since the pruner runs only on the nodes carrying that label. A job that can reach a privileged Docker daemon can write to the store, so give it only to a profile whose workflows are all of one trust level.
           - max_runners, when set, is the release's maxRunners and takes precedence over its values file and its sizing.
           - scale_set_labels replaces runs_on_label with the whole list of scaleSetLabels; an empty list sets none, so jobs target the release name.
           - tolerations is a list of Kubernetes tolerations for the profile's runner pods, for nodes the eligibility label selects that also carry a taint to keep other workloads off. Measured sizing counts a node whose only repelling taints these match, as the pods can be placed there; without them it leaves such a node out. With burst they come before burst.tolerations.
           - sizing derives a runner ceiling from uniform node figures, or with measured true from each eligible node's allocatable CPU and memory (and pod capacity, the kubelet's max-pods) less its current requests and reserve_cpu and reserve_memory_gib.
           - capabilities names entries of github_runner_arc_capabilities that the profile's runner pods carry, in order.
+          - work_volume_medium, '' (the default, the node's disk) or Memory, is what the runner's work volume sits on. Memory makes it a tmpfs of work_volume_size_limit (a Kubernetes quantity, required with Memory and refused without it), whose pages count against the pod's memory, so the profile's runner memory request and limit, and its sizing's pod_memory_request_gib, must be raised by that size. The kubernetes container modes are refused with it.
+          - node_selector restricts the profile's runner pods, and the nodes its measured sizing counts, to nodes carrying those labels as well as the eligibility label. Two profiles whose ceilings the role measures may not count the same node (each would count all of its room), and autoscaled profiles must share one node_selector and one set of tolerations, since their pool's ceiling counts only the nodes all of them can use.
           - burst, on a profile with sizing and without max_runners or autoscale, lets its runner pods overflow onto nodes carrying burst.node_label_key (any value, or one of burst.node_label_values), with burst.tolerations for their taint, and adds burst.max_runners to the sized maxRunners. With burst.exclusive true (and no sizing, max_runners or autoscale) the pods run on burst nodes only and burst.max_runners is the whole maxRunners.
       github_runner_arc_capabilities:
         type: dict