Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- guest: a fresh CVM with a verity volume could take the volume for its data disk and try to LUKS-format it: volumes labelled their data partition `dstack-data`, the data disk's label. `dstack verity` now labels it `dstack-volume-data`, and the guest skips disks that carry a `dstack-volume` partition, so volumes built by older releases are safe too

### Changed
- guest-agent: the external listener (port 8090) is no longer anonymous. Every route needs a scope, granted by a signed Ed25519 JWT or by `[cvm.guest_agent_api].anonymous_scopes` in sys-config. There is no built-in anonymous default; the sample `vmm.toml` opens `guest:dashboard`, `guest:info`, `guest:version` and `guest:health`, which is what an unconfigured 8090 used to serve, while containers, logs, metrics and app-key attestation stay behind a token. The gateway gets its token minted inside the CVM from the app root key. `public_logs` / `public_sysinfo` / `public_tcbinfo` are unchanged and still decide what may leave the TEE. The dashboard is filtered block by block by scope. See `docs/guest-api-auth.md`. An old gateway sees 401s and drops new guests from rotation until it is upgraded
- vmm: `qmp_socket` now defaults to `true`. Each CVM's QEMU exposes `qmp.sock` in its work dir, so the host can change disk options or compact a bloated `hda.img` (`blockdev-mirror`) without stopping the CVM. Applies to CVMs started after the change; set `qmp_socket = false` to opt out

## [0.6.1] - 2026-10-12
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,7 @@ Apps communicate with the guest agent via HTTP over `/var/run/dstack.sock`. Use
**Reference**
- [Guest Agent API v1](./docs/guest-api-v1.md) - Key derivation, signature chains, and the versioned guest API
- [Guest Agent API v0](./docs/guest-api-v0.md) - The frozen legacy surface, for clients that still speak the unversioned API
- [Guest Agent API access control](./docs/guest-api-auth.md) - Scopes, tokens, and anonymous access on port 8090
- [App Compose Format](./docs/normalized-app-compose.md) - Compose file specification
- [Intel TDX Attestation](./docs/attestation-tdx.md) - Measurement and runtime-event verification
- [Native TEE Interfaces](./docs/native-tee-interfaces.md) - Advanced compatibility with Linux TEE devices and configfs-tsm
Expand Down
178 changes: 178 additions & 0 deletions docs/guest-api-auth.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
# Guest Agent API access control

The guest agent's external listener (port 8090) is reachable by anyone who can
route to the CVM. It refuses callers it cannot identify: every request names a
scope, and a scope is granted either by a signed token or by the host's
configuration.

This is an exposure control, not a confidentiality one. It decides who may
*ask*. Whether the answer may leave the TEE is decided by the app's measured
`public_logs` / `public_sysinfo` / `public_tcbinfo` switches, and a token can
only narrow what those allow. See
[the two gates](#the-two-gates) below.

## Scopes

| Scope | Grants |
|---|---|
| `guest:version` | `Worker.Version` on either pRPC surface |
| `guest:health` | `Worker.Health` (`/prpc/v1`) |
| `guest:info` | `Worker.Info` on either surface, and the identity block on the dashboard |
| `guest:attest-app-key` | the frozen `Worker.GetAttestationForAppKey` |
| `guest:dashboard` | the dashboard page itself |
| `guest:containers` | the container table on the dashboard |
| `guest:logs` | `/logs/<container>`, when the app sets `public_logs` |
| `guest:metrics` | `/metrics`, and the sysinfo block on the dashboard |

`/prpc`, `/prpc/v0` and `/prpc/v1` are matched after percent-decoding and
segment normalisation, so `/prpc/Worker.Info`, `/prpc/v0/Info` and
`/%70rpc/Info` all need `guest:info`.

The check fails closed: a path the table does not name is answered with 401,
not passed to the router.

## Default: nothing anonymous

With no `[cvm.guest_agent_api]` block, an anonymous caller is granted nothing:
the dashboard, its identity block, `/metrics` and the log views all need a
token. The gateway still works, because it is issued one internally. The app's
own SDKs are unaffected: they use the internal Unix socket, which is
unauthenticated and reachable only by the application.

The sample `vmm.toml` ships the block filled in with what an unconfigured 8090
used to serve, so a host that copies the sample keeps the page it had:

| Scope | Effect |
|---|---|
| `guest:version` | `Worker.Version` answers |
| `guest:health` | `Worker.Health` answers |
| `guest:info` | the identity block renders |
| `guest:dashboard` | the page loads at all |

Everything else is about the app's *contents* and stays behind a token even in
that sample: the container table, `/metrics`, the log views and
`Worker.GetAttestationForAppKey`.

The dashboard does not fetch `Info` unless a block would show it, so an
anonymous visitor granted only `guest:dashboard` costs the TEE no attestation.

## Configuring anonymous access

Anonymous access is a host decision, made in sys-config. The VMM writes it from
`[cvm.guest_agent_api]`:

```toml
[cvm.guest_agent_api]
anonymous_scopes = ["guest:dashboard", "guest:version", "guest:health", "guest:info"]
```

The same block in the guest's `/dstack/.host-shared/.sys-config.json`:

```json
{
"guest_agent_api": {
"anonymous_scopes": ["guest:dashboard", "guest:version", "guest:health", "guest:info"]
}
}
```

The list is whole, not additive, and there is no built-in default to fall back
on: omitting `anonymous_scopes`, or leaving the list empty, closes the listener
to anonymous callers. Removing the block entirely does the same.

Listing `guest:dashboard` without `guest:containers`, `guest:info`, `guest:logs`
or `guest:metrics` gives an anonymous visitor the page with every block hidden.
Scopes are independent: each one adds exactly its own block.

An unknown name is a boot error rather than a permission that silently never
matches.

## External issuers

Other than its own token, the agent accepts tokens from issuers the host names:

```toml
[[cvm.guest_agent_api.external_issuers]]
name = "ops"
public_keys = ["2f8d...c1"] # hex Ed25519 public keys, 32 bytes each
```

`name` is the exact `iss` claim a token must carry; `public_keys` is a list so a
key can be rotated by publishing the new one beside the old, waiting out the
longest token lifetime (one hour), then dropping the old one.

Tokens must be Ed25519. `alg: none`, HMAC, and headers carrying key material
(`jku`, `x5u`, `jwk`) are not rejected by policy -- the header is a constant,
so they cannot be expressed.

### Where the internal issuer comes from

The gateway does not hold the app root key; the CVM mints its token and hands
it over on registration, bound to the app id and short-lived:

1. The agent and `dstack-util`, both inside the CVM, derive the same Ed25519
key: `derive_app_key(app_root_key, "dstack:guest-api:issuer:v1", Ed25519)`.
That domain string is final: it is a wire contract between `dstack-util` and
the agent, which are upgraded together but may briefly disagree, so it is
never versioned again.
2. `dstack-util` signs a token with `iss = "dstack:guest-api:<app_id>"`,
`aud = <app_id>`, a 10-minute lifetime, and only the `guest:health` scope.
3. Registration carries it to the gateway, which presents it on the v1 `Health`
poll.

The host cannot mint an internal token: the signing key is derived from the app
root key, which KMS releases only after the boot measurement is verified. The
host can still name its own [external issuer](#external-issuers) and grant
itself any scope; that is the same exposure as listing the scope in
`anonymous_scopes`, and still bounded by the measured switches.

### Trust inside the CVM

The CVM is one trust domain; the agent does not sandbox the processes in it. The
issuer key is an ordinary v1 application key, so any process that can reach
`dstack.sock` can obtain it with
`GetKey(domain = "dstack:guest-api:issuer:v1", algorithm = ed25519)` and mint a
token with any scope. That is by design -- the same process can already read
the data those scopes expose -- and it is also how an app issues tokens of its
own. Do not mount `dstack.sock` into a container you would not trust with
that.

## The two gates

Two independent checks decide what a caller sees, and both must pass:

| Gate | Source | Measured | Answers |
|---|---|---|---|
| `public_logs`, `public_sysinfo`, `public_tcbinfo` | app-compose | yes | may this data leave the TEE at all? |
| scopes | sys-config | no | may this caller ask? |

They combine with `&&`. A token cannot reveal logs the app kept private -- with
`public_logs = false` the route is not even mounted -- and an anonymous caller
cannot see a block it was not granted, however public the app made it.

Scope configuration is deliberately unmeasured. Host-supplied sys-config is not
part of the attestation, so a host could grant a caller more than the operator
intended. That grants nothing beyond what anonymous 8090 access already granted
before this change: the data is still gated by the measured switches, which a
host cannot alter without changing the app id.

## Compatibility

A guest with this change answers 401 to an old gateway that presents no token.
The gateway counts that as unreachable and drops the instance after
`failure_threshold` consecutive polls, so a mixed cluster rebalances away from
new guests until the gateway is upgraded.

During the rollout, either upgrade the gateway first, or leave the host open:

```toml
[cvm.guest_agent_api]
anonymous_scopes = ["guest:version", "guest:health", "guest:info", "guest:dashboard"]
```

The reverse direction is quieter. A new guest registers with an old gateway
node that does not store `agent_token`; when that record syncs to an upgraded
node, the missing token means the upgraded node's health probe is refused too.
The instance drops out of rotation while `order_by_stickiness` keeps routing
user traffic to it. Upgrade every gateway node in the cluster before rolling
guests, not just the one the VMM talks to.
5 changes: 5 additions & 0 deletions docs/guest-api-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -458,6 +458,11 @@ The v1 `Worker` has three methods, served at `/prpc/v1`.
Nothing here returns key material, and no caller chooses what gets signed or
attested.

Every method also needs a scope, granted by a signed token or by the host's
configuration. With no configuration the listener answers only token holders;
see [Guest Agent API access control](./guest-api-auth.md) for the vocabulary and
for how the gateway authenticates its health polls.

### There is no v1 AttestAppKey

The frozen surface has `Worker.GetAttestationForAppKey`, which attests a key the
Expand Down
11 changes: 10 additions & 1 deletion docs/security/cvm-boundaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,16 @@ Full specification: [host_api.proto](../../dstack/host-api/proto/host_api.proto)

### HTTP-based Public Guest API Service

The dstack-guest-agent runs an HTTP server on port 8090 inside the CVM. This port is publicly accessible, allowing external clients to view basic CVM information.
The dstack-guest-agent runs an HTTP server on port 8090 inside the CVM. By
default it refuses callers it cannot identify: every route needs a scope, granted
by a signed token or by the host's `guest_agent_api` sys-config. Anonymous access
is an explicit host choice. [guest-api-auth.md](../guest-api-auth.md) covers the
scope vocabulary and the token issuers.

Anonymous access is an exposure decision, not a confidentiality one. Whether the
data may leave the TEE at all is still decided by the app's measured
`public_logs` / `public_sysinfo` / `public_tcbinfo` switches, which a scope
cannot widen.

Since dstack 0.6.0 the listener serves two API surfaces, selected by URL path
alone: the frozen v0.5.11 `Worker` service at `/prpc` (equivalently
Expand Down
2 changes: 1 addition & 1 deletion docs/security/security-best-practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ Example app-compose.json:

## Don't expose unexpected ports

In dstack CVM, dstack-guest-agent listens on port 8090, allowing public access to basic CVM information.
In dstack CVM, dstack-guest-agent listens on port 8090. It is not anonymous by default; see [guest-api-auth.md](../guest-api-auth.md) for what the host can grant anonymously.

In docker-compose.yaml, all declared ports will be exposed to the public internet. Do not expose unnecessary ports.

Expand Down
2 changes: 1 addition & 1 deletion docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Open the dstack-vmm webpage [http://localhost:9080](http://localhost:9080) (chan
After the container is deployed, it should take some time to start the CVM and the containers. Time would vary depending on your workload.

- **[Logs]**: Click this button to view the CVM logs and monitor container startup progress
- **[Dashboard]**: Once the container is running, click this button to view container information and logs. (Note: This button is only visible when dstack-gateway is enabled. If disabled, you'll need to add a port mapping to port 8090 to access the CVM dashboard)
- **[Dashboard]**: Once the container is running, click this button to view container information and logs. (Note: This button is only visible when dstack-gateway is enabled. If disabled, you'll need to add a port mapping to port 8090 and allow the dashboard scopes in `[cvm.guest_agent_api]`; see [Guest Agent API access control](./guest-api-auth.md))

<div align="center">
<img src="./assets/guest-agent.png" alt="Guest Agent Dashboard" height="300">
Expand Down
5 changes: 5 additions & 0 deletions dstack/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions dstack/crates/api-auth/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@ license.workspace = true
anyhow.workspace = true
base64.workspace = true
bcrypt.workspace = true
ed25519-dalek.workspace = true
rocket.workspace = true
serde.workspace = true
serde_json = { workspace = true, features = ["std"] }
sha2.workspace = true
subtle.workspace = true
Loading
Loading