Sessions stop processing when MCP config watch registration hits the inotify limit (ENOSPC) #5242

Describe the bug

We run copilot --server in a container on shared Linux hosts. When the host's inotify watch budget was already used up the runtime failed to register the MCP configuration watches and the session queue was abandoned after one retry. After that sessions stopped processing prompts until the server was restarted.

[ERROR] [rust:copilot_runtime::shared_api::lifecycle] Scheduled queue processing failed; retrying once {"error":"Message(\"Failed to register MCP configuration watches: OS file watch limit reached. about [\\\"/root/.copilot\\\"]\")"}
[WARNING] [rust:copilot_runtime::shared_api::lifecycle] Queue recovery was rearmed after the bounded recovery attempt; releasing its admission {"reserved_order":"Some(0)"}
[ERROR] [rust:copilot_runtime::shared_api::lifecycle] Scheduled queue processing failed after retry; queue drain abandoned {"error":"Message(\"Failed to register MCP configuration watches: OS file watch limit reached. about [\\\"/root/.copilot\\\"]\")"}

In our case the watches seem to have been used up by another workload on the same host running as the same uid and not by Copilot itself. As far as I understand max_user_watches is tracked per uid across the host so a container doesn't get its own budget. The retry happens almost immediately so it most likely runs into the same limit.

Affected version

1.0.93, Linux x86_64, copilot --server

Steps to reproduce the behavior

I haven't reproduced this in isolation yet but I think something like this should trigger it:

1. sudo sysctl -w fs.inotify.max_user_watches=512 (or another low value)
2. As the same user use up the watches, e.g. inotifywait -m -r /usr
3. Start copilot --server and create a session via the SDK
4. Check the logs for the errors above and see whether the session responds

Expected behavior

It would be helpful if the runtime could fall back to polling the MCP config files in this case or continue without hot-reload and log a warning rather than abandoning the session queue.

From the binary there appears to be a polling path already (MCP configuration native watches unavailable; using polling). It may only be used when creating the watcher fails and not when adding a watch fails with MaxFilesWatch, but I may be wrong about that. If it's the latter then catching that error kind and switching to PollWatcher might be a fairly small change.

Polling seems like a reasonable fallback here since it's only a few config files and they don't change often. A full inotify budget also tends to stay full for a while so an immediate retry may not help much.

It might also be worth checking whether the other watchers (LSP, git worktree, agent registry) handle this the same way. Some of them appear to log "explicit refreshes remain available" and carry on, which seems like a reasonable approach.

Additional context

This looks similar to #4998, where a different MCP setup error also led to queue drain abandoned and unusable sessions. #3892 (session-state not being pruned) may also be relevant if the ~/.copilot watch is recursive.

For now we're working around it by raising fs.inotify.max_user_watches and max_user_instances on the hosts though that requires host-level access.