[geonmo]

Motivation:
Clients can't see how much space dCache has. The NFS door reports a fixed size of 1 EiB, so df on an NFS mount is of no use to users or monitoring. The xrootd door doesn't answer kXR_Qspace at all, so xrdfs spaceinfo fails and df hides xrootdfs mounts (size 0).

Modification:
Add dcache.capacity.source, inherited by nfs.capacity.source and xrootd.capacity.source:

• namespace (default): a fixed 1 EiB, as the NFS door reports today
• pools: the total and free space of the pools that files in the queried path are first written to, counting removable replicas as free, as the frontend does for SRR; 1 EiB if no write link matches. With space reservation, link groups are chosen as the spacemanager does: that of the WriteToken, or else a matching one as if unreserved uploads to link groups were allowed, since doors can't see that spacemanager setting
• reservation: the size and available space of the space reservation given by the WriteToken tag of the queried path, as the FTP door does for SITE USAGE, or as 'pools' for paths without one

Capacity.Provider computes the value for both doors and logs a failure once, until the capacity can be determined again. The xrootd door answers kXR_Qspace in the format of xrootd's oss plugin for paths the user can access. As nfs4j doesn't pass the queried file to the door, the NFS door reports all pools for 'pools' and needs a space token in nfs.capacity.reservation for 'reservation', or it fails to start. It keeps the namespace file counts and falls back to the namespace statistics on failure.

Result:
By default, NFS is unchanged, and xrootd clients now see 1 EiB, so df shows xrootdfs mounts. Disk-only sites can report the capacity of their pools, and sites using space reservations the capacity of a reservation.

Fixes: #8275
Target: master
Require-book: yes

[DmitryLitvintsev]

In case of capacity.reservation - it works if you have each mount for the directory root having a single space reservation. But what if the root of the tree you mount has multiple space reservations (a differebnt WriteToken in different subdirectories)? I suspect what we really need is not the df, but ability to query space reservations by path via xrootd kXR_Qspace. The behavior I would suggest should be:

if directory is associated with space reservation (via WriteToken tag) rerturn the value if the corresponding space reservation (total, used, available) else (if there is no space reservation) return the (total, used, available) based on pool group that this path is associated based on selection criteria (storage tags in the directory).

This is still could be confusing. Say I have top level directory “/“ and I have two subdirectories under it, say “foo” w/ its WriteToken and “bar” w/ its WriteToken

If I mount:

mount localhost:/ /mnt

what the df should show? Technically it should show 0 as there is no WriteToken associated with “/“ and therefore you can’t write into “/“ or any subdirectorty created under it besides existing “foo” and “bar”.

But I am afraid showing 0 is not good.

What if your system does not use space reservation at all (we do not use space reservations at Fermilab). Instead we use quota and we can query quotas using REST interface to know how much is available.

W/o space reservation and not using quotas, the same problem remaibs, say directory “/“ has no tags and therefore has not matchig selection and therefore has no pool group associated. Whereas “foo” and “bar” directories under it do have tags and associated different pool groups. So again, then , mounting “/“ would give df total space 0.

[geonmo]

Thanks, Dmitry. Before I push an update (still one squashed commit), here is how I'd handle your points. It is tested at KISTI on 11.2.10.

Per-path capacity (xrootd). kXR_Qspace now reports the space that new files in the queried directory would go to, much like xrootd's own oss, which reports the cache group of the path:

• reservation: the reservation given by the directory's WriteToken. Directories without one fall back to pools.
• pools: the online pools that PoolManager selects for writing to that path (PSU match for the path's storage info, the client address and Xrootd/<major>, first preference level). If xrootd.enable.space-reservation is true, I use the link group of the WriteToken. Without a WriteToken I use a matching link group, as SpaceManager does when unreserved-uploads-to-linkgroups is on; doors can't see that setting. Only then do I fall back to links outside link groups.
• If no write link matches, I report 1 EiB instead of 0, so df still shows the mount.

Properties. dcache.capacity.source (namespace|pools|reservation) stays shared by NFS and xrootd. I dropped dcache.capacity.reservation and xrootd.capacity.reservation, because xrootd always follows the path. Only nfs.capacity.reservation is left. It is required for reservation, and the door fails to start without it, because nfs4j's VirtualFileSystem.getFsStat() gets no file handle. NFS therefore reports all pools or that one reservation per door. I'll propose a getFsStat(Inode) in nfs4j separately; once that is in, this property can go.

df semantics. The value means "space for new writes here", not the usage of the files below the directory. Files placed with an explicit SRM token, or moved in, are not reflected. With SRM and GridFTP gone, the WriteToken tag is the only way an HTTPS or xroot upload picks a reservation. Note also that xrootdfs caches statfs for 120 s per mount, not per path (XrdFfsFsinfo.cc), so df shows whichever path was queried first. df is therefore only consistent if the tree below the mount point shares one reservation or pool group, e.g. /store rather than /. I'll raise the caching with xrootd.

Tests (11.2.10, temporary doors):

• reservation: /, /store → 6924; /ops → 56351.
• pools: every path equals the sum in /info/pools (20 pools, one link group, unreserved uploads on).
• NFS: matches as well.
• Unit tests cover the link group choice.

Quotas. I'd keep quotas out of this PR and add a quota source in a follow-up, reporting min(remaining quota, free space of the path's write pools) for the retention policy of the directory. Since dCache quotas are per uid/gid rather than per path, the only thing a door can report for a mount is the quota of the directory's owner. Is that how you set it up at Fermilab, i.e. experiment areas owned by the experiment's gid with group quotas? If you use user quotas instead, I don't think a per-path value makes sense and I'd leave it out. If you'd rather have quotas in this PR, I can add them here.