Repository navigation
Add RDMA and URMA runtime recovery and TCP fallback - #92
Draft
Spicy-cream wants to merge 9 commits into
Draft
Spicy-cream wants to merge 9 commits into
Spicy-cream wants to merge 9 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RDMA 和 URMA 运行期故障后,后续连接可能反复尝试同一路径。本 PR 增加 LEGACY、AUTO、TCP_ONLY 恢复模式。默认 LEGACY;AUTO 按 RDMA 端口和对端状态、URMA 对端状态控制新连接,冷却期间选择 TCP,默认冷却 5 秒。两种传输分别维护模式、epoch、冷却缓存和指标,模式切换过滤旧连接错误。
统一完成错误、QP/CQ 异步错误、CQ/JFC 轮询及通知错误和 WR 提交失败的上报。同一连接只记录首个故障,访问错误单独计数,flush 不重复归因,窗口耗尽保持 EAGAIN。RDMA QP/CQ 校验资源归属,故障资源直接释放;URMA 支持 v2/v3 服务端协商降级。ACK 提交失败向连接错误处理传播。
新增恢复状态 API、协商与 TCP 准入指标,以及支持运行中切换模式和 CSV 统计的恢复客户端。URMA 示例增加 urma_recovery_client 构建入口,复用原恢复客户端源码,通过 URMA 示例链接依赖。URMA 诊断记录完成方向、发送槽位、缓冲区地址和注册段范围;发送 SGE 完整初始化并校验范围与 context。默认关闭的 urma_copy_send 将数据复制合并到 64 字节对齐的注册缓冲区并持有至完成,用于对比原发送路径,增加一次 CPU 拷贝。
本地生产静态库、恢复客户端和 URMA 性能示例构建通过。策略 8 项、RDMA mock 97 项、URMA 单测 25 项和复制发送 RPC 测试 2 项已通过。URMA 单独启用时 endpoint、helper 和恢复客户端源码通过 C++14 编译。新增入口在本地 mock 环境完成 TCP_ONLY 和模式文件覆盖验证,两轮共 40 次 RPC,失败数为 0;非法模式被拒绝。文档补充 URMA 构建入口及低频功能检查。
模式切换作用于新连接;存量连接退休、单连接探测、发送 watchdog 和多端口主备为后续迭代。
原 node1 udmac0d1e2 与 node2 udmac0d1e3 组合中,brpc 协商成功后服务端 SEND 报 status 4(本地访问错误),描述符修正、单 SGE 和对齐复制下均可复现。原生 RM、CTP、共享 JFR、64 字节 SEND 在非 inline 和 inline 模式也失败。node3 udmac0d1e2 与 node2 udmac0d1e3 的原生测试同样失败。两端已检查的 liburma、UDMA provider 哈希及 UDMA 驱动 srcversion 一致;已采集设备辅助寄存器,原生故障根因待定位。
新增对照中,node2 和 node3 均使用 udmac0d1e2,原生 inline SEND 完成 1000 次交换,两端退出码均为 0。接下来使用该组合验证 PR92 的正常 RPC、TCP_ONLY 和运行中模式切换;对端故障冷却与自动 TCP 回落的实机验证尚未完成。当前执行功能验收,吞吐和 P99 另作性能测试。