Skip to content

fix(scheduler): initialize RabbitMQ IO thread before shutdown - #2532

Open
linsheng-li wants to merge 2 commits into
MemTensor:dev-v2.0.36from
linsheng-li:fix/rabbitmq-shutdown-thread-init
Open

linsheng-li wants to merge 2 commits into
MemTensor:dev-v2.0.36from
linsheng-li:fix/rabbitmq-shutdown-thread-init

Conversation

@linsheng-li

Copy link
Copy Markdown

Description

Stopping a scheduler that has never initialized RabbitMQ logs AttributeError: 'OptimizedScheduler' object has no attribute '_io_loop_thread' during API shutdown. This includes local-queue deployments and initialization skipped in CI. No restart is required to trigger it.

The constructor initializes _rabbitmq_io_loop_thread, but both initialize_rabbitmq() and rabbitmq_close() use _io_loop_thread. Initialize the same field in the constructor. The production change is one line; connection cleanup, thread joins, shutdown ordering, and exception handling are unchanged. No new dependencies or configuration are required.

Related issue: N/A; the reproduction and regression tests are included here.

Type of change

  • Bug fix (non-breaking)

How Has This Been Tested?

  • Unit tests: pytest tests/mem_scheduler/test_rabbitmq_shutdown.py -q — 5 passed. Without the fix, 3 failed, 2 passed. Covers uninitialized/repeated close, CI initialization skip, connection cleanup with a real IO thread, native optimized-scheduler lifecycle cleanup, and propagation of unexpected join errors.
  • Test steps: isolated API shutdown with RabbitMQ uninitialized, followed by restart and a second shutdown — both exits were clean; graph/vector/business snapshots were unchanged. No generation or embedding requests were made.
  • make format — all Ruff checks passed; 631 files unchanged.
  • Pre-commit checks on both changed files — passed.

Additional directory run: pytest tests/mem_scheduler/ -q — 70 passed, 1 skipped, 1 failed. The existing test_dynamic_cache_layers_access requires PyTorch, which is absent in the test image; the same failure was reproduced against the unmodified source. Pytest also warns that the image lacks the asyncio plugin. The full repository suite was not run.

Checklist

  • Self-reviewed the one-line fix and focused tests.
  • Added tests demonstrating the failure and preserving existing cleanup behavior.
  • No documentation/API/schema changes are needed for this internal field-name correction.

@Memtensor-AI Memtensor-AI added area:scheduler 调度模块 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Oct 21, 2026
@Memtensor-AI
Memtensor-AI requested a review from wustzdy October 21, 2026 15:05
@Memtensor-AI

Memtensor-AI commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2532
Task: c326249ca4d67364
Base: dev-v2.0.36
Head: fix/rabbitmq-shutdown-thread-init

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

Generated by cloud-assistant via Open Code Review.

@endxxxx
endxxxx changed the base branch from main to dev-v2.0.36 October 24, 2026 13:57
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

❌ Automated Test Results: FAILED

Auto-fix retry 1/2 triggered.

Failed tests:

  • test_close_without_rabbitmq_initialization_is_safe_and_repeatable
  • test_early_initialization_skip_still_closes
  • test_real_io_thread_and_connection_cleanup_still_execute
  • test_actual_optimized_scheduler_lifecycle_joins_threads_and_shuts_executor
  • test_unexpected_join_error_is_not_hidden
Error details
Tests failed. Failed cases: test_close_without_rabbitmq_initialization_is_safe_and_repeatable, test_early_initialization_skip_still_closes, test_real_io_thread_and_connection_cleanup_still_execute, test_actual_optimized_scheduler_lifecycle_joins_threads_and_shuts_executor, test_unexpected_join_error_is_not_hidden

Branch: fix/rabbitmq-shutdown-thread-init

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scheduler 调度模块 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants