Skip to content

Decentralized controller consistency - #1273

Open
leonard-amsler wants to merge 7 commits into
mainfrom
decentralized-controller-consistency
Open

leonard-amsler wants to merge 7 commits into
mainfrom
decentralized-controller-consistency

Conversation

@leonard-amsler

Copy link
Copy Markdown
Collaborator

Context

#1227 reworked the federated setting. This PR moves the shared parts to the abstract classes and uses them also in the decentralized setting.

Server

  • Moved send, crashClient, disconnect, the client ids, the ClientConnected counter and the connect/disconnect cleanup from the federated controller to TrainingController, so both controllers share them.
  • The decentralized controller now crashes a peer that:
    • sends a message before ClientConnected
    • sends more than 5 ClientConnected
    • sends an invalid message, or a message the server fails to handle
  • A repeated ClientConnected gets the same answer again, without counting the peer twice or notifying the other peers.
  • Every message goes through the send helper, so a failed send disconnects the peer.
  • A peer leaving is now also removed from #syncingNodes.
  • Documented the sets of the controller and moved to # methods.
  • Fixed the connection timer: after a retry, the old timer kept running in the next rounds and could send RetryPeerConnections, and then ConnectionFail, to peers that were fine. There is now one timer per PeersForRound, and the retry count restarts at each round.

Client

  • disconnect() is now shared in Client. It closes the server connection if there is one (local training) and stops waiting for participants, so a client no longer keeps waiting after disconnecting. It also works when called multiple times.
  • The decentralized client retries its join with sendAndWaitWithRetry.
  • It stops with a ClientCrashError when the server crashes it, at the start of a round and while waiting for the server.
  • CrashClient is now part of the decentralized messages.

Tests

  • Decentralized controller: join retry, crashing or disconnecting a peer
  • Decentralized client: crashed and disconnected by the server after too many joins

Fixes #1250

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decentralized and federated controller consistency

1 participant