[nbayati]

A startup latency can occur where the environment variable pointing to the certificate configuration is set before the files are actually mounted on the filesystem. While token refresh already polls for the certificate file, mTLS transport discovery did not, causing transports initialized at container startup to fall back to standard TLS and fail with 401 once a bound token was minted.

• Return True from check_use_client_cert() when GOOGLE_API_CERTIFICATE_CONFIG points inside the well-known SPIFFE directory so mTLS endpoint selection is enabled at startup without blocking on disk I/O.
• Poll up to 30 seconds (50 retries at 100ms followed by 50 retries at 500ms) in _get_workload_cert_and_key() and _get_workload_cert_and_key_paths() for the config, certificate, and private key files to be written, failing fast on invalid config schemas, or PermissionError.
• Add unit tests in TestSpiffeWorkloadPolling covering cold-start polling recovery, timeout, and fail-fast conditions.

fixes b/580018256

[gemini-code-assist]

Code Review

This pull request introduces SPIFFE workload credential polling in the mTLS helper. It adds support for polling up to 30 seconds for SPIFFE workload configuration, certificate, and key files to become ready. It also refactors certificate configuration extraction and adds comprehensive unit tests covering various polling scenarios, timeouts, and error handling. I have no further feedback to provide.

[macastelaz]

Calling _get_spiffe_polling_config_path() and _poll_spiffe_workload_credentials() independently in both _get_workload_cert_and_key() (lines 419–424) and _get_workload_cert_and_key_paths() (lines 495–500) has a few side effects across callers:

1. Duplicate _get_spiffe_polling_config_path() call on every non-SPIFFE call to _get_workload_cert_and_key(): When _get_workload_cert_and_key(certificate_config_path, include_context_aware) runs for a non-SPIFFE path, _get_spiffe_polling_config_path(certificate_config_path) returns None at line 419, and line 426 then calls _get_workload_cert_and_key_paths(certificate_config_path, include_context_aware), which immediately calls _get_spiffe_polling_config_path(config_path) a second time with the same argument at line 495. Extracting lines 502–538 into a helper (e.g. _get_non_spiffe_workload_cert_and_key_paths(config_path=None, include_context_aware=True)) called by both functions after their _get_spiffe_polling_config_path() check avoids this duplicate call while keeping SPIFFE polling for direct callers of _get_workload_cert_and_key_paths().
2. 60s–90s total timeout when SPIFFE files fail to mount (mtls.py callers): In google/auth/transport/mtls.py, has_default_client_cert_source() (lines 45–55) wraps _get_workload_cert_and_key_paths() in try ... except (exceptions.ClientCertError, OSError): return True so the caller will surface the error. When SPIFFE files fail to mount within 30s, has_default_client_cert_source() blocks for 30s, catches the timeout ClientCertError, and returns True; the caller (load_default_client_cert(), default_client_cert_source(), default_client_encrypted_cert_source(), or grpc.SslCredentials) then calls _get_workload_cert_and_key(), which runs a second 30s polling loop (60s total; or 90s total in get_default_ssl_context() and GAPIC clients, which call has_default_client_cert_source() twice before loading credentials).
3. 2x–3x file reads/parses on the SPIFFE happy path: Because _get_workload_cert_and_key_paths() calls _poll_spiffe_workload_credentials() and discards (cert_data, key_data), callers that check has_default_client_cert_source() before calling get_client_ssl_credentials() (or identity_pool.Credentials._get_cert_bytes(), which calls _get_workload_cert_and_key_paths() followed by _read_cert_file(cert_path)) read and regex-validate the files 2–3 times.

[nbayati]

Thanks for the catch! I extracted the non-SPIFFE path lookup into a shared helper and updated mtls.has_default_client_cert_source() to return True without polling when the SPIFFE config path is configured.

[macastelaz]

Extracting the non-SPIFFE path lookup into a shared helper cleanly fixes the first point (_get_spiffe_polling_config_path() being called twice on non-SPIFFE calls).

However, returning True from mtls.has_default_client_cert_source() without polling/reading the config when _get_spiffe_polling_config_path() is not None introduces a regression when the SPIFFE config has {"cert_configs": {}} (no "workload" section, as tested in test_config_without_workload_section_does_not_poll):

Because _get_workload_cert_and_key() returns (None, None) (has_cert=False) rather than raising ClientCertError when "workload" is absent, having has_default_client_cert_source() return True without checking the file means:

1. mtls.has_default_client_cert_source() returns True when no cert source exists (previously it returned False).
2. mtls.default_client_cert_source() no longer raises MutualTLSChannelError("Default client cert source doesn't exist"), and its returned callback() returns (None, None) instead of (bytes, bytes) (causing GAPIC clients in _get_client_cert_source() to switch to *.mtls.googleapis.com and pass (None, None) to the transport).
3. mtls.default_client_encrypted_cert_source()() crashes with TypeError: a bytes-like object is required, not 'NoneType' at cert_file.write(cert_bytes).
4. grpc.SslCredentials().is_mtls is set to True on construction before .ssl_credentials is accessed.

Also, for the 3rd point, identity_pool.Credentials._get_cert_bytes() (google/auth/identity_pool.py:412-418) calls _mtls_helper._get_workload_cert_and_key_paths() directly (not has_default_client_cert_source()) and then calls _mtls_helper._read_cert_file(cert_path), so _get_workload_cert_and_key_paths() calling _read_cert_and_key_files() inside _poll_spiffe_workload_credentials() still reads the cert file twice there.

[macastelaz]

Because this raise is outside the except blocks, both __cause__ and __context__ on the raised ClientCertError are None when polling times out after 30 seconds. Callers only see "Identity certificate configuration or certificate/key files at ... were not ready after 30.0s" without knowing whether the config file never appeared (FileNotFoundError), the JSON was truncated, or a cert/key PEM file was missing/incomplete.

Consider initializing last_error = None before the loop, capturing last_error = caught_exc in both except (OSError, exceptions.ClientCertError) as caught_exc: blocks (lines 597 and 613), and chaining it on timeout:

    raise exceptions.ClientCertError(
        "Identity certificate configuration or certificate/key files at {} were not ready after {}s".format(
            spiffe_config_path, _agent_identity_utils._TOTAL_TIMEOUT
        )
    ) from last_error

[nbayati]

Done!

[macastelaz]

Nit: The docstring of check_use_client_cert() (lines 912–919) states that when GOOGLE_API_USE_CLIENT_CERTIFICATE is unset, True is inferred "if a workload config file exists (pointed at by GOOGLE_API_CERTIFICATE_CONFIG ...) containing a 'workload' section". Could we update the docstring to note that when GOOGLE_API_CERTIFICATE_CONFIG points inside the well-known SPIFFE directory, check_use_client_cert() returns True without requiring the file to already exist on disk?

[macastelaz]

In both test_polling_config_retry_success (lines 2420–2443) and test_polling_cert_and_key_retry_success (lines 2445–2476), the files are pre-created before _get_workload_cert_and_key() is called (either empty ""/b"" or truncated), which only exercises ClientCertError -> retry -> success. Consider adding None (where the config file or cert/key files do not exist on attempt 0 and are created on the first time.sleep) to @pytest.mark.parametrize so the FileNotFoundError (OSError) -> retry -> success path is covered as well:

    @pytest.mark.parametrize("initial_config", [None, "", "{truncated"])
    @mock.patch("google.auth.transport._mtls_helper.time.sleep", autospec=True)
    def test_polling_config_retry_success(
        self, mock_sleep, spiffe_dir, monkeypatch, initial_config
    ):
        cert_file = spiffe_dir / "certificates.pem"
        key_file = spiffe_dir / "private_key.pem"
        config_file = spiffe_dir / "credentials.json"
        cert_file.write_bytes(pytest.public_cert_bytes)
        key_file.write_bytes(pytest.private_key_bytes)
        if initial_config is not None:
            config_file.write_text(initial_config)

[nbayati]

Done!

[macastelaz]

nit: In test_external_cert_and_key_paths_not_polled, both external_cert and external_key are outside spiffe_dir, so _agent_identity_utils._is_in_well_known_dir(cert_path) and _agent_identity_utils._is_in_well_known_dir(key_path) (_mtls_helper.py:615-616) always short-circuits on the first operand (False) when _read_cert_and_key_files() fails. Consider also testing the case where cert_path is inside spiffe_dir (valid cert) and key_path is outside spiffe_dir (missing or invalid key) so the second operand of the and check is exercised.

[nbayati]

Done!