Skip to content

chore(kokoro): add Maven timeouts to prevent macOS VM hangs - #14899

Closed
sakthivelmanii wants to merge 9 commits into
mainfrom
chore-kokoro-kokoro-macos-vm-timeout
Closed

sakthivelmanii wants to merge 9 commits into
mainfrom
chore-kokoro-kokoro-macos-vm-timeout

Conversation

@sakthivelmanii

Copy link
Copy Markdown
Contributor

Problem

  • The Kokoro java8-osx build fails with ERROR: Aborting VM command due to timeout of 10800 seconds.
  • Maven Central artifact resolution (nexus-staging-maven-plugin, etc.) occasionally hangs on macOS runners.
  • Because Maven 3 has no default read timeout, the mvn install command can block indefinitely (hanging for > 2 hours), leading to the 3-hour Kokoro VM hard timeout without triggering the retry_with_backoff logic.

Solution

  • Inject explicit Aether and HTTP Wagon connection/read timeouts (-Daether.connector.requestTimeout=300000, -Daether.connector.connectTimeout=30000, etc.) into MAVEN_OPTS before executing mvn install within retry_with_backoff.
  • This ensures network hangs fail fast, allowing the existing shell backoff logic to successfully retry the transient network issue.

Injects Aether and Wagon read timeouts into MAVEN_OPTS inside retry_with_backoff to fail fast on Maven Central hangs instead of blocking for 3 hours.
@sakthivelmanii
sakthivelmanii enabled auto-merge (squash) November 3, 2026 02:43
@sakthivelmanii
sakthivelmanii requested a review from a team as a code owner November 3, 2026 02:43
@sakthivelmanii

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds defensive timeouts to Maven commands in .kokoro/common.sh to prevent Kokoro VM hangs. The review feedback recommends refining the command matching logic to support Maven wrappers (mvnw), avoid false positives, and prevent MAVEN_OPTS from growing indefinitely on repeated invocations.

Comment thread .kokoro/common.sh Resolved

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces defensive timeouts to Maven commands in .kokoro/common.sh to prevent Kokoro VM hangs. The review feedback suggests a more robust way to detect Maven commands by checking the executable name (the first element of the command array) rather than searching the entire command string, which avoids false positives/negatives. Additionally, it recommends checking if the timeout options are already present in MAVEN_OPTS before appending them to prevent accumulation over multiple calls.

Comment thread .kokoro/common.sh Resolved
@sakthivelmanii

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces defensive timeouts and retry configurations to Maven commands in '.kokoro/common.sh' to prevent Kokoro VM hangs. The reviewer suggests using the '${MAVEN_OPTS:-}' parameter expansion syntax to ensure robustness against unbound variable errors when 'set -u' is enabled.

Comment thread .kokoro/common.sh Resolved
@sakthivelmanii

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds defensive timeouts to Maven commands in .kokoro/common.sh to prevent Kokoro VM timeouts. The reviewer suggested using a cleaner bash parameter expansion to avoid introducing a leading space in MAVEN_OPTS when it is initially unset or empty.

Comment thread .kokoro/common.sh Resolved
@sakthivelmanii

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds defensive timeouts to Maven commands in .kokoro/common.sh to prevent Kokoro VM hangs. The feedback suggests also adding Wagon-specific timeout properties (maven.wagon.rto and maven.wagon.connectionTimeout) to MAVEN_OPTS to ensure that Wagon-based plugins, which do not use Aether, are also covered.

Comment thread .kokoro/common.sh Resolved
@sakthivelmanii

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds defensive timeouts to Maven commands in .kokoro/common.sh to prevent Kokoro VM hangs. The feedback suggests using ${command[0]:-} instead of ${command[0]} to avoid potential 'unbound variable' errors under set -u when the command array is empty.

Comment thread .kokoro/common.sh Resolved
@sakthivelmanii

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces defensive timeouts to Maven commands in .kokoro/common.sh to prevent Kokoro VM timeouts. The review feedback suggests simplifying the multiple OR conditions used to detect Maven commands by using a regular expression match.

Comment thread .kokoro/common.sh Resolved
@sakthivelmanii

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds defensive timeouts to Maven commands in .kokoro/common.sh to prevent Kokoro VM hangs. The feedback suggests storing the regular expression in a local variable to ensure compatibility with older Bash versions, such as Bash 3.2 on macOS.

Comment thread .kokoro/common.sh Resolved
@sakthivelmanii

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces defensive timeouts to Maven commands in '.kokoro/common.sh' to prevent Kokoro VM timeouts. The feedback suggests using 'local -x' instead of 'export' to scope the modified 'MAVEN_OPTS' to the function and splitting the long string into multiple lines to improve readability.

Comment thread .kokoro/common.sh Resolved
@sakthivelmanii

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates .kokoro/common.sh to inject defensive timeout and retry configuration options into MAVEN_OPTS when executing Maven commands (mvn or mvnw). This is designed to prevent long-running hangs and VM timeouts during Kokoro builds. There are no review comments, and I have no additional feedback to provide on these changes.

@sakthivelmanii

Copy link
Copy Markdown
Contributor Author

Closing this PR as we will configure Maven HTTP timeouts in our Spanner CI runner script instead of modifying the shared monorepo script.

auto-merge was automatically disabled November 3, 2026 09:56

Pull request was closed

@sakthivelmanii
sakthivelmanii deleted the chore-kokoro-kokoro-macos-vm-timeout branch November 3, 2026 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant