Skip to content

Support assume-role for Glue and DynamoDB catalogs - #4132

Open
David-Banquet wants to merge 2 commits into
apache:mainfrom
David-Banquet:assume-role-catalogs
Open

David-Banquet wants to merge 2 commits into
apache:mainfrom
David-Banquet:assume-role-catalogs

Conversation

@David-Banquet

Copy link
Copy Markdown

Closes #2747

Rationale for this change

This picks up #3838 by @lemon0333, which was closed for inactivity before anyone reviewed it (see my comment there). Their commit is kept as is, with them as author.

client.role-arn / client.role-session-name are honored by the S3 FileIO but ignored by the Glue and DynamoDB catalogs, which build their own boto3.Session. #3838 extends them to both catalogs, with service-prefixed overrides (glue.role-arn, dynamodb.role-arn, and the matching *-role-session-name) falling back to the client.* ones, like the other session properties.

The second commit changes one thing: #3838 built the catalog session from the static credentials of a single AssumeRole call, so a long-lived catalog started failing with ExpiredToken once the role session ended (one hour by default). The session now uses botocore's AssumeRoleCredentialFetcher with DeferredRefreshableCredentials: the role is assumed on first use and again before the credentials expire, the same way botocore handles role_arn in an AWS profile.

Are these changes tested?

Yes, with moto:

uv run python -m pytest tests/catalog/test_glue.py tests/catalog/test_dynamodb.py
# 99 passed
uv run prek run --files <changed files>
# all passed

Are there any user-facing changes?

Yes: the Glue and DynamoDB catalogs now honor client.role-arn / client.role-session-name, plus the new glue.role-arn, glue.role-session-name, dynamodb.role-arn and dynamodb.role-session-name. Documented in configuration.md. Nothing changes when no role ARN is set.

🤖 Generated with Claude Code

lemon0333 and others added 2 commits October 21, 2026 18:37
Extend the existing S3 FileIO assume-role mechanism (client.role-arn /
client.role-session-name) to the boto3 sessions built by GlueCatalog and
DynamoDbCatalog. When a role ARN is configured, an STS AssumeRole is
performed from the base session and the resulting temporary credentials
are used for the Glue/DynamoDB client.

Role configuration follows the same precedence already used for the other
client.* properties: a service-prefixed override (glue.role-arn /
dynamodb.role-arn, glue.role-session-name / dynamodb.role-session-name)
falls back to the unified client.role-arn / client.role-session-name.
Default behavior (no role ARN configured) is unchanged.

Closes apache#2747

Signed-off-by: lemon0333 <147061193+lemon0333@users.noreply.github.com>
The catalog session was built from the static credentials of a single
AssumeRole call, so a long-lived catalog failed with ExpiredToken once the
role session ended (one hour by default). Use botocore's
AssumeRoleCredentialFetcher with DeferredRefreshableCredentials: the role is
assumed on first use and again before the credentials expire.

Co-authored-by: lemon0333 <147061193+lemon0333@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow to assume role when connecting to AWS catalogs (glue/dynamodb)

2 participants