Repository navigation
fix(storage): honor timeout in upload_chunks_concurrently - #18765
abhishekacharya200 wants to merge 2 commits into
Conversation
upload_chunks_concurrently accepted a timeout but never passed it to the XML multipart requests, so initiate, every part upload, finalize and cancel always ran with the (61, 60) default. Pass it through to all of them, including the _upload_part worker. Callers who don't pass a timeout now get the documented 60 seconds (_DEFAULT_TIMEOUT, as every Blob upload and download already sends) instead of (61, 60), so only the connect timeout moves by one second. Fixes googleapis#18764
There was a problem hiding this comment.
Code Review
This pull request propagates the timeout parameter to various container and part operations within upload_chunks_concurrently (such as initiate, finalize, cancel, and _upload_part) and adds corresponding unit tests. The review feedback recommends wrapping the container.cancel call inside the exception handler in a nested try...except block to prevent cancellation failures from masking the original exception.
| except Exception: | ||
| container.cancel(blob._get_transport(client)) | ||
| container.cancel(blob._get_transport(client), timeout=timeout) | ||
| raise |
There was a problem hiding this comment.
If container.cancel raises an exception (for example, due to a timeout or network issue during the cleanup attempt), it will mask the original exception that triggered the except block. To ensure the original error is preserved and propagated to the caller for proper debugging, wrap the container.cancel call in a nested try...except block. Avoid silently ignoring the exception with a broad except Exception: pass block; instead, log the exception using logger.warning to aid in debugging and prevent masking underlying issues.
except Exception:
try:
container.cancel(blob._get_transport(client), timeout=timeout)
except Exception as e:
logger.warning("Failed to cancel container: %s", e)
raiseReferences
- Avoid broad
except Exception:blocks that silently returnNone. Instead, log the exception (e.g., usinglogger.warning) to aid in debugging and prevent masking underlying issues.
There was a problem hiding this comment.
Good point. This was already the case before this change, but passing the caller's timeout to cancel makes it a little more likely. I kept the PR focused on the timeout, and I'm happy to wrap the cancel call so the original error is always the one raised, or to leave cancel on the default timeout, whichever the maintainers prefer.
There was a problem hiding this comment.
Thanks for checking! Let's keep passing timeout=timeout to container.cancel (so fail-fast or extended caller timeouts are respected consistently across all requests) and wrap container.cancel in a try...except Exception block so a cleanup failure never masks the original upload exception.
Two small notes when adding that:
-
transfer_manager.py doesn't define a logger yet — please use _logger = logging.getLogger(name) (matching blob.py) and include upload_id in the _logger.warning(...) message.
-
Since nox -s cover enforces 100% branch coverage, please also add a unit test where container.cancel raises during cleanup to verify the original exception is preserved and the warning is logged.
There was a problem hiding this comment.
Thanks @shradhakatyal, done in 1520e8b. cancel keeps the caller's timeout and is wrapped so a failed cleanup never masks the upload error. The warning includes the upload id, and a new test covers a failing cancel. Ready for another look.
|
/gcbrun (ccc12b7) |
A failure in the cleanup cancel is now logged with the upload id, so it can no longer replace the exception that caused the cleanup.
Fixes #18764
upload_chunks_concurrentlytakes atimeoutargument and documents it as the time to wait for the server response, but it never passed it on. The XML multipart requests it makes (initiate, each part upload, finalize, and cancel when something fails) all ran with the(61, 60)default of the media layer, so a caller could not make a stuck upload fail sooner or give a slow connection more time.This passes
timeouttoXMLMPUContainer.initiate,finalizeandcancel, and adds it to the_upload_partworker so everyXMLMPUPart.uploadgets it as well. Thread and process workers behave the same, since the timeout is a plain number or tuple.One small change for callers who don't pass a timeout: these requests now use the documented default of 60 seconds (
_DEFAULT_TIMEOUT, the value everyBlobupload and download already sends) instead of(61, 60). Only the connect timeout moves, by one second.The cleanup
cancelgets the same timeout as the other requests. If the cancel itself fails, for example with a very short timeout, the failure is logged as a warning with the upload id and the original upload error is still the one raised.test_upload_chunks_concurrently_keeps_original_error_when_cancel_failscovers that path.With the script from the issue, which records the timeout each request gets:
Added
test_upload_chunks_concurrently_passes_timeoutandtest_upload_chunks_concurrently_passes_timeout_to_cancel, which fail onmainand pass with this change. The existing tests that checked these calls now also check that the default timeout is sent, andtest__upload_partchecks that the worker hands its timeout to the part upload. The fulltests/unitsuite passes locally (2068 passed), and the noxfile'sruff==0.14.14import and format checks andflake8report nothing.