UploadStream for Azure Client #27
Open
+278
−49
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- UploadStream function upload by chunking of size 4MB and with 5 go routines - This function is used when file size is bigger then 32MB - These magic numbers are based on fog/azurebs library, check in this link https://github.com/fog/fog-azure-rm/blob/68ea3f7dd14c0a524146f1e426fa2ad1f6192cc8/lib/fog/azurerm/constants.rb
cf-foundation-community-automation
bot
moved this to Inbox
in Foundational Infrastructure Working Group
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Large file uploads require a different strategy than small, single-shot uploads. To stay compatible with existing behavior, the chunking and concurrency model should follow the same conventions used in the Fog Azure implementation.
Solution
This PR introduces an
UploadStreamfunction that uploads files in chunks of 4MB using 5 goroutines. The function is triggered when the file size exceeds 32MB.These values are aligned with the Fog Azure implementation and are based on the constants defined in:
https://github.com/fog/fog-azure-rm/blob/68ea3f7dd14c0a524146f1e426fa2ad1f6192cc8/lib/fog/azurerm/constants.rb