aboutsummaryrefslogtreecommitdiff
path: root/weed/s3api/chunked_reader_v4.go
AgeCommit message (Collapse)AuthorFilesLines
2025-12-01Fix issue #6847: S3 chunked encoding includes headers in stored content (#7595)Chris Lu1-39/+38
* Fix issue #6847: S3 chunked encoding includes headers in stored content - Add hasTrailer flag to s3ChunkedReader to track trailer presence - Update state transition logic to properly handle trailers in unsigned streaming - Enhance parseChunkChecksum to handle multiple trailer lines - Skip checksum verification for unsigned streaming uploads - Add test case for mixed format handling (unsigned headers with signed chunks) - Remove redundant CRLF reading in trailer processing This fixes the issue where chunk-signature and x-amz headers were appearing in stored file content when using chunked encoding with newer AWS SDKs. * Fix checksum validation for unsigned streaming uploads - Always validate checksum for data integrity regardless of signing - Correct checksum value in test case - Addresses PR review feedback about checksum verification * Add warning log when multiple checksum headers found in trailer - Log a warning when multiple valid checksum headers appear in trailers - Uses last checksum header as suggested by CodeRabbit reviewer - Improves debugging for edge cases with multiple checksum algorithms * Improve trailer parsing robustness in parseChunkChecksum - Remove redundant trimTrailingWhitespace call since readChunkLine already trims - Use bytes.TrimSpace for both key and value to handle whitespace around colon separator - Follows HTTP header specifications for optional whitespace around separators - Addresses Gemini Code Assist review feedback
2025-10-31S3: fix TestSignedStreamingUploadInvalidSignature test (#7421)Chris Lu1-1/+0
* Added continue statements after all state transitions in the state machine to ensure immediate state processing * simplify * remove redundant continue clause * ensure wrong signature
2025-10-25s3: combine all signature verification checks into a single function (#7330)Tom Crasset1-75/+7
2025-08-11S3 API: unsigned streaming (no cred) but chunks contain signatures (#7118)Chris Lu1-13/+31
* This handles the case where we have unsigned streaming (no cred) but chunks contain signatures * Update chunked_reader_v4.go * address comments
2025-08-05fix signature hashing for iam (#7100)Chris Lu1-15/+17
* fix signature hashing for iam * add tests * address comments * Update weed/s3api/auto_signature_v4_test.go Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * indention * fix test --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-07-15rewrite, simplify, avoid unused functions (#6989)Chris Lu1-44/+27
* adding cors support * address some comments * optimize matchesWildcard * address comments * fix for tests * address comments * address comments * address comments * path building * refactor * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * address comment Service-level responses need both Access-Control-Allow-Methods and Access-Control-Allow-Headers. After setting Access-Control-Allow-Origin and Access-Control-Expose-Headers, also set Access-Control-Allow-Methods: * and Access-Control-Allow-Headers: * so service endpoints satisfy CORS preflight requirements. * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix * refactor * Update weed/s3api/s3api_bucket_config.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_object_handlers.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update weed/s3api/s3api_server.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * simplify * add cors tests * fix tests * fix tests * remove unused functions * fix tests * simplify * address comments * fix * Update weed/s3api/auth_signature_v4.go Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * rename variable * Revert "Apply suggestion from @Copilot" This reverts commit fce2d4e57e6f712672e62e8c63468c6b89878c6c. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-07-06fix: s3 return BadDigest (#6714)Konstantin Lebedev1-4/+3
* fix: s3 return BadDigest * adjust error message checking --------- Co-authored-by: chrislu <chris.lu@gmail.com>
2025-04-23[s3] fix checksum algorithm CRC64NVMe (#6722)Konstantin Lebedev1-2/+2
2025-03-31chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 (#6679)dependabot[bot]1-1/+1
* chore(deps): bump gocloud.dev from 0.40.0 to 0.41.0 Bumps [gocloud.dev](https://github.com/google/go-cloud) from 0.40.0 to 0.41.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](https://github.com/google/go-cloud/compare/v0.40.0...v0.41.0) --- updated-dependencies: - dependency-name: gocloud.dev dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * fix error * fix printing errors * Update go.mod --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>
2025-03-07chore: make function comment match function name (#6607)NinaLua1-1/+1
2025-02-12implement s3 streaming-unsigned-payload-trailer (#6539)Tom Crasset1-20/+276
* implement s3 streaming-unsigned-payload-trailer * chore: remove print
2025-02-07add s3 signature tests and prepare implementation of ↵Tom Crasset1-3/+11
STREAMING-UNSIGNED-PAYLOAD-TRAILER (#6525) * add tests for s3 signature * add test for newSignV4ChunkedReader.Read() * add glog import
2023-09-05Improve S3 request signing performancePatrick Schmidt1-27/+33
This change is caching HMAC hashers for repeated use in subsequent requests and chunks, so they don't have to be initialized from scratch every time. On my local computer this gives me ~5-6 times faster signature calculation and ~5-6.5% more throughput in S3 requests. The smaller the payload the better the throughput gets.
2022-09-15docs(s3api): readability improvements (#3696)Ryan Russell1-1/+1
Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14go fmtchrislu1-2/+4
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu1-2/+2
2022-05-30move s3 related constants from package http to s3_constantschrislu1-2/+1
2022-01-03wildcard prefix to restrict access to directories in s3 bucketchrislu1-2/+3
https://github.com/chrislusf/seaweedfs/discussions/2551
2021-12-07audit logKonstantin Lebedev1-1/+2
2021-07-03S3 authorization: StreamingSigned enforces access controlChris Lu1-1/+7
fix https://github.com/chrislusf/seaweedfs/issues/2180
2021-04-08add auth aws signV4Konstantin Lebedev1-2/+2
2020-09-19refactoringChris Lu1-12/+13
2020-02-09add streaming v4Chris Lu1-11/+144
2020-02-09support aclChris Lu1-5/+0
2018-09-03go fmtChris Lu1-2/+2
2018-09-02add s3ChunkedReaderChris Lu1-0/+276
fix https://github.com/chrislusf/seaweedfs/issues/718
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-21 06:03:27 +0000