aboutsummaryrefslogtreecommitdiff
path: root/weed/util/chunk_cache
AgeCommit message (Collapse)AuthorFilesLines
2025-09-24FUSE Mount: enhance disk cache with volume ID and cookie validation (#7269)Chris Lu2-24/+131
* enhance disk cache with volume ID and cookie validation * address comments * fix test * fmt
2024-12-19"golang.org/x/exp/slices" => "slices" and go fmtchrislu1-1/+1
2024-11-21ReadAt may return io.EOF t end of filechrislu1-6/+12
related to https://github.com/seaweedfs/seaweedfs/issues/6219
2024-11-04merge current message queue code changes (#6201)Chris Lu1-0/+20
* listing files to convert to parquet * write parquet files * save logs into parquet files * pass by value * compact logs into parquet format * can skip existing files * refactor * refactor * fix compilation * when no partition found * refactor * add untested parquet file read * rename package * refactor * rename files * remove unused * add merged log read func * parquet wants to know the file size * rewind by time * pass in stop ts * add stop ts * adjust log * minor * adjust log * skip .parquet files when reading message logs * skip non message files * Update subscriber_record.go * send messages * skip message data with only ts * skip non log files * update parquet-go package * ensure a valid record type * add new field to a record type * Update read_parquet_to_log.go * fix parquet file name generation * separating reading parquet and logs * add key field * add skipped logs * use in memory cache * refactor * refactor * refactor * refactor, and change compact log * refactor * rename * refactor * fix format * prefix v to version directory
2024-09-14fix file read crash (#6021)Bruce1-4/+7
2024-09-11Feature limit caching to prescribed number of bytes per file (#6009)Eugeniy E. Mikhailov1-0/+46
* feature: we can check if a fileId is already in the cache We using this to protect cache from adding the same needle to the cache over and over. * fuse mount: Do not start dowloader if needle is already in the cache * added maxFilePartSizeInCache property to ChunkCache If file very large only first maxFilePartSizeInCache bytes are going to be put to the cache (subject to the needle size constrains). * feature: for large files put in cache no more than prescribed number of bytes Before this patch only the first needle of a large file was intended for caching. This patch uses maximum prescribed amount of bytes to be put in cache. This allows to bypass default 2MB maximum for a file part stored in the cache. * added dummy mock methods to satisfy interfaces of ChunkCache
2024-09-10bug fix in the data received from cache processing (#6002)Eugeniy E. Mikhailov1-5/+5
The patch addresses #3745. The cache should return the exact amount of data requested by the buffer. By construction of the cache it is always all requested data range or we have error happening. The old use of minsize miscalculate the requested data size, if non zero offset is requested.
2024-09-04Revert "weed mount, weed dav add option to force cache"chrislu2-6/+4
This reverts commit 7367b976b05bfa69158a60f205dec970c48f50f0.
2024-09-04weed mount, weed dav add option to force cachechrislu2-4/+6
2023-09-25Revert "Revert "Merge branch 'master' of ↵chrislu1-2/+2
https://github.com/seaweedfs/seaweedfs"" This reverts commit 8cb42c39
2023-09-18Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs"chrislu1-2/+2
This reverts commit 2e5aa06026750c99ea283181974d2ccfe5eb0468, reversing changes made to 4d414f54a224142f3f4d934f4af3b5dceb6fec6b.
2023-09-18Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 (#4850)dependabot[bot]1-2/+2
* Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.63.1 to 1.64.0. - [Release notes](https://github.com/rclone/rclone/releases) - [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md) - [Commits](https://github.com/rclone/rclone/compare/v1.63.1...v1.64.0) --- updated-dependencies: - dependency-name: github.com/rclone/rclone dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * API changes * go mod --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>
2022-11-14Lazy loading (#3958)Guo Lei1-1/+1
* types packages is imported more than onece * lazy-loading * fix bugs * fix bugs * fix unit tests * fix test error * rename function * unload ldb after initial startup * Don't load ldb when starting volume server if ldbtimeout is set. * remove uncessary unloadldb * Update weed/command/server.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> * Update weed/command/volume.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: guol-fnst <goul-fnst@fujitsu.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-09-15docs: `panicing` -> `panicking` (#3687)Ryan Russell1-2/+2
Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu4-12/+12
2022-04-18enhancement: replace sort.Slice with slices.SortFunc to reduce reflectionjustin1-6/+4
2022-02-26better control for reader cachingchrislu1-103/+0
2022-02-25chunk cache adds function ReadChunkAtchrislu5-9/+125
2022-02-14test: use `T.TempDir` to create temporary test directoryEng Zer Jun1-4/+1
The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-16add warning errorChris Lu1-1/+1
2021-10-14refactor: move from io/ioutil to io and os packageEng Zer Jun1-2/+1
The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-05-31mount: avoid exception if disk cache is not initializedChris Lu1-0/+4
related to https://github.com/chrislusf/seaweedfs/issues/2102
2021-04-28make reader_at handle random reads more efficiently for FUSENathan Hawkins4-4/+128
2021-04-14truncate is a bit faster to reuse the storageChris Lu1-4/+6
2021-03-22make a local copy of the in memory cached dataChris Lu1-1/+3
2021-02-07fix typo offset.ToAcutalOffset to offset.ToActualOffsetbingoohuang1-2/+2
2021-01-08update ccache versionChris Lu1-1/+1
2020-10-03re-enable caching larger than 16MBChris Lu1-2/+2
revert https://github.com/chrislusf/seaweedfs/commit/62ce85610e2fcd08488ee6026266e617509f6d46
2020-09-27fix testChris Lu1-19/+19
2020-09-27adjust for testChris Lu1-0/+19
2020-09-27adjustChris Lu1-2/+2
2020-09-27skip caching too large chunksChris Lu1-2/+4
2020-09-27more testsChris Lu2-11/+31
2020-09-27fix pre allocated volume sizeChris Lu1-1/+1
2020-09-27refactorChris Lu3-23/+21
adjust for faster test
2020-08-30change log level 5 to 4Chris Lu1-1/+1
2020-08-18adjust logsChris Lu1-1/+1
2020-08-18refactoring to typed SizeChris Lu1-1/+1
Go is amazing with refactoring!
2020-08-17refactoring: use interfaceChris Lu1-0/+5
2020-08-17renameChris Lu2-10/+10
2020-08-17rename variablesChris Lu1-10/+10
2020-08-15adjust logsChris Lu1-1/+1
2020-06-27since we already know the chunk size, no need to iterateChris Lu1-6/+20
2020-06-27incase the memory data is too smallChris Lu1-19/+6
2020-06-26a little bit more efficientChris Lu1-3/+15
2020-06-26error loggingChris Lu3-2/+6
2020-06-25FUSE mount: fix file id written twice?Chris Lu1-1/+1
fix https://github.com/chrislusf/seaweedfs/issues/1373
2020-04-20go fmtChris Lu1-3/+3
2020-04-13tiered cachingChris Lu3-19/+39
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
2020-04-13refactoringChris Lu3-63/+112
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-24 22:17:33 +0000