aboutsummaryrefslogtreecommitdiff
path: root/weed/replication
AgeCommit message (Collapse)AuthorFilesLines
2023-01-20grpc connection to filer add sw-client-id headerchrislu4-3/+7
2023-01-02more solid weed mount (#4089)Chris Lu1-3/+4
* compare chunks by timestamp * fix slab clearing error * fix test compilation * move oldest chunk to sealed, instead of by fullness * lock on fh.entryViewCache * remove verbose logs * revert slat clearing * less logs * less logs * track write and read by timestamp * remove useless logic * add entry lock on file handle release * use mem chunk only, swap file chunk has problems * comment out code that maybe used later * add debug mode to compare data read and write * more efficient readResolvedChunks with linked list * small optimization * fix test compilation * minor fix on writer * add SeparateGarbageChunks * group chunks into sections * turn off debug mode * fix tests * fix tests * tmp enable swap file chunk * Revert "tmp enable swap file chunk" This reverts commit 985137ec472924e4815f258189f6ca9f2168a0a7. * simple refactoring * simple refactoring * do not re-use swap file chunk. Sealed chunks should not be re-used. * comment out debugging facilities * either mem chunk or swap file chunk is fine now * remove orderedMutex as *semaphore.Weighted not found impactful * optimize size calculation for changing large files * optimize performance to avoid going through the long list of chunks * still problems with swap file chunk * rename * tiny optimization * swap file chunk save only successfully read data * fix * enable both mem and swap file chunk * resolve chunks with range * rename * fix chunk interval list * also change file handle chunk group when adding chunks * pick in-active chunk with time-decayed counter * fix compilation * avoid nil with empty fh.entry * refactoring * rename * rename * refactor visible intervals to *list.List * refactor chunkViews to *list.List * add IntervalList for generic interval list * change visible interval to use IntervalList in generics * cahnge chunkViews to *IntervalList[*ChunkView] * use NewFileChunkSection to create * rename variables * refactor * fix renaming leftover * renaming * renaming * add insert interval * interval list adds lock * incrementally add chunks to readers Fixes: 1. set start and stop offset for the value object 2. clone the value object 3. use pointer instead of copy-by-value when passing to interval.Value 4. use insert interval since adding chunk could be out of order * fix tests compilation * fix tests compilation
2022-12-20add a simple file replication progress barchrislu1-2/+21
2022-12-19filer sink retries reading file chunks, skipping missing chunkschrislu2-9/+15
if the file chunk is not available during replication time, the file is skipped
2022-11-15refactor filer_pb.Entry and filer.Entry to use GetChunks()chrislu5-10/+10
for later locking on reading chunks
2022-10-28refactor filer proto chunk variable from mtime to modified_ts_nschrislu1-1/+1
2022-10-11fix invalid memory address or nil pointer dereference on filer.syncchrislu1-1/+1
fix https://github.com/seaweedfs/seaweedfs/issues/3826
2022-10-04fix parameterschrislu1-9/+8
2022-10-04filer.sync: limit concurrency when fetching file chunkschrislu2-9/+13
fix https://github.com/seaweedfs/seaweedfs/issues/3787
2022-09-20filer replication: compare content changes directlychrislu1-4/+0
Fix https://github.com/seaweedfs/seaweedfs/issues/3714 The destination chunks may be empty. For example, the file is updated and the volume is vacuumed. In this case, the sync would miss the old chunks. This is fine. However, the entry would have correct metadata but missing chunks. For this case, the simple metadata comparison would be wrongly skipping data changes, and the file will stay empty unless file content md5 is changed.
2022-09-14refactor: `Directory` readability (#3665)Ryan Russell1-1/+1
2022-09-14docs: `replicte` -> `replicate` (#3664)Ryan Russell1-2/+2
2022-09-14refactor(notification_kafka): `parition` -> `partition` (#3663)Ryan Russell1-2/+2
2022-09-04filer.backup and filer.sync: include headers during backup and syncchrislu1-0/+1
fix https://github.com/seaweedfs/seaweedfs/issues/3532
2022-08-31Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 (#3541)dependabot[bot]1-1/+1
* Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 Bumps [gocloud.dev/pubsub/rabbitpubsub](https://github.com/google/go-cloud) from 0.25.0 to 0.26.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](https://github.com/google/go-cloud/compare/v0.25.0...v0.26.0) --- updated-dependencies: - dependency-name: gocloud.dev/pubsub/rabbitpubsub dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * update code * more code fix Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>
2022-08-27simplifychrislu2-26/+14
2022-08-27clean upchrislu2-159/+1
2022-08-26s3 sink use s3 upload managerchrislu1-44/+32
fix https://github.com/seaweedfs/seaweedfs/issues/3531
2022-08-23remove old raft servers if they don't answer to pings for too long (#3398)askeipx2-3/+4
* remove old raft servers if they don't answer to pings for too long add ping durations as options rename ping fields fix some todos get masters through masterclient raft remove server from leader use raft servers to ping them CheckMastersAlive for hashicorp raft only * prepare blocking ping * pass waitForReady as param * pass waitForReady through all functions * waitForReady works * refactor * remove unneeded params * rollback unneeded changes * fix
2022-08-20filer sink: retryable data chunk uploadingchrislu1-52/+29
2022-08-20cleaner codechrislu1-19/+19
2022-08-19filer.backup: backup small files if the file is saved in filer ↵chrislu6-1/+34
(saveToFilerLimit > 0) fix https://github.com/seaweedfs/seaweedfs/issues/3468
2022-08-17also migrate jsonpbchrislu1-1/+1
2022-08-17move proto packagechrislu4-4/+4
2022-08-04filer prefer volume server in same data center (#3405)Konstantin Lebedev3-1/+19
* initial prefer same data center https://github.com/seaweedfs/seaweedfs/issues/3404 * GetDataCenter * prefer same data center for ReplicationSource * GetDataCenterId * remove glog
2022-07-29move to https://github.com/seaweedfs/seaweedfschrislu18-86/+86
2022-07-27exclude directories to sync on filerKonstantin Lebedev1-4/+13
2022-07-06rabbitpubsub enable durableKonstantin Lebedev1-4/+4
2022-06-29use const multipart uploads folderKonstantin Lebedev1-1/+2
avoid error bucket NotEmpty if multipart uploads folder exist
2022-05-11fix compilationchrislu1-1/+1
2022-05-06filer.sync: pass attributes for mountchrislu1-0/+6
fix https://github.com/chrislusf/seaweedfs/issues/3012
2022-04-21conditionally build elastic, gocdk to reduce binary sizechrislu1-0/+3
2022-02-28filer.backup: fix backing up encrypted chunkschrislu1-1/+1
I have done filer.backup test: replication.toml: [sink.local] enabled = true directory = "/srv/test" ___ system@dat1:/srv/test$ weed filer.backup -filer=app1:8888 -filerProxy I0228 12:39:28 19571 filer_replication.go:129] Configure sink to local I0228 12:39:28 19571 filer_backup.go:98] resuming from 2022-02-28 12:04:20.210984693 +0100 CET I0228 12:39:29 19571 filer_backup.go:113] backup app1:8888 progressed to 2022-02-28 12:04:20.211726749 +0100 CET 0.33/sec system@dat1:/srv/test$ ls -l total 16 drwxr-xr-x 2 system system 4096 Feb 28 12:39 a -rw-r--r-- 1 system system 48 Feb 28 12:39 fu.txt -rw-r--r-- 1 system system 32 Feb 28 12:39 _index.html -rw-r--r-- 1 system system 68 Feb 28 12:39 index.php system@dat1:/srv/test$ cat fu.txt ? ?=?^??`?f^};?{4?Z%?X0=??rV????|"?1??踪~?? system@dat1:/srv/test$ On the active mount on the target server it's: system@app1:/srv/app$ ls -l total 2 drwxrwxr-x 1 system system 0 Feb 28 12:04 a -rw-r--r-- 1 system system 20 Feb 28 12:04 fu.txt -rw-r--r-- 1 system system 4 Feb 28 12:04 _index.html -rw-r--r-- 1 system system 40 Feb 28 12:04 index.php system@app1:/srv/app$ cat fu.txt This is static boy! Filer was started with: weed filer master="app1:9333,app2:9333,app3:9333" -encryptVolumeData It seems like it's still encrypted?
2022-02-27ensure compatibilityelee1-1/+3
2022-02-27set canned acl on replication createelee2-1/+6
2022-02-07filer.sync: fix replicating partially updated filechrislu1-1/+1
Run two servers with volumes and fillers: server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002 server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008 Run Active-Passive filler.sync: filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive Upload file to 11007 port: curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/" If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again: curl "http://localhost:11007/testFacebook/9.xml" EQUALS curl "http://localhost:11001/testFacebook/9.xml" However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file Снимок экрана 2022-02-07 в 14 21 11 This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten. Deleted chunks are not actually deleted on the server to which the file is replicated.
2022-02-04change to util.WriteFilechrislu1-1/+1
2021-12-26use streaming mode for long poll grpc callschrislu4-10/+10
streaming mode would create separate grpc connections for each call. this is to ensure the long poll connections are properly closed.
2021-11-28revertChris Lu1-1/+1
2021-11-28read deleted chunks when replcating dataChris Lu1-1/+1
2021-10-14refactor: move from io/ioutil to io and os packageEng Zer Jun1-3/+3
The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-09-12change server address from string to a typeChris Lu2-2/+2
2021-09-06refactoringChris Lu1-1/+10
2021-09-01go fmtChris Lu2-4/+4
2021-08-25cloud drive: add support for WasabiChris Lu2-0/+2
* disable md5, sha256 checking to avoid reading one chunk twice * single threaded upload to avoid chunk swapping (to be enhanced later)
2021-08-24update azure libraryChris Lu1-3/+3
2021-08-23cloud drive: s3 configurable force path styleChris Lu1-0/+1
2021-08-23do not force path style for better compatibilityChris Lu1-1/+0
2021-08-14go fmtChris Lu1-1/+1
2021-08-12use grpc and jwtChris Lu1-2/+2
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-25 23:29:47 +0000