aboutsummaryrefslogtreecommitdiff
path: root/go/storage/volume_vacuum.go
AgeCommit message (Collapse)AuthorFilesLines
2016-06-02directory structure change to work with glideChris Lu1-93/+0
glide has its own requirements. My previous workaround caused me some code checkin errors. Need to fix this.
2016-05-09close current needle map when switchingChris Lu1-0/+1
2015-07-11add loggingchrislusf1-0/+1
2015-05-23Rename variableschrislusf1-2/+2
2015-05-08Increase compact revision during each compactionchrislusf1-0/+1
2015-04-16Change all chrislusf/weed-fs links to point to chrislu/seaweedfsStuart P. Bentley1-1/+1
2015-03-29Add boltdb for volume needle mapchrislusf1-20/+21
boltdb is fairly slow to write, about 6 minutes for recreating index for 1553934 files. Boltdb loads 1,553,934 x 16 = 24,862,944bytes from disk, and generate the boltdb as large as 134,217,728 bytes in 6 minutes. To compare, for leveldb, it recreates index in leveldb as large as 27,188,148 bytes in 8 seconds. For in memory version, it loads the index in To test the memory consumption, the leveldb or boltdb index are created. And the server is restarted. Using the benchmark tool to read lots of files. There are 7 volumes in benchmark collection, each with about 1553K files. For leveldb, the memory starts at 142,884KB, and stays at 179,340KB. For boltdb, the memory starts at 73,756KB, and stays at 144,564KB. For in-memory, the memory starts at 368,152KB, and stays at 448,032KB.
2015-03-27add leveldb support for needle mapchrislusf1-2/+2
This supposedly should reduce memory consumption. However, for tests with millions of, this shows consuming more memories. Need to see whether this will work out. If not, later boltdb will be tested.
2014-10-26formatting code by: goimports -w=true .Chris Lu1-1/+2
2014-09-25use github.com/chrislusf instead of github.com/aszxqwwyy1-1/+1
2014-09-25use github.com/aszxqw instead of code.google.com/pwyy1-1/+1
2014-09-20add TTL supportChris Lu1-4/+10
The volume TTL and file TTL are not necessarily the same. as long as file TTL is smaller than volume TTL, it'll be fine. volume TTL is used when assigning file id, e.g. http://.../dir/assign?ttl=3h file TTL is used when uploading
2014-05-31Reduce memory usage for "weed fix"Chris Lu1-1/+1
2014-05-26fix compilation errorChris Lu1-1/+1
2014-05-26Adjust "weed server" parameters.Chris Lu1-1/+8
2014-05-19Refactor out volume vacuum.Chris Lu1-0/+40
2014-05-19Refactor out volume vacuum.Chris Lu1-0/+35