I was working on indirecting version vectors, and that resulted in some refactoring and improving the existing block indirection stuff. We may or may not end up doing the version vector indirection, but I think these changes are reasonable anyhow and will simplify the diff significantly if we do go there. The main points are: - A bunch of renaming to make the indirection and GC not about "blocks" but about "indirection". - Adding a cutoff so that we don't actually indirect for small block lists. This gets us better performance when handling small files as it cuts out the indirection for quite small loss in space efficiency. - Being paranoid and always recalculating the hash on put. This costs some CPU, but the consequences if a buggy or malicious implementation silently substituted the block list by lying about the hash would be bad.
This commit is contained in:
@@ -148,7 +148,7 @@ func idxck(ldb backend.Backend) (success bool) {
|
||||
}
|
||||
}
|
||||
|
||||
if fi.BlocksHash != nil {
|
||||
if len(fi.Blocks) == 0 && len(fi.BlocksHash) != 0 {
|
||||
key := string(fi.BlocksHash)
|
||||
if _, ok := blocklists[key]; !ok {
|
||||
fmt.Printf("Missing block list for file %q, block list hash %x\n", fi.Name, fi.BlocksHash)
|
||||
|
||||
@@ -119,8 +119,8 @@ are mostly useful for developers. Use with care.
|
||||
"h", "m" and "s" abbreviations for hours minutes and seconds.
|
||||
Valid values are like "720h", "30s", etc.
|
||||
|
||||
STGCBLOCKSEVERY Set to a time interval to override the default database
|
||||
block GC interval of 13 hours. Same format as the
|
||||
STGCINDIRECTEVERY Set to a time interval to override the default database
|
||||
indirection GC interval of 13 hours. Same format as the
|
||||
STRECHECKDBEVERY variable.
|
||||
|
||||
GOMAXPROCS Set the maximum number of CPU cores to use. Defaults to all
|
||||
|
||||
Reference in New Issue
Block a user