felixfoertsch/syncthing
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity
5,596 Commits 9 Branches 684 Tags
0fb2cd52ff8bea00fbfb183dc6df2e2df34eba93
Commit Graph
321 Commits
Author SHA1 Message Date
Simon FreiandGitHub a596e5e2f0 lib/model: Consistent error return values for folder methods on model (#6325) 2020-02-12 07:35:24 +01:00
Jakob Borg 5823e7a5ce lib/model: Sort and group model initialization better (ref #6303) 2020-02-01 08:12:25 +01:00
Jakob BorgandGitHub 55937b61ca lib/model: Add global request limiter (fixes #6302) (#6303) 
This adds a new config with the simple and concise name
maxConcurrentIncomingRequestKiB. This limits how many bytes we have "in
the air" in the form of response data being read and processed.

After some testing I think that not having this limiter is seldom a
great idea and thus I propose a default value of 256 MiB for this new
setting.

I also refactored the folder IO limiter to be a model/folder attribute
instead of a package global.
 2020-02-01 08:02:18 +01:00
Jakob BorgandGitHub d91c4b010b lib/config, lib/model: Limit concurrent pulls (fixes #5914) (#6290) 
Adds a new folder state "Waiting to Sync" in the same vein as the
existing "Waiting to Scan". This vastly improves performances in the
rare cases when there are lots and lots of folders operating.
 2020-01-27 17:31:17 +01:00
Jakob BorgandGitHub 8fc2dfad0c lib/db: Deduplicate block lists in database (fixes #5898) (#6283) 
* lib/db: Deduplicate block lists in database (fixes #5898)

This moves the block list in the database out from being just a field on
the FileInfo to being an object of its own. When putting a FileInfo we
marshal the block list separately and store it keyed by the sha256 of
the marshalled block list. When getting, if we are not doing a
"truncated" get, we do an extra read and unmarshal for the block list.

Old block lists are cleared out by a periodic GC sweep. The alternative
would be to use refcounting, but:

- There is a larger risk of getting that wrong and either dropping a
  block list in error or keeping them around forever.

- It's tricky with our current database, as we don't have dirty reads.
  This means that if we update two FileInfos with identical block lists in
  the same transaction we can't just do read/modify/write for the ref
  counters as we wouldn't see our own first update. See above about
  tracking this and risks about getting it wrong.

GC uses a bloom filter for keys to avoid heavy RAM usage. GC can't run
concurrently with FileInfo updates so there is a new lock around those
operation at the lowlevel.

The end result is a much more compact database, especially for setups
with many peers where files get duplicated many times.

This is per-key-class stats for a large database I'm currently working
with, under the current schema:

```
 0x00:  9138161 items, 870876 KB keys + 7397482 KB data, 95 B +  809 B avg, 1637651 B max
 0x01:   185656 items,  10388 KB keys + 1790909 KB data, 55 B + 9646 B avg,  924525 B max
 0x02:   916890 items,  84795 KB keys +    3667 KB data, 92 B +    4 B avg,     192 B max
 0x03:      384 items,     27 KB keys +       5 KB data, 72 B +   15 B avg,      87 B max
 0x04:     1109 items,     17 KB keys +      17 KB data, 15 B +   15 B avg,      69 B max
 0x06:      383 items,      3 KB keys +       0 KB data,  9 B +    2 B avg,      18 B max
 0x07:      510 items,      4 KB keys +      12 KB data,  9 B +   24 B avg,      41 B max
 0x08:     1349 items,     12 KB keys +      10 KB data,  9 B +    8 B avg,      17 B max
 0x09:      194 items,      0 KB keys +     123 KB data,  5 B +  634 B avg,   11484 B max
 0x0a:        3 items,      0 KB keys +       0 KB data, 14 B +    7 B avg,      30 B max
 0x0b:   181836 items,   2363 KB keys +   10694 KB data, 13 B +   58 B avg,     173 B max
 Total 10426475 items, 968490 KB keys + 9202925 KB data.
```

Note 7.4 GB of data in class 00, total size 9.2 GB. After running the
migration we get this instead:

```
 0x00:  9138161 items, 870876 KB keys + 2611392 KB data, 95 B +  285 B avg,    4788 B max
 0x01:   185656 items,  10388 KB keys + 1790909 KB data, 55 B + 9646 B avg,  924525 B max
 0x02:   916890 items,  84795 KB keys +    3667 KB data, 92 B +    4 B avg,     192 B max
 0x03:      384 items,     27 KB keys +       5 KB data, 72 B +   15 B avg,      87 B max
 0x04:     1109 items,     17 KB keys +      17 KB data, 15 B +   15 B avg,      69 B max
 0x06:      383 items,      3 KB keys +       0 KB data,  9 B +    2 B avg,      18 B max
 0x07:      510 items,      4 KB keys +      12 KB data,  9 B +   24 B avg,      41 B max
 0x09:      194 items,      0 KB keys +     123 KB data,  5 B +  634 B avg,   11484 B max
 0x0a:        3 items,      0 KB keys +       0 KB data, 14 B +   17 B avg,      51 B max
 0x0b:   181836 items,   2363 KB keys +   10694 KB data, 13 B +   58 B avg,     173 B max
 0x0d:    44282 items,   1461 KB keys +   61081 KB data, 33 B + 1379 B avg, 1637399 B max
 Total 10469408 items, 969939 KB keys + 4477905 KB data.
```

Class 00 is now down to 2.6 GB, with just 61 MB added in class 0d.

There will be some additional reads in some cases which theoretically
hurts performance, but this will be more than compensated for by smaller
writes and better compaction.

On my own home setup which just has three devices and a handful of
folders the difference is smaller in absolute numbers of course, but
still less than half the old size:

```
 0x00:  297122 items,  20894 KB keys + 306860 KB data, 70 B + 1032 B avg, 103237 B max
 0x01:  115299 items,   7738 KB keys +  17542 KB data, 67 B +  152 B avg,    419 B max
 0x02: 1430537 items, 121223 KB keys +   5722 KB data, 84 B +    4 B avg,    253 B max
 ...
 Total 1947412 items, 151268 KB keys + 337485 KB data.
```

to:

```
 0x00:  297122 items,  20894 KB keys +  37038 KB data, 70 B +  124 B avg,    520 B max
 0x01:  115299 items,   7738 KB keys +  17542 KB data, 67 B +  152 B avg,    419 B max
 0x02: 1430537 items, 121223 KB keys +   5722 KB data, 84 B +    4 B avg,    253 B max
 ...
 0x0d:   18041 items,    595 KB keys +  71964 KB data, 33 B + 3988 B avg, 101109 B max
 Total 1965447 items, 151863 KB keys + 139628 KB data.
```

* wip

* wip

* wip

* wip
 2020-01-24 08:35:44 +01:00
Simon FreiandJakob Borg 08f0e125ef all: Transactionalize db.FileSet (fixes #5952) (#6239) 2020-01-21 18:23:08 +01:00
Jakob BorgandGitHub 6c8e8f0391 lib/model: Remove legacy handling of symlinks (#6276) 
This hardly seams relevant any more; 0.14.14 is dead since a long time.
 2020-01-19 12:02:20 +01:00
Simon FreiandJakob Borg f56a5545d4 gui, lib/model: Prevent negative sync completion (fixes #4570) (#6248) 2020-01-03 14:07:57 +01:00
Jakob BorgandGitHub be0508cf26 lib/model, lib/protocol: Use error handling to avoid panic on non-started folder (fixes #6174) (#6212) 
This adds error returns to model methods called by the protocol layer.
Returning an error will cause the connection to be torn down as the
message couldn't be handled. Using this to signal that a folder isn't
currently available will then cause a reconnection a few moments later,
when it'll hopefully work better.

Tested manually by running with STRECHECKDBEVERY=0 on a nontrivially
sized setup. This panics reliably before this patch, but just causes a
disconnect/reconnect now.
 2019-12-04 10:46:55 +01:00
Simon FreiandJakob Borg 6fd5e78740 lib: Consistently unsubscribe from config-wrapper (fixes #6133) (#6205) 2019-12-04 07:15:00 +01:00
Jakob BorgandGitHub e82a7e3dfa all: Propagate errors from NamespacedKV (#6203) 
As foretold by the prophecy, "once the database refactor is merged, then
shall appear a request to propagate errors from the store known
throughout the land as the NamedspacedKV, and it shall be good".
 2019-11-30 13:03:24 +01:00
Jakob BorgandAudrius Butkevicius 4e151d380c lib/versioner: Reduce surface area (#6186) 
* lib/versioner: Reduce surface area

This is a refactor while I was anyway rooting around in the versioner.
Instead of exporting every possible implementation and the factory and
letting the caller do whatever, this now encapsulates all that and
exposes a New() that takes a config.VersioningConfiguration.

Given that and that we don't know (from the outside) how a versioner
works or what state it keeps, we now just construct it once per folder
and keep it around. Previously it was recreated for each restore
request.

* unparam

* wip
 2019-11-26 07:39:31 +00:00
Simon FreiandJakob Borg 4d368a37e2 lib/model, lib/protocol: Add contexts sending indexes and download-progress (#6176) 2019-11-25 11:07:36 +01:00
Simon FreiandAudrius Butkevicius cf312abc72 lib: Wrap errors with errors.Wrap instead of fmt.Errorf (#6181) 2019-11-23 15:20:54 +00:00
Simon FreiandGitHub 65d4dd32cb lib/model: Also handle ServeBackground (#6173) 2019-11-22 21:30:16 +01:00
Simon FreiandGitHub 57d668ed1d lib/config: Do introductions in a single config change (#6162) 2019-11-21 08:41:41 +01:00
Simon FreiandGitHub 90d85fd0a2 lib: Replace done channel with contexts in and add names to util services (#6166) 2019-11-21 08:41:15 +01:00
Simon FreiandGitHub 0d14ee4142 lib/model: Don't info log repeat pull errors (#6149) 2019-11-19 09:56:53 +01:00
Simon FreiandJakob Borg 28edf2f5bb lib/model: Keep fmut locked while adding/starting/restarting folders (#6156) 2019-11-18 21:15:26 +01:00
Simon FreiandAudrius Butkevicius cd290d2d05 lib/model: Add initial deviceStatRefs on model creation (fixes #6136) (#6137) 
This is a regression introduced in PR #6005 / commit
f7b2e79fdc
 2019-11-08 11:32:51 +00:00
Simon FreiandGitHub bee7cce081 lib/model: Add folders on start in model (#6135) 2019-11-08 10:56:16 +01:00
Ruslan YevdokymovandAudrius Butkevicius 3b5dff3f34 lib/model: Fix removal of a marker when there are still folders referencing it (#6114) 2019-10-30 15:11:07 +00:00
Simon FreiandJakob Borg f7b2e79fdc lib/model: Use read-locks wherever possible (#6005) 2019-09-12 05:55:23 +01:00
Simon FreiandJakob Borg b677464dfa lib/model: Optimise locking around conn-close and puller states (#5954) 2019-08-16 09:35:19 +02:00
Simon FreiandGitHub b1c74860e8 all: Remove global events.Default (ref #4085) (#5886) 2019-08-15 16:29:37 +02:00
Simon FreiandJakob Borg 6fa02d5081 lib/model: Fix a few more problematic locks (ref #5929) (#5944) 2019-08-13 09:04:43 +02:00
Simon FreiandJakob Borg 86e35f1879 lib/model: Less locking in ClusterConfig (#5943) 2019-08-11 19:30:24 +02:00
Simon FreiandJakob Borg 4a619e74f2 lib/model: Fix incorrect locking (#5939) 2019-08-11 16:10:30 +02:00
Jakob BorgandAudrius Butkevicius 669bcb748f lib/config, lib/model: Don't save on every pending folder/device update (fixes #5888) (#5890) 
Wrapper methods generally don't save by themselves.
 2019-07-27 11:05:00 +01:00
Simon FreiandJakob Borg 35f40e9a58 lib/model: Create new file-set after stopping folder (fixes #5882) (#5883) 2019-07-23 20:39:25 +02:00
Simon FreiandGitHub 6f08162376 lib/model: Remove incorrect/useless panics (#5881) 2019-07-23 10:51:16 +02:00
Simon FreiandJakob Borg b5b54ff057 lib/model: No watch-error on missing folder (fixes #5833) (#5876) 2019-07-19 19:41:16 +02:00
Simon FreiandJakob Borg 1cb55904bc lib/model: Prevent panic in NeedFolderFiles (fixes #5872) (#5875) 2019-07-19 19:39:52 +02:00
Simon FreiandJakob Borg 2b622d0774 lib/model: Close conn on dev pause (fixes #5873) (#5874) 2019-07-19 19:37:29 +02:00
Simon FreiandJakob Borg 82b70b9fae lib/model, lib/protocol: Track closing connections (fixes #5828) (#5829) 2019-07-14 11:03:55 +02:00
Simon FreiandJakob Borg 20c8dbd9ed lib/model: Fix integer conversion (fixes #5837) (#5851) 2019-07-12 16:37:12 +02:00
Simon FreiandJakob Borg 9fef1552fc lib/db, lib/model: Remove folder info from panics (ref #5839) (#5840) 2019-07-10 10:57:49 +02:00
Simon FreiandGitHub ba056578ec lib: Add util.Service as suture.Service template (fixes #5801) (#5806) 2019-07-09 11:40:30 +02:00
Simon FreiandJakob Borg 863fe23347 gui, lib/model: Fix download progress accounting (fixes #5811) (#5815) 2019-06-30 09:23:47 +02:00
Simon FreiandJakob Borg 1cf352a722 lib/model: NewFileSet outside fmut (#5818) 2019-06-29 08:49:30 +02:00
Simon FreiandAudrius Butkevicius 3c7e7e971d lib/model: Make jobQueue.Jobs paginated (fixes #5754) (#5804) 
* lib/model: Make jobQueue.Jobs paginated (fixes #5754)

* fix, no test yet

* add test
 2019-06-27 19:25:38 +01:00
Simon FreiandJakob Borg 13a746e0fb lib/model: Prevent nil deref if folder stopped (fixes #5780) (#5778) 2019-06-11 11:48:51 +02:00
Audrius ButkeviciusandJakob Borg 21f50e2f8f lib/versioner: Use mtime for version cleanup (fixes #5765) (#5769) 2019-06-11 09:16:55 +02:00
Simon FreiandGitHub 1054ce9354 lib/model: Refactor sending indexes as suture service (#5757) 2019-06-10 13:27:22 +02:00
Simon FreiandGitHub 64518b0f7e lib/model: Close connections when model is stopped (#5733) 2019-05-25 16:00:32 +02:00
André ColombandJakob Borg e4956358fb lib/model: Remove superfluous check for IndexID in remote ClusterConfig (#5717) 
The check in ClusterConfig() when iterating through announced devices
in a folder explicitly skips entries without a non-zero IndexID.
Therefore, the check for IndexID == 0 just below will never be true
and the intended cleanup of local index data will not happen.

Plainly remove that check to make the intended case distinction work.
 2019-05-12 21:17:55 +02:00
Simon FreiandAudrius Butkevicius 5fa8467756 lib/model: Consistent use of locks (#5684) 2019-05-02 18:55:39 +01:00
Simon FreiandGitHub 5954b105cd lib/model: Let fakeConnection call Model.Closed on close (#5682) 2019-05-02 14:24:55 +02:00
Simon FreiandGitHub defc5dca65 lib/model: Use correct lock (#5683) 2019-05-02 14:09:42 +02:00
Audrius ButkeviciusandGitHub 0ca1f26ff8 lib/versioner: Restore for all versioners, cross-device support (#5514) 
* lib/versioner: Restore for all versioners, cross-device support

Fixes #4631
Fixes #4586
Fixes #1634
Fixes #5338
Fixes #5419
 2019-04-28 23:30:16 +01:00