release-nightly
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Marcus B Spencer
|
655b4568c1 |
fix(fs): only apply case option to fakefs in stress test (#10440)
Fixes a regression introduced in #10439, mentioned in [a comment](https://github.com/syncthing/syncthing/pull/10439#issuecomment-3436515824) made by @imsodin: > That might not be the greatest way to do this, but nevertheless it afaik means that the benchmarks now do case checking once when it shouldn't happen at all, and twice otherwise. Benchmarks do approximately as well as before the regression, and I think most of them are random chance: ``` │ ../oldold.txt │ ../new.txt │ │ sec/op │ sec/op vs base │ WalkCaseFakeFS100k/rawfs-8 654.6m ± 1% 652.6m ± 3% ~ (p=0.971 n=10) WalkCaseFakeFS100k/casefs-8 1.049 ± 2% 1.071 ± 3% ~ (p=0.190 n=10) WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 1.053 ± 3% 1.081 ± 5% ~ (p=0.165 n=10) geomean 897.7m 910.8m +1.46% │ ../oldold.txt │ ../new.txt │ │ B/entry │ B/entry vs base │ WalkCaseFakeFS100k/rawfs-8 1.274Ki ± 0% 1.274Ki ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-8 1.771Ki ± 0% 1.771Ki ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 1.772Ki ± 0% 1.772Ki ± 0% ~ (p=1.000 n=10) ¹ geomean 1.587Ki 1.587Ki +0.00% ¹ all samples are equal │ ../oldold.txt │ ../new.txt │ │ DirNames/entry │ DirNames/entry vs base │ WalkCaseFakeFS100k/rawfs-8 512.5m ± 0% 512.5m ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-8 1.025 ± 0% 1.025 ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 1.025 ± 0% 1.025 ± 0% ~ (p=1.000 n=10) ¹ geomean 813.5m 813.5m +0.00% ¹ all samples are equal │ ../oldold.txt │ ../new.txt │ │ DirNames/op │ DirNames/op vs base │ WalkCaseFakeFS100k/rawfs-8 51.25k ± 0% 51.25k ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-8 102.5k ± 0% 102.5k ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 102.5k ± 0% 102.5k ± 0% ~ (p=1.000 n=10) ¹ geomean 81.35k 81.35k +0.00% ¹ all samples are equal │ ../oldold.txt │ ../new.txt │ │ Lstat/entry │ Lstat/entry vs base │ WalkCaseFakeFS100k/rawfs-8 5.535 ± 0% 5.535 ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-8 5.535 ± 0% 5.535 ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 5.540 ± 0% 5.540 ± 0% ~ (p=1.000 n=10) ¹ geomean 5.537 5.537 +0.00% ¹ all samples are equal │ ../oldold.txt │ ../new.txt │ │ Lstat/op │ Lstat/op vs base │ WalkCaseFakeFS100k/rawfs-8 553.5k ± 0% 553.5k ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-8 553.5k ± 0% 553.5k ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 554.0k ± 0% 554.0k ± 0% ~ (p=1.000 n=10) ¹ geomean 553.7k 553.7k +0.00% ¹ all samples are equal │ ../oldold.txt │ ../new.txt │ │ allocs/entry │ allocs/entry vs base │ WalkCaseFakeFS100k/rawfs-8 19.00 ± 0% 19.00 ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-8 35.35 ± 0% 35.35 ± 0% ~ (p=1.000 n=10) ¹ WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 35.38 ± 0% 35.38 ± 0% ~ (p=1.000 n=10) ¹ geomean 28.75 28.75 +0.00% ¹ all samples are equal │ ../oldold.txt │ ../new.txt │ │ sec/entry │ sec/entry vs base │ WalkCaseFakeFS100k/rawfs-8 4.328µ ± 1% 4.315µ ± 3% ~ (p=0.971 n=10) WalkCaseFakeFS100k/casefs-8 6.936µ ± 2% 7.082µ ± 3% ~ (p=0.171 n=10) WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 6.965µ ± 3% 7.147µ ± 5% ~ (p=0.165 n=10) geomean 5.935µ 6.022µ +1.46% │ ../oldold.txt │ ../new.txt │ │ B/op │ B/op vs base │ WalkCaseFakeFS100k/rawfs-8 188.3Mi ± 0% 188.3Mi ± 0% -0.00% (p=0.006 n=10) WalkCaseFakeFS100k/casefs-8 261.5Mi ± 0% 261.5Mi ± 0% ~ (p=0.142 n=10) WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 261.7Mi ± 0% 261.7Mi ± 0% ~ (p=0.315 n=10) geomean 234.4Mi 234.4Mi -0.00% │ ../oldold.txt │ ../new.txt │ │ allocs/op │ allocs/op vs base │ WalkCaseFakeFS100k/rawfs-8 2.873M ± 0% 2.873M ± 0% -0.00% (p=0.026 n=10) WalkCaseFakeFS100k/casefs-8 5.346M ± 0% 5.346M ± 0% ~ (p=0.136 n=10) WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 5.351M ± 0% 5.351M ± 0% ~ (p=0.305 n=10) geomean 4.348M 4.348M -0.00% ``` Signed-off-by: Marcus B Spencer <marcus@marcusspencer.us> |
||
|
Marcus B Spencer
|
c6a887865f |
fix(fs): apply case option to fakefs in casefs tests (#10439)
Required for the casefs tests/benchmarks to test the casefs.
Benchmarks do significantly worse (as expected).
```
│ ../old.txt │ ../new.txt │
│ sec/op │ sec/op vs base │
WalkCaseFakeFS100k/rawfs-8 626.5m ± 5% 993.4m ± 1% +58.56% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-8 1.011 ± 1% 1.425 ± 1% +40.94% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 1.014 ± 2% 1.439 ± 1% +41.97% (p=0.002 n=6)
geomean 862.9m 1.268 +46.94%
│ ../old.txt │ ../new.txt │
│ B/entry │ B/entry vs base │
WalkCaseFakeFS100k/rawfs-8 1.274Ki ± 0% 1.766Ki ± 0% +38.54% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-8 1.771Ki ± 0% 2.354Ki ± 0% +32.98% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 1.772Ki ± 0% 2.356Ki ± 0% +32.95% (p=0.002 n=6)
geomean 1.587Ki 2.140Ki +34.80%
│ ../old.txt │ ../new.txt │
│ DirNames/entry │ DirNames/entry vs base │
WalkCaseFakeFS100k/rawfs-8 512.5m ± 0% 1025.0m ± 0% +100.00% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-8 1.025 ± 0% 1.537 ± 0% +49.95% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 1.025 ± 0% 1.537 ± 0% +49.95% (p=0.002 n=6)
geomean 813.5m 1.343 +65.06%
│ ../old.txt │ ../new.txt │
│ DirNames/op │ DirNames/op vs base │
WalkCaseFakeFS100k/rawfs-8 51.25k ± 0% 102.49k ± 0% +100.00% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-8 102.5k ± 0% 153.7k ± 0% +50.00% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 102.5k ± 0% 153.7k ± 0% +50.00% (p=0.002 n=6)
geomean 81.35k 134.3k +65.10%
│ ../old.txt │ ../new.txt │
│ allocs/entry │ allocs/entry vs base │
WalkCaseFakeFS100k/rawfs-8 19.00 ± 0% 35.35 ± 0% +86.05% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-8 35.35 ± 0% 54.40 ± 0% +53.89% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 35.38 ± 0% 54.46 ± 0% +53.93% (p=0.002 n=6)
geomean 28.75 47.14 +63.95%
│ ../old.txt │ ../new.txt │
│ sec/entry │ sec/entry vs base │
WalkCaseFakeFS100k/rawfs-8 4.143µ ± 5% 6.568µ ± 1% +58.55% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-8 6.686µ ± 1% 9.424µ ± 1% +40.95% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 6.703µ ± 2% 9.517µ ± 1% +41.97% (p=0.002 n=6)
geomean 5.705µ 8.383µ +46.94%
│ ../old.txt │ ../new.txt │
│ B/op │ B/op vs base │
WalkCaseFakeFS100k/rawfs-8 188.3Mi ± 0% 260.8Mi ± 0% +38.51% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-8 261.5Mi ± 0% 347.7Mi ± 0% +32.98% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 261.7Mi ± 0% 348.0Mi ± 0% +32.96% (p=0.002 n=6)
geomean 234.4Mi 316.0Mi +34.79%
│ ../old.txt │ ../new.txt │
│ allocs/op │ allocs/op vs base │
WalkCaseFakeFS100k/rawfs-8 2.873M ± 0% 5.346M ± 0% +86.04% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-8 5.346M ± 0% 8.228M ± 0% +53.91% (p=0.002 n=6)
WalkCaseFakeFS100k/casefs-otherOpEvery1000-8 5.351M ± 0% 8.236M ± 0% +53.92% (p=0.002 n=6)
geomean 4.348M 7.129M +63.96%
```
Signed-off-by: Marcus B Spencer <marcus@marcusspencer.us>
|
||
  Jakob BorgandRoss Smith II
|
836045ee87 |
feat: switch logging framework (#10220)
This updates our logging framework from legacy freetext strings using the `log` package to structured log entries using `log/slog`. I have updated all INFO or higher level entries, but not yet DEBUG (😓)... So, at a high level: There is a slight change in log levels, effectively adding a new warning level: - DEBUG is still debug (ideally not for users but developers, though this is something we need to work on) - INFO is still info, though I've added more data here, effectively making Syncthing more verbose by default (more on this below) - WARNING is a new log level that is different from the _old_ WARNING (more below) - ERROR is what was WARNING before -- problems that must be dealt with, and also bubbled as a popup in the GUI. A new feature is that the logging level can be set per package to something other than just debug or info, and hence I feel that we can add a bit more things into INFO while moving some (in fact, most) current INFO level warnings into WARNING. For example, I think it's justified to get a log of synced files in INFO and sync failures in WARNING. These are things that have historically been tricky to debug properly, and having more information by default will be useful to many, while still making it possible get close to told level of inscrutability by setting the log level to WARNING. I'd like to get to a stage where DEBUG is never necessary to just figure out what's going on, as opposed to trying to narrow down a likely bug. Code wise: - Our logging object, generally known as `l` in each package, is now a new adapter object that provides the old API on top of the newer one. (This should go away once all old log entries are migrated.) This is only for `l.Debugln` and `l.Debugf`. - There is a new level tracker that keeps the log level for each package. - There is a nested setup of handlers, since the structure mandated by `log/slog` is slightly convoluted (imho). We do this because we need to do formatting at a "medium" level internally so we can buffer log lines in text format but with separate timestamp and log level for the API/GUI to consume. - The `debug` API call becomes a `loglevels` API call, which can set the log level to `DEBUG`, `INFO`, `WARNING` or `ERROR` per package. The GUI is updated to handle this. - Our custom `sync` package provided some debugging of mutexes quite strongly integrated into the old logging framework, only turned on when `STTRACE` was set to certain values at startup, etc. It's been a long time since this has been useful; I removed it. - The `STTRACE` env var remains and can be used the same way as before, while additionally permitting specific log levels to be specified, `STTRACE=model:WARN,scanner:DEBUG`. - There is a new command line option `--log-level=INFO` to set the default log level. - The command line options `--log-flags` and `--verbose` go away, but are currently retained as hidden & ignored options since we set them by default in some of our startup examples and Syncthing would otherwise fail to start. Sample format messages: ``` 2009-02-13 23:31:30 INF A basic info line (attr1="val with spaces" attr2=2 attr3="val\"quote" a=a log.pkg=slogutil) 2009-02-13 23:31:30 INF An info line with grouped values (attr1=val1 foo.attr2=2 foo.bar.attr3=3 a=a log.pkg=slogutil) 2009-02-13 23:31:30 INF An info line with grouped values via logger (foo.attr1=val1 foo.attr2=2 a=a log.pkg=slogutil) 2009-02-13 23:31:30 INF An info line with nested grouped values via logger (bar.foo.attr1=val1 bar.foo.attr2=2 a=a log.pkg=slogutil) 2009-02-13 23:31:30 WRN A warning entry (a=a log.pkg=slogutil) 2009-02-13 23:31:30 ERR An error (a=a log.pkg=slogutil) ``` --------- Co-authored-by: Ross Smith II <ross@smithii.com> |
||
 Marcel Meyer
|
48b757cac1 |
refactor: use slices package for sort (#10132)
The sort package is still used in places that were not trivial to change. Since Go 1.21 slices package can be uswed for sort. See https://go.dev/doc/go1.21#slices ### Purpose Make some progress with the migration to a more up-to-date syntax. |
||
 Tommy van der Vorst
|
d23cd197e1 |
chore(fs): changes to allow Filesystem to be implemented externally (#10040)
### Purpose The `fs.Filesystem` interface contains two parts that cannot be implemented externally because they are private: * `filesystemWrapperType`: this PR changes `unwrapFilesystem` to downcast to a specific concrete type * `underlying`: this PR simply moves it to an unexported interface ### Testing Regular tests pass. |
||
|
Jakob Borg
|
e82ed6e3d3 |
style: gofumpt all the things (#9829)
Literally `gofumpt -w .` from the top level dir. Guaranteed to be minor style changes only and nothing else. @imsodin per request? |
||
 Simon Frei
|
ac8b3342ac |
chore(fs): only cache the cache for case FS, not the entire FS (#9701)
This would have addressed a recent issue that arose when re-ordering our "filesystem layers". Specifically moving the caseFilesystem to the outermost layer. The previous cache included the filesystem, and as such all the layers below. This isn't desirable (to put it mildly), as you can create different variants of filesystems with different layers for the same path and options. Concretely this did happen with the mtime layer, which isn't always present. A test for the mtime related breakage was added in #9687, and I intend to redo the caseFilesystem reordering after this. Ref: #9677 Followup to: #9687 |
||
 luzpaz
|
837ffcfab5 |
all: Fix various user-facing and non-user-facing typos (#8509)
Found via `codespell -q 3 -S lang,./gui/default/vendor -L benchs,bu,inflight,ro` |
||
 Eng Zer Jun
|
bc27aa12cd |
all: use T.TempDir to create temporary test directory (#8280)
This commit replaces `os.MkdirTemp` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `os.MkdirTemp`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
|
||
|
Simon Frei
|
db72579f0e | lib: Get rid of buggy filesystem wrapping (#8257) | ||
 Audrius Butkevicius
|
87a0eecc31 |
lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file (#7624)
* lib/fs, lib/api, lib/model: Expose mtime remappings as part of /db/file * Fix wrong error returned by CLI * Gofmt * Better names * Review comments * Review comments |
||
|
Simon Frei
|
e19d6e993d | lib/fs: Cache all real-case results (fixes #7270) (#7286) | ||
|
Jakob Borg
|
780fb3bac1 |
lib/fs: More efficient casefs cache (#6974)
This changes the cache to cache less things, yet retain the required efficiency for our walk usecase. This uses less memory. Specifically, instead of keeping result and child caches for each path level, only keep a single cached child. In practice our operations are depth-first, or almost depth-first, and then we retain the same hit ratio for a smaller cache size. I improved the benchmark so that it counts the Lstat and DirNames operations performed, and they do not change significantly. The amount of allocated memory is reduced by 20% and the walk itself is actually slightly faster. This also removes the clear based on number of cached names (as that is not a thing any more) and the timer based clear (which was unused). This means we'll retain the last cache state forever until it's cleared by a write operation, but we did that before too and that state is now a lot smaller... The overhead compared to not using a casefs, for our typical "double walk" (walk the tree then stat everything again) is 2x the dirnames we would otherwise call, and no overhead on the stats (unchanged from old implementation) ``` name old time/op new time/op delta WalkCaseFakeFS100k/rawfs-8 306ms ± 1% 305ms ± 2% ~ (p=0.182 n=9+10) WalkCaseFakeFS100k/casefs-8 579ms ± 5% 557ms ± 1% -3.77% (p=0.000 n=10+10) name old B/entry new B/entry delta WalkCaseFakeFS100k/rawfs-8 590 ± 0% 590 ± 0% ~ (all equal) WalkCaseFakeFS100k/casefs-8 1.09k ± 0% 0.87k ± 0% -19.98% (p=0.000 n=10+10) name old DirNames/entry new DirNames/entry delta WalkCaseFakeFS100k/rawfs-8 0.51 ± 0% 0.51 ± 0% ~ (all equal) WalkCaseFakeFS100k/casefs-8 1.02 ± 0% 1.02 ± 0% ~ (all equal) name old DirNames/op new DirNames/op delta WalkCaseFakeFS100k/rawfs-8 51.2k ± 0% 51.2k ± 0% ~ (all equal) WalkCaseFakeFS100k/casefs-8 102k ± 0% 102k ± 0% ~ (all equal) name old Lstat/entry new Lstat/entry delta WalkCaseFakeFS100k/rawfs-8 3.02 ± 0% 3.02 ± 0% ~ (all equal) WalkCaseFakeFS100k/casefs-8 3.02 ± 0% 3.02 ± 0% ~ (all equal) name old Lstat/op new Lstat/op delta WalkCaseFakeFS100k/rawfs-8 302k ± 0% 302k ± 0% ~ (all equal) WalkCaseFakeFS100k/casefs-8 302k ± 0% 302k ± 0% ~ (all equal) name old allocs/entry new allocs/entry delta WalkCaseFakeFS100k/rawfs-8 15.7 ± 0% 15.7 ± 0% ~ (all equal) WalkCaseFakeFS100k/casefs-8 27.5 ± 0% 26.1 ± 0% -5.09% (p=0.000 n=10+10) name old ns/entry new ns/entry delta WalkCaseFakeFS100k/rawfs-8 2.02k ± 1% 2.02k ± 2% ~ (p=0.163 n=9+10) WalkCaseFakeFS100k/casefs-8 3.83k ± 5% 3.68k ± 1% -3.77% (p=0.000 n=10+10) name old alloc/op new alloc/op delta WalkCaseFakeFS100k/rawfs-8 89.2MB ± 0% 89.2MB ± 0% ~ (p=0.364 n=9+10) WalkCaseFakeFS100k/casefs-8 164MB ± 0% 131MB ± 0% -19.97% (p=0.000 n=10+10) name old allocs/op new allocs/op delta WalkCaseFakeFS100k/rawfs-8 2.38M ± 0% 2.38M ± 0% ~ (all equal) WalkCaseFakeFS100k/casefs-8 4.16M ± 0% 3.95M ± 0% -5.05% (p=0.000 n=10+10) ``` |
||
|
Simon FreiandJakob Borg
|
932d8c69de |
lib/fs: Properly handle case insensitive systems (fixes #1787, fixes #2739, fixes #5708)
With this change we emulate a case sensitive filesystem on top of insensitive filesystems. This means we correctly pick up case-only renames and throw a case conflict error when there would be multiple files differing only in case. This safety check has a small performance hit (about 20% more filesystem operations when scanning for changes). The new advanced folder option `caseSensitiveFS` can be used to disable the safety checks, retaining the previous behavior on systems known to be fully case sensitive. Co-authored-by: Jakob Borg <jakob@kastelo.net> |