Cleanup after #9275.
This renames `fmut` -> `mut`, removes the deadlock detector and
associated plumbing, renames some things from `...PRLocked` to
`...RLocked` and similar, and updates comments.
Apart from the removal of the deadlock detection machinery, no
functional code changes... i.e. almost 100% diff noise, have fun
reviewing.
I'm tired of the fmut/pmut shenanigans. This consolidates both under one
lock; I'm not convinced there are any significant performance
differences with this approach since we're literally just protecting map
juggling...
- The locking goes away when we were already under an appropriate fmut
lock.
- Where we had fmut.RLock()+pmut.Lock() it gets upgraded to an
fmut.Lock().
- Otherwise s/pmut/fmut/.
In order to avoid diff noise for an important change I did not do the
following cleanups, which will be filed in a PR after this one, if
accepted:
- Renaming fmut to just mut
- Renaming methods that refer to being "PRLocked" and stuff like that
- Removing the no longer relevant deadlock detector
- Comments referring to pmut and locking sequences...
`syncthing cli` subcommand was using urfave/cli as the command parser.
This PR replace it with kong, which the main command uses.
Some help texts and error message format are changed. Other than that,
all the command usage and logic remains unchanged.
There's only one place which still uses urfave/cli, which is `syncthing
cli config`, because it uses some magic to dynamically build commands
from struct reflects. I used kong's `passthrough:""` tag to pass any
argument following `syncthing cli config` to urfave/cli parser.
This PR also fixes#9041
---------
Co-authored-by: Jakob Borg <jakob@kastelo.net>
This pull request allows syncthing to request an IPv6
[pinhole](https://en.wikipedia.org/wiki/Firewall_pinhole), addressing
issue #7406. This helps users who prefer to use IPv6 for hosting their
services or are forced to do so because of
[CGNAT](https://en.wikipedia.org/wiki/Carrier-grade_NAT). Otherwise,
such users would have to configure their firewall manually to allow
syncthing traffic to pass through while IPv4 users can use UPnP to take
care of network configuration already.
### Testing
I have tested this in a virtual machine setup with miniupnpd running on
the virtualized router. It successfully added an IPv6 pinhole when used
with IPv6 only, an IPv4 port mapping when used with IPv4 only and both
when dual-stack (IPv4 and IPv6) is used.
Automated tests could be added for SOAP responses from the router but
automatically testing this with a real network is likely infeasible.
### Documentation
https://docs.syncthing.net/users/firewall.html could be updated to
mention the fact that UPnP now works with IPv6, although this change is
more "behind the scenes".
---------
Co-authored-by: Simon Frei <freisim93@gmail.com>
Co-authored-by: bt90 <btom1990@googlemail.com>
Co-authored-by: André Colomb <github.com@andre.colomb.de>
gui: Show folder/device status on small screens
On larger screens, folder and device status is shown in a textual form
directly next to folder and device titles. However, on small screens,
only icons are currently shown, which may be ambiguous to new users, who
cannot possibly know what a specific icon means (see [1]). Thus, on
small screens only, display a new entry in folder/device info that
contains the same textual information that is shown in the title on
larger screens.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Co-authored-by: André Colomb <src@andre.colomb.de>
Before introducing the service map and using it for folder runners, the
entries in folderCfgs and folderRunners for the same key/folder were
removed under a single lock. Stopping the folder happens separately
before that with just the read lock. Now with the service map stopping
the folder and removing it from the map is a single operation. And that
still happens with just a read-lock. However even with a full lock it’s
still problematic: After the folder stopped, the runner isn’t present
anymore while the folder-config still is and sais the folder isn't
paused.
The index handler in turn looks at the folder config that is not paused,
thus assumes the runner has to be present -> nil deref on the runner.
A better solution might be to push most of these fmut maps into the
folder - they anyway are needed in there. Then there's just a single
map/source of info that's necessarily consistent. That's quite a bit of
work though, and probably/likely there will be corner cases there too.
This reverts commit e477777f49.
In principle, we could have stayed with `~1.21.1`, but `check-latest:
true` apparently checks some cache/manifest/something that is only
periodically refreshed and isn't aware of 1.21.5 yet. So update the
constraints to force an upgrade.
Also the infrastructure images weren't actually using the constraint
since there was no `setup-go` action...
This reduces allocations, in number and in size, while getting extended
attributes. This is mostly noticable when there is a large number of new
files to scan and we're running with the default scanProgressInterval --
then a queue of files is built in-memory, and this queue includes
extended attributes as part of file metadata. (Arguable it shouldn't,
but that's a more difficult and involved change.)
With 1M files to scan, each with one extended attribute, current peak
memory usage looks like this:
Showing nodes accounting for 1425.30MB, 98.19% of 1451.64MB total
Dropped 1435 nodes (cum <= 7.26MB)
Showing top 10 nodes out of 54
flat flat% sum% cum cum%
976.56MB 67.27% 67.27% 976.56MB 67.27%
github.com/syncthing/syncthing/lib/fs.getXattr
305.44MB 21.04% 88.31% 305.44MB 21.04%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.78MB 3.15% 91.47% 1045.23MB 72.00%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
22.89MB 1.58% 93.04% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 1.58% 94.62% 22.89MB 1.58%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
16MB 1.10% 95.72% 16.01MB 1.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
After the change, it's this:
Showing nodes accounting for 502.32MB, 95.70% of 524.88MB total
Dropped 1400 nodes (cum <= 2.62MB)
Showing top 10 nodes out of 91
flat flat% sum% cum cum%
305.43MB 58.19% 58.19% 305.43MB 58.19%
github.com/syncthing/syncthing/lib/scanner.(*walker).walk.func1
45.79MB 8.72% 66.91% 68.68MB 13.09%
github.com/syncthing/syncthing/lib/fs.(*BasicFilesystem).GetXattr
32MB 6.10% 73.01% 32.01MB 6.10%
github.com/syndtr/goleveldb/leveldb/memdb.New
22.89MB 4.36% 77.37% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/fs.listXattr
22.89MB 4.36% 81.73% 22.89MB 4.36%
github.com/syncthing/syncthing/lib/protocol.(*PlatformData).SetXattrs
15.35MB 2.92% 84.66% 15.36MB 2.93%
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).Get
15.28MB 2.91% 87.57% 15.28MB 2.91% strings.(*Builder).grow
(The usage for xattrs is reduced from 976 MB to 68 MB)
LastSeen for a device was only updated when they connected. This now
updates it when they disconnect, so that we remember the last time we
actually saw them. When asking the API for current stats, currently
connected devices get a last seen value of the current time.
The string is currently hard-coded in English, so allow to translate it
into other languages. It appears mainly in the browser title bar before
logging in into the GUI or when the GUI is still loading.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
This adds our short device ID to the basic auth realm. This has at least
two consequences:
- It is different from what's presented by another device on the same
address (e.g., if I use SSH forwards to different dives on the same
local address), preventing credentials for one from being sent to
another.
- It is different from what we did previously, meaning we avoid cached
credentials from old versions interfering with the new login flow.
I don't *think* there should be things that depend on our precise realm
string, so this shouldn't break any existing setups...
Sneakily this also changes the session cookie and CSRF name, because I
think `id.Short().String()` is nicer than `id.String()[:5]` and the
short ID is two characters longer. That's also not a problem...
### Purpose
The OCI image spec specifies well-defined
[annotations](https://github.com/opencontainers/image-spec/blob/main/annotations.md)
that can be added to images.
Theses annotations can then be used by other tools to gather more
information of an image.
This PR adds the `org.opencontainers.image.source` to allow tools such
as [renovate](https://github.com/renovatebot/renovate) to find the
release notes of a give version.
~~I've only done this change for `Dockerfile`. Should I also add the
label to the other dockerfiles?~~
I've now added the source annotations to all `Dockerfile`s & action
workflows.
### Testing
None, change was done by following the [renovate
documentation](https://docs.renovatebot.com/modules/datasource/docker/).
Following up on #9192, this makes use of the new mechanism for the theme
names.
The dummy string added for testing is removed again here. All
translations are updated to the new nested syntax, except Chinese
(zh-HK) where the string weren't actually translated.
Some translations, especially single words or other short
labels for buttons and the like, may not be transferable between
contexts even if they happen to be equal in English. In these cases,
setting an explicit translation ID is important for context separation.
Angular Translate also supports nested JSON in translation tables,
addressed using `.` as namespace separator; this enhancement makes use
of this when extracting translation with an explicit translation ID.
We use `env.GO_VERSION` as cache key for the build cache, but this is
nowadays typically something like `~1.21.1` which doesn't change when
1.21.2, 1.21.3 etc are released, making the cache fairly useless as
everything gets rebuilt. This re-sets the `GO_VERSION` variable after
installing Go so that it contains the actual installed version.
### Purpose
Treat X-Forwarded-For as a comma-separated string to prevent nil IP being returned by the Discovery Server
### Testing
Unit Tests implemented
Testing with a Discovery Client can be done as follows:
```
A simple example to replicate this entails running Discovery with HTTP, use Nginx as a reverse proxy and hardcode (as an example) a list of IPs in the X-Forwarded-For header.
1. Send an Announcement with tcp://0.0.0.0:<some-port>
2. Query the DeviceID
3. Observe the returned IP Address is no longer nil; i.e. `tcp://<nil>:<some-port>`
```
Add an id attribute to the submit button shown on the login form. This
allows my password manager's form filling function to interact with the
button after filling in username and password (which already have the id
attribute in place).
This makes the new default $XDG_STATE_HOME/syncthing or
~/.local/state/syncthing, while still looking in legacy locations first
for existing installs.
Note that this does not *move* existing installs, and nor should we.
Existing paths will continue to be used as-is, but the user can move the
dir into the new place if they want to use it (as they could prior to
this change as well, for that matter).
### Documentation
Needs update to the config docs about our default locations.
lib/fs: Fix conflicts on Android due to fluctuating inode change time
[1] added inode change time to file info in order to support syncing
extended attributes. However, in the case of Android, this inode change
time fluctuates, leading to unexpected conflicts even when the user has
not even touched the files on the Android device itself. Thus, in order
to prevent those conflicts from happening, do not write inode change
time on Android.
[1] 6cac308bcd
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
### Purpose
Deduplicated files are apparently considered 'irregular' under the hood,
this causes them to simply be ignored by Syncthing. This change is more
of a workaround than a proper fix, as the fix should probably happen in
the underlying libraries? - which may take some time. In the meanwhile,
this change should make deduplicated files be treated as regular files
and be indexed and synced as they should.
### Testing
Create some volume where deduplication is turned on (see the relevant
issue for details, including a proper description of how to reproduce
it). Prior to this change, the deduplicated files were simply ignored
(even by the indexer). After this change, the deduplicated files are
being index and synced properly.
This is motivated by the Android app:
https://github.com/syncthing/syncthing-android/pull/1982#issuecomment-1752042554
The planned fix in response to basic auth behaviour changing in #8757
was to add the `Authorization` header when opening the WebView, but it
turns out the function used only applies the header to the initial page
load, not any subsequent script loads or AJAX calls. The
`basicAuthAndSessionMiddleware` checks for no-auth exceptions before
checking the `Authorization` header, so the header has no effect on the
initial page load since the `/` path is a no-auth exception. Thus the
Android app fails to log in when opening the WebView.
This changes the order of checks in `basicAuthAndSessionMiddleware` so
that the `Authorization` header is always checked if present, and a
session cookie is set if it is valid. Only after that does the
middleware fall back to checking for no-auth exceptions.
`api_test.go` has been expanded with additional checks:
- Check that a session cookie is set whenever correct basic auth is
provided.
- Check that a session cookie is not set when basic auth is incorrect.
- Check that a session cookie is not set when authenticating with an API
token (either via `X-Api-Key` or `Authorization: Bearer`).
And an additional test case:
- Check that requests to `/` always succeed, but receive a session
cookie when correct basic auth is provided.
I have manually verified that
- The new assertions fail if the `createSession` call is removed in
`basicAuthAndSessionMiddleware`.
- The new test cases in e6e4df4d70 fail
before the change in 0e47d37e73 is
applied.
improve parsing of gui-address overrides
make checks for whether the gui-address is overridden consistent by
checking whether the environment variable is set and not an empty
string. the `Network()` function however checked for the inclusion of
a slash instead of the presence of any characters. If the config file's
gui address was set to a unix socket and the gui override to a tcp
address, then the function would have wrongly returned "unix".
the `URL()` function always returned the config file's gui address if a
unix socket was configured, even if an override was specified.
the `URL()` function wrongly formatted unix addresses. the http(s)
protocol was used as the sheme and the path was percent escaped. because
of the previous bug, this could only be triggered if the config file's
gui address was tcp and an unix socket override was given.
simplify the `useTLS()` function's codepath for overrides.
Co-authored-by: digital <didev@dinid.net>
Add a new entry to the unfolded device info to inform the user that the
device has been marked as "untrusted" and all folders shared with it
have to be password-protected or already Receive Encrypted.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Add a new entry to the unfolded device info to inform the user that the
device has Auto Accept enabled.
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Opening and hiding multiple modals at the same time as well as opening a
modal before fully hiding the previous one can lead to the body padding
infinitely increasing by the scrollbar width each time, with the only
way to fix it being refreshing the GUI.
Therefore, always try to ensure to open and hide multiple modals one by
one, and also that the previous modal has fully been hidden before
proceeding to open the next one. The most common case when this problem
happens is when saving config changes which displays a GUI blocking
modal that overlaps, e.g. with folder or device modals that have not
been hidden yet.
Ref: https://github.com/twbs/bootstrap/issues/3902#issuecomment-1547187799
Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
Jakob Borg
dependabot[bot]
cjc7373andJakob Borg
Syncthing Release Automation
tomasz1986andAndré Colomb
Emil Lundberg
Anatoli Babenia
DerRockWolf
vapatel2
Catfriend1
Eric P
orangekame3
DeflateAwningandAndré Colomb
d-volution
Martin Polehla