Commit Graph
8094 Commits
Author SHA1 Message Date
Jakob Borg b4776ea4e0 feat(stdiscosrv): configurable desired not-found rate 2025-04-13 09:41:16 +02:00
Jakob Borg b5ffd0a796 chore(blobs): generalised blob storage 2025-04-13 09:41:16 +02:00
Jakob Borg c74299b59a chore(stdiscosrv): path style s3 2025-04-13 09:40:14 +02:00
Jakob Borg 8b6d837483 feat(ursv): add os/arch/distribution metric 2025-04-13 09:40:14 +02:00
Jakob Borg 3e74b3dee2 chore(strelaypoolsrv): limit number of returned relays
Avoid unnecessarily enormous responses by returning a random subset of
relays.
2025-04-13 09:40:14 +02:00
Jakob Borg 2902da996c build(infra): run in Docker environment for pushes 2025-04-13 09:40:14 +02:00
Jakob Borg f6f144bf17 chore(stupgrades): expose latest release as a metric 2025-04-13 09:40:11 +02:00
Sébastien WENSKE ab5c42f4a0 feat(api, gui): allow authentication bypass for metrics (#10045)
### Purpose

Give the ability to skip authentication for prometheus metrics
("/metrics").

### Testing

When authentication is enabled and "Metrics Without Auth" is checked
(not the default), the "/metrics" path remains accessible even when
disconnected.

### Screenshots


![image](https://github.com/user-attachments/assets/144b696b-dd72-46f4-94d5-cd21848e4a4c)

### Documentation

https://github.com/syncthing/docs/pull/906
2025-04-13 07:35:57 +00:00
Jakob Borg 780b8fd3bc Merge branch 'main' into v2
* main:
  build: push artifacts to Azure (#10044)
  fix(syncthing): use separate lock file instead of locking the certificate (fixes #10053) (#10054)
  fix(syncthing): use separate lock file instead of locking the certificate (fixes #10053) (#10054)
v2.0.0-beta.9
2025-04-12 15:16:36 +02:00
Jakob Borg 7db3f7eaac Merge branch 'release-1.29.5'
* release-1.29.5:
  build: push artifacts to Azure (#10044)
  fix(syncthing): use separate lock file instead of locking the certificate (fixes #10053) (#10054)
2025-04-12 14:57:04 +02:00
Jakob Borg f0b666269b build: push artifacts to Azure (#10044)
Provider migration
v1.29.5
2025-04-12 14:55:24 +02:00
Jakob Borg 190a59842c fix(syncthing): use separate lock file instead of locking the certificate (fixes #10053) (#10054)
Apparently that nukes the cert under some circumstances on some Windows
🤷
2025-04-12 14:49:23 +02:00
Jakob Borg 40888c1a66 fix(syncthing): use separate lock file instead of locking the certificate (fixes #10053) (#10054)
Apparently that nukes the cert under some circumstances on some Windows
🤷
2025-04-12 14:46:57 +02:00
Jakob Borg ddea2e449c fix(db): version vector serialisation :( (#10050)
ffs
v2.0.0-beta.8
2025-04-09 17:46:49 +02:00
Jakob Borg 7cfa871d58 fix(db): skip invalid files as remote need when local is deleted 2025-04-09 16:24:17 +02:00
Jakob Borg 95b39a791d Merge branch 'main' into v2
* main:
  fix(gui): fix previous commit
  fix(gui): mark unseen disconnected devices as inactive (#10048)
  fix(strings): differentiate setup(n) and set(v) up (#10024)
  chore(fs): changes to allow Filesystem to be implemented externally (#10040)
  chore(config): resolve primary STUN servers via SRV record (fixes #10029) (#10031)
  build: push artifacts to Azure (#10044)
  chore(gui, man, authors): update docs, translations, and contributors
v2.0.0-beta.7
2025-04-09 15:40:25 +02:00
Jakob Borg fa0d933e49 fix(gui): fix previous commit 2025-04-09 15:39:09 +02:00
Tommy van der Vorst e0c1abc5fe fix(sqlite): apply options (#10049)
@calmh it seems `sqlite.Open` forgets to apply options supplied to it
(currently only the delete retention interval).
2025-04-09 06:29:46 +02:00
tomasz1986 8372c0288f fix(gui): mark unseen disconnected devices as inactive (#10048)
Currently, the "Disconnected (Inactive)" status is only given to devices
that have not been seen for 7 days or longer. However, this is not the
case when adding a new device, or after resetting the database. Those
devices are only marked as "Disconnected", and they will stay like that
even if a long time passes without any connectivity. Moreover, the lack
of an "Inactive" status may confuse the user to believe that their
disconnect is only temporary.

For this reason, always mark devices that have not been seen yet as
"Disconnected (Inactive)".

Signed-off-by: Tomasz Wilczyński <twilczynski@naver.com>
2025-04-08 22:08:00 +02:00
Paul Donald 5f5d672a7d fix(strings): differentiate setup(n) and set(v) up (#10024)
Correct GUI strings, translations and comments to use proper grammar.
v1.29.5-rc.1
2025-04-08 12:45:05 +00:00
Tommy van der Vorst d23cd197e1 chore(fs): changes to allow Filesystem to be implemented externally (#10040)
### Purpose

The `fs.Filesystem` interface contains two parts that cannot be
implemented externally because they are private:

* `filesystemWrapperType`: this PR changes `unwrapFilesystem` to
downcast to a specific concrete type
* `underlying`: this PR simply moves it to an unexported interface

### Testing

Regular tests pass.
2025-04-08 12:39:39 +00:00
bt90 d7ca483df1 chore(config): resolve primary STUN servers via SRV record (fixes #10029) (#10031)
### Purpose

Fixes #10029

### Testing

```
[3JPXJ] 2025/04/03 14:36:44.601454 stun.go:146: DEBUG: Running stun for Stun@udp://[::]:22000 via fyc5mja4mz5s0vmz1txx.syncthing.net:9999
[3JPXJ] 2025/04/03 14:36:54.185157 stun.go:170: DEBUG: Stun@udp://[::]:22000 stun discovery on fyc5mja4mz5s0vmz1txx.syncthing.net:9999 resulted in no address
[3JPXJ] 2025/04/03 14:36:54.185204 stun.go:146: DEBUG: Running stun for Stun@udp://[::]:22000 via stun.internetcalls.com:3478
```

### Documentation

https://github.com/syncthing/docs/pull/904
2025-04-08 12:23:57 +00:00
Jakob Borg e48be98cd5 build: push artifacts to Azure (#10044)
Provider migration
2025-04-08 09:43:19 +02:00
Jakob Borg cbded11c43 Merge branch 'main' into v2
* main:
  fix(config): zero filesystemtype is "basic" (#10038)
v2.0.0-beta.6
2025-04-07 11:43:08 +02:00
Jakob Borg d5aa991b73 chore(db): use pseudo random naming for folder databases 2025-04-07 11:35:31 +02:00
Jakob Borg 05210d0325 fix(db): wrong prepare method 2025-04-07 10:58:14 +02:00
Jakob Borg 55da878452 chore: improved perf stats 2025-04-07 09:10:16 +02:00
Syncthing Release Automation e9a2ff3aa6 chore(gui, man, authors): update docs, translations, and contributors 2025-04-07 03:50:00 +00:00
Jakob Borg c9650fc7d5 chore(model): delay starting a pull while there are incoming index updates (#10041)
This adds a simple delay to the process for starting the pull, by
default one second. In practice this means we're likely to wait for
initial index transfer, or multiple messages sent as part of a larger
change. This is better because we're more likely to have the whole
change for the purpose of handling renames etc, and also it's more
efficient to do one larger puller iteration instead of multiple while
also processing changes.

It does however introduce a certain amount of delay into the sync
process, so it can be tuned down or turned off entirely.
2025-04-06 14:31:02 +02:00
Jakob Borgandbt90 cf1cf85ce6 chore(db): use one SQLite database per folder (#10042)
This changes the database structure to use one database per folder, with
a small main database to coordinate. Reverts the prior change to buffer
all files in memory when pulling, meaning there is now a phase where the
WAL file will grow significantly, at least for initial sync of folders
with many directories.

---------

Co-authored-by: bt90 <btom1990@googlemail.com>
2025-04-06 14:30:43 +02:00
Jakob Borg 2301f72c5b fix(config): zero filesystemtype is "basic" (#10038)
For legacy purposes
2025-04-04 19:28:39 +00:00
Jakob Borg 7d51b1b620 Merge branch 'main' into v2
* main:
  fix(config): properly apply defaults when reading folder configuration (#10034)
  chore(model): add metric for total number of conflicts (#10037)
  build: replace underscore in Debian version (#10032)
v2.0.0-beta.5
2025-04-04 19:05:08 +02:00
Tommy van der Vorst f7c8efd93c fix(config): properly apply defaults when reading folder configuration (#10034) 2025-04-04 16:46:12 +00:00
Sébastien WENSKE 3e7ccf7c48 chore(model): add metric for total number of conflicts (#10037) 2025-04-04 09:24:04 -07:00
Jakob Borg fa3b9acca3 chore(db): buffer pulled files for smaller WAL (#10036)
We can't hold a long select open while pulling.
2025-04-04 08:15:59 +02:00
bt90 bae976905c chore(db): fix debug logging (#10033)
Looks like a copy&paste error
2025-04-03 20:21:39 +02:00
bt90 6bc2784e9a build: replace underscore in Debian version (#10032)
The workflow building Debian packages chokes on branches containing
underscores:

```
{:timestamp=>"2025-04-03T10:31:46.749835+0000", :message=>"Invalid package configuration: The version looks invalid for Debian packages. Debian version field must contain only alphanumerics and . (period), + (plus), - (hyphen) or ~ (tilde). I have '1.29.5~dev.13.ga38df11f~srv_stun' which which isn't valid.", :level=>:error}
```

This replaces the offending `_` with a `~` which should yield a valid
version.
2025-04-03 14:28:33 +02:00
Jakob Borg 1dbdd6b720 Merge branch 'main' into v2
* main:
  feat(fs, config): add support for custom filesystem type construction (#9887)
  build(deps): update dependencies (#10020)
v2.0.0-beta.4
2025-04-03 10:21:01 +02:00
Tommy van der Vorst f15d50c2e8 feat(fs, config): add support for custom filesystem type construction (#9887)
For Synctrain I would like to create a virtual filesystem that exposes
iOS' photo library. This can only be accessed through APIs.
2025-04-03 10:12:23 +02:00
Jakob Borg 8a2d8ebf81 chore: configurable delete retention interval (#10030)
Command line flag, as it also needs to be able to take effect during
migration.
2025-04-03 09:55:19 +02:00
Jakob Borg b88aea34b6 fix(syncthing): make directory flags global for all commands (#10028)
The home/config/data flags and end vars apply equally to all subcommands
2025-04-03 08:58:46 +02:00
Jakob Borg 82a0dd8eaa chore(db): use shorter read transactions and periodic checkpoint for smaller WAL (#10027)
Also make sure our journal size limit is in effect when checkpointing.
2025-04-02 22:19:34 +02:00
Jakob Borg 4096a35b86 fix(db): handle large numbers of blocks in update (#10025)
Avoid failure when inserting file with very large block list
2025-04-02 19:35:37 +02:00
Jakob Borg 86cbc2486f chore: forget deleted files older than six months (fixes #6284) (#10023)
This reduces the number of file entries we carry in the database,
sometimes significantly. The downside is that if a file is deleted while
a device is offline, and that device comes back more than the cutoff
interval (six months) later, those files will get resurrected at some
point.
2025-04-02 14:58:59 +02:00
bt90 0bcc31d058 chore(db): increase journal limit to 64MiB (#10022)
The current limit is far too low for our workloads. Perhaps we should
aim even higher than the 64MiB this patch proposes?

Citing the sqlite docs:

> [...] after committing a transaction the rollback journal file may
remain in the file-system. **This increases performance for subsequent
transactions since overwriting an existing file is faster than append to
a file**, but it also consumes file-system space.

tl;dr: if the limit is too low, we're shooting ourselves in the foot in
terms of performance.
2025-04-02 14:52:44 +02:00
Jakob Borg 2c3a890d2f fix(syncthing): remove duplicate --no-console flag v2.0.0-beta.3 2025-04-02 12:26:24 +02:00
Jakob Borg f9007ed106 build(deps): update dependencies (#10020)
deps deps deps
2025-04-02 08:51:37 +02:00
Jakob Borg 2953630bc3 Merge branch 'main' into v2
* main:
  chore(fs): speed up case normalization (#10013)
  chore(config): remove discontinued secondary STUN servers (fixes #10011) (#10012)
  chore(gui, man, authors): update docs, translations, and contributors
  fix(stun): better error handling (ref #10008) (#10010)
  fix(config): remove discontinued primary STUN server (fixes #10008) (#10009)
  fix(gui): validate device ID in canonical form (fixes #7291) (#10006)
v2.0.0-beta.2
2025-04-01 13:43:33 +02:00
Jakob Borg a99e670ebb chore: harmonise command line flags (#10007)
(v2 change)

This cleans up the command line parsing a little:
- Remove the hack for supporting legacy single-dash long options (e.g.
`-home`), thus enabling actual short options
- Move legacy imperative flags from under the serve command into
separate commands, e.g. `syncthing serve --paths` to see the paths list
is now `syncthing paths`, `syncthing --upgrade-check` is now `syncthing
upgrade --check`
- Add environment variable support for all remaining flags for the
`serve` command (with one exception, left for the reader to discover),
as these are now all modifiers and not imperative

```
% syncthing --help
Usage: syncthing <command>

Flags:
  -h, --help    Show context-sensitive help.

Commands:
  serve                  Run Syncthing (default)
  cli                    Command line interface for Syncthing
  browser                Open GUI in browser, then exit
  decrypt                Decrypt or verify an encrypted folder
  device-id              Show device ID, then exit
  generate               Generate key and config, then exit
  paths                  Show configuration paths, then exit
  upgrade                Perform or check for upgrade, then exit
  version                Show current version, then exit
  debug                  Various debugging commands
  install-completions    Print commands to install shell completions

Run "syncthing <command> --help" for more information on a command.
```

```
% syncthing serve --help
Usage: syncthing serve [flags]

Run Syncthing (default)

Flags:
  -h, --help                          Show context-sensitive help.

  -C, --config=PATH                   Set configuration directory (config and keys) ($STCONFDIR)
  -D, --data=PATH                     Set data directory (database and logs) ($STDATADIR)
  -H, --home=PATH                     Set configuration and data directory ($STHOMEDIR)
      --allow-newer-config            Allow loading newer than current config version ($STALLOWNEWERCONFIG)
      --audit                         Write events to audit file ($STAUDIT)
      --auditfile=PATH                Specify audit file (use "-" for stdout, "--" for stderr) ($STAUDITFILE)
      --db-maintenance-interval=8h    Database maintenance interval ($STDBMAINTINTERVAL)
      --gui-address=URL               Override GUI address (e.g. "http://192.0.2.42:8443") ($STGUIADDRESS)
      --gui-apikey=API-KEY            Override GUI API key ($STGUIAPIKEY)
      --no-console                    Hide console window ($STHIDECONSOLE)
      --logfile=PATH                  Log file name (see below) ($STLOGFILE)
      --logflags=BITS                 Select information in log line prefix (see below) ($STLOGFLAGS)
      --log-max-old-files=N           Number of old files to keep (zero to keep only current) ($STNUMLOGFILES)
      --log-max-size=BYTES            Maximum size of any file (zero to disable log rotation) ($STLOGMAXSIZE)
      --no-browser                    Do not start browser ($STNOBROWSER)
      --no-default-folder             Don't create the "default" folder on first startup ($STNODEFAULTFOLDER)
      --no-port-probing               Don't try to find free ports for GUI and listen addresses on first startup ($STNOPORTPROBING)
      --no-restart                    Do not restart Syncthing when exiting due to API/GUI command, upgrade, or crash ($STNORESTART)
      --no-upgrade                    Disable automatic upgrades ($STNOUPGRADE)
      --paused                        Start with all devices and folders paused ($STPAUSED)
      --unpaused                      Start with all devices and folders unpaused ($STUNPAUSED)
      --verbose                       Print verbose log output ($STVERBOSE)
      --debug-gui-assets-dir=PATH     Directory to load GUI assets from ($STGUIASSETS)
      --debug-perf-stats              Write running performance statistics to perf-$pid.csv (Unix only) ($STPERFSTATS)
      --debug-profile-block           Write block profiles to block-$pid-$timestamp.pprof every 20 seconds ($STBLOCKPROFILE)
      --debug-profile-cpu             Write a CPU profile to cpu-$pid.pprof on exit ($STCPUPROFILE)
      --debug-profile-heap            Write heap profiles to heap-$pid-$timestamp.pprof each time heap usage increases ($STHEAPPROFILE)
      --debug-profiler-listen=ADDR    Network profiler listen address ($STPROFILER)
      --debug-reset-delta-idxs        Reset delta index IDs, forcing a full index exchange
...
```
2025-04-01 13:42:16 +02:00
bt90 05cc6b0f43 chore(fs): speed up case normalization (#10013)
### Purpose

Resurrecting https://github.com/syncthing/syncthing/pull/9365

### Testing

Current benchmark results:

```
goos: linux
goarch: amd64
pkg: github.com/syncthing/syncthing/lib/fs
cpu: AMD EPYC 7763 64-Core Processor                
                                           │  ../old.txt  │             ../new.txt              │
                                           │    sec/op    │   sec/op     vs base                │
UnicodeLowercase/ASCII_lowercase-4            43.68n ± 4%   12.19n ± 1%  -72.09% (p=0.000 n=10)
UnicodeLowercase/ASCII_mixedcase_start-4     200.65n ± 2%   59.49n ± 3%  -70.35% (p=0.000 n=10)
UnicodeLowercase/ASCII_mixedcase_end-4        95.50n ± 2%   59.10n ± 2%  -38.12% (p=0.000 n=10)
UnicodeLowercase/Latin1_lowercase-4           122.5n ± 1%   131.4n ± 1%   +7.27% (p=0.000 n=10)
UnicodeLowercase/Latin1_mixedcase_start-4     339.9n ± 2%   309.2n ± 1%   -9.05% (p=0.000 n=10)
UnicodeLowercase/Latin1_mixedcase_end-4       183.6n ± 2%   174.3n ± 1%   -5.04% (p=0.000 n=10)
UnicodeLowercase/Unicode_lowercase-4          456.6n ± 1%   440.5n ± 1%   -3.53% (p=0.000 n=10)
UnicodeLowercase/Unicode_mixedcase_start-4    625.9n ± 1%   595.6n ± 1%   -4.83% (p=0.000 n=10)
UnicodeLowercase/Unicode_mixedcase_end-4      516.2n ± 1%   495.5n ± 1%   -4.02% (p=0.000 n=10)
geomean                                       214.1n        150.4n       -29.72%

                                           │  ../old.txt  │             ../new.txt              │
                                           │     B/op     │    B/op     vs base                 │
UnicodeLowercase/ASCII_lowercase-4           0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/ASCII_mixedcase_start-4     24.00 ± 0%     24.00 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/ASCII_mixedcase_end-4       24.00 ± 0%     24.00 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Latin1_lowercase-4          0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Latin1_mixedcase_start-4    32.00 ± 0%     32.00 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Latin1_mixedcase_end-4      32.00 ± 0%     32.00 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Unicode_lowercase-4         0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Unicode_mixedcase_start-4   48.00 ± 0%     48.00 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Unicode_mixedcase_end-4     48.00 ± 0%     48.00 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                                 ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                           │  ../old.txt  │             ../new.txt              │
                                           │  allocs/op   │ allocs/op   vs base                 │
UnicodeLowercase/ASCII_lowercase-4           0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/ASCII_mixedcase_start-4     1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/ASCII_mixedcase_end-4       1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Latin1_lowercase-4          0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Latin1_mixedcase_start-4    1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Latin1_mixedcase_end-4      1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Unicode_lowercase-4         0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Unicode_mixedcase_start-4   1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=10) ¹
UnicodeLowercase/Unicode_mixedcase_end-4     1.000 ± 0%     1.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                                 ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean
```

I think the `+7%` for the lowercase Latin1 testcase is easily outweighed
by the ASCII and unicode improvements 🙂
2025-04-01 13:41:57 +02:00