Compare commits

..

19 Commits

Author SHA1 Message Date
mattn 8ca3cca0a0 chore: use path/filepath for local file system paths (#10705)
Build Syncthing / Gather common facts (push) Successful in 32s
Build Syncthing / Build and test (~1.25.0, macos-latest) (push) Has been cancelled
Build Syncthing / Build and test (~1.25.0, windows-latest) (push) Has been cancelled
Build Syncthing / Build and test (~1.26.0, macos-latest) (push) Has been cancelled
Build Syncthing / Build and test (~1.26.0, ubuntu-latest) (push) Has been cancelled
Build Syncthing / Build and test (~1.26.0, windows-latest) (push) Has been cancelled
Build Syncthing / Basic checks passed (push) Has been cancelled
Build Syncthing / Package for Windows (push) Has been cancelled
Build Syncthing / Build and test (~1.25.0, ubuntu-latest) (push) Has been cancelled
Build Syncthing / Codesign for Windows (push) Has been cancelled
Build Syncthing / Package for Linux (common) (push) Has been cancelled
Mirrors / Mirror to Codeberg (push) Has been cancelled
Build Syncthing / Package for Linux (other) (push) Has been cancelled
Build Syncthing / Package for Linux (mips) (push) Has been cancelled
Build Syncthing / Package for illumos (push) Has been cancelled
Build Syncthing / Package for macOS (push) Has been cancelled
Build Syncthing / Notarize for macOS (push) Has been cancelled
Build Syncthing / Package cross compiled (push) Has been cancelled
Build Syncthing / Package source code (push) Has been cancelled
Build Syncthing / Sign for upgrade (push) Has been cancelled
Build Syncthing / Package for Debian (push) Has been cancelled
Build Syncthing / Publish nightly build (push) Has been cancelled
Build Syncthing / Publish release files (push) Has been cancelled
Build Syncthing / Publish APT (push) Has been cancelled
Build Syncthing / Build and push Docker images (GHCR) (Dockerfile, syncthing, syncthing) (push) Has been cancelled
Build Syncthing / Build and push Docker images (GHCR) (Dockerfile.stdiscosrv, discosrv, stdiscosrv) (push) Has been cancelled
Build Syncthing / Build and push Docker images (GHCR) (Dockerfile.strelaysrv, relaysrv, strelaysrv) (push) Has been cancelled
Build Syncthing / Sync images to Docker hub (push) Has been cancelled
Build Syncthing / Run govulncheck (push) Has been cancelled
Build Syncthing / Run golangci-lint (push) Has been cancelled
Build Syncthing / Run meta checks (push) Has been cancelled
### Purpose

`path` is for slash-separated paths (URLs, BEP protocol); local file
system paths should use `path/filepath`. Fixed in
`cmd/stdiscosrv/database.go` (3 sites) and
`internal/db/sqlite/db_test.go` (1 site).

### Testing

`go build ./cmd/stdiscosrv/...` and `go vet` pass.

Signed-off-by: Yasuhiro Matsumoto <mattn.jp@gmail.com>
2026-05-23 22:31:33 +02:00
mattn 0c489f4ae2 fix(stcrashreceiver): close source loader responses on errors (#10704)
Fix a response body leak in `githubSourceCodeLoader.Load` where the body
was not closed when the HTTP status was non-200.

Signed-off-by: Yasuhiro Matsumoto <mattn.jp@gmail.com>
2026-05-23 13:50:03 +00:00
Jakob Borg deb1e5b38a Revert "build: temporarily disable illumos for release"
This reverts commit c0c401efeb.
2026-05-23 12:00:07 +02:00
Jakob Borg 49a2688caa Merge branch 'infrastructure'
* infrastructure:
  build: let infra containers builds fail individually
  chore(ur): move structs to reduce dependency chain
  chore(stcrashreceiver): add profiler on metrics port
  chore(stcrashreceiver): compact diskstore in-memory representation
  chore(stcrashreceiver): better source cache & metrics
  chore(stcrashreceiver): metrics on ignore matches
2026-05-23 09:36:25 +02:00
Jakob Borg 05b4f6abda build: let infra containers builds fail individually
Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-23 09:18:15 +02:00
Jakob Borg 9152d7fb2f chore(ur): move structs to reduce dependency chain
lib/ur brings in a lot of dependencies we don't need in e.g.
stcrashreceiver, who only needs the small failure reporting structs.
Make those part of the lean `contract` package instead.

Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-23 09:13:47 +02:00
Jakob Borg 4404b4dfb4 chore(stcrashreceiver): add profiler on metrics port
Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-23 08:51:34 +02:00
Jakob Borg b537090d91 chore(stcrashreceiver): compact diskstore in-memory representation
Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-23 08:51:34 +02:00
Jakob Borg 79423edbdf chore(stcrashreceiver): better source cache & metrics
Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-23 08:51:34 +02:00
Jakob Borg 33075974cb chore(stcrashreceiver): metrics on ignore matches
Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-23 08:51:32 +02:00
Jakob Borg 8a3a06f7ca build(deps): x/net for govulncheck (#10703)
Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-23 06:48:21 +00:00
Jakob Borg d0b35021c6 chore(syncthing): include runtime context in GC crashes (#10702)
The runtime prints a lot of context for crashes due to bad pointers etc,
which is required to understand the crash, but this context comes before
the `fatal error: ...` line. Currently those lines get filtered out and
not included in the crash report. This change modifies the criteria so
that we start collecting crash data also at a line that begins with
`runtime:`, and tweaks the parsing later to look for the specific
`panic:` or `fatal error:` which may come later as the subject.

---------

Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-23 08:40:43 +02:00
Jakob Borg 6322091462 fix(discover): only announce wildcard for TCP punching when listening on wildcard address (fixes #10503) (#10691)
If we aren't announcing e.g. tcp://0.0.0.0:22000 then also do not
announce tcp://0.0.0.0:0.

Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-23 06:37:48 +00:00
Jakob Borg 5464970c5d fix(versioner): ensure user read/write/execute on archived dirs (fixes #10532) (#10696)
This makes sure the user running Syncthing, and hence Synchting itself,
has read/write/execute on directories in .stversions. The other
permission bits remain copied from the source directory, ensuring
whatever group and other permissions were set remain in effect.

Closes #10695.

---------

Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-23 06:21:04 +00:00
Jakob Borg 3962a23723 fix(syncthing): properly upgrade via REST when Syncthing is running (fixes #10697) (#10699)
The locking logic for upgrades got inverted in the lockfile changes. If
we got the lock it means Syncthing wasn't already running, so we can do
a direct upgrade. If we failed to get the lock it means Syncthing was
running and we should tell the REST interface to do the upgrade.

Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-21 10:15:18 +02:00
Jakob Borg feaa90408e Merge branch 'infrastructure'
* infrastructure:
  fix(stcrashreceiver): allow extra pre/post data in version line
  chore(stcrashreceiver): improve logging
  chore(stdiscosrv): prewarm counters at startup
2026-05-21 09:57:46 +02:00
Jakob Borg a8ed6e4855 fix(stcrashreceiver): allow extra pre/post data in version line
Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-19 08:46:22 +02:00
Jakob Borg 5b1e1c0520 chore(stcrashreceiver): improve logging
Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-19 08:46:21 +02:00
Jakob Borg c17be06192 chore(stdiscosrv): prewarm counters at startup
Signed-off-by: Jakob Borg <jakob@kastelo.net>
2026-05-18 23:40:38 +02:00
26 changed files with 405 additions and 171 deletions
@@ -22,6 +22,7 @@ jobs:
if: github.repository_owner == 'syncthing'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
pkg:
- stcrashreceiver
+2 -2
View File
@@ -164,7 +164,7 @@ jobs:
needs:
- build-test
- package-linux
# - package-illumos
- package-illumos
- package-cross
- package-source
- package-debian
@@ -646,7 +646,7 @@ jobs:
needs:
- codesign-windows
- package-linux
# - package-illumos
- package-illumos
- package-macos
- package-cross
- package-source
@@ -0,0 +1,76 @@
2026-05-21 10:03:01 INF syncthing v2.1.1-dev.9.gb3b7d228.dirty-morecrashrep "Hafnium Hornet" (go1.26.3 darwin-arm64) jb@jbo-m3wl72rv 2026-05-21 07:58:11 UTC [stnoupgrade] (log.pkg=main)
2026-05-21 10:03:01 INF No automatic upgrades; STNOUPGRADE environment variable defined (log.pkg=main)
2026-05-21 10:03:01 INF Calculated our device ID (device=I6KAH76-66SLLLB-5PFXSOA-UFJCDZC-YAOMLEK-CP2GB32-BV5RQST-3PSROAU log.pkg=syncthing)
2026-05-21 10:03:01 INF Overall rate limit in use (send="is unlimited" recv="is unlimited" log.pkg=connections)
2026-05-21 10:03:01 INF Using discovery mechanism (identity="IPv4 local broadcast discovery on port 21027" log.pkg=discover)
2026-05-21 10:03:01 INF Using discovery mechanism (identity="IPv6 local multicast discovery on address [ff12::8384]:21027" log.pkg=discover)
2026-05-21 10:03:01 INF TCP listener starting (address=127.0.0.1:22001 log.pkg=connections)
2026-05-21 10:03:01 INF Ready to synchronize (folder.id=default folder.type=sendreceive log.pkg=model)
2026-05-21 10:03:01 INF QUIC listener starting (address=127.0.0.1:22001 log.pkg=connections)
2026-05-21 10:03:01 INF GUI and API listening (address=127.0.0.1:8081 log.pkg=api)
...
2026-05-21 10:03:01 INF Access the GUI via the following URL: http://127.0.0.1:8081/ (log.pkg=api)
2026-05-21 10:03:01 INF Loaded configuration (name=s1 log.pkg=syncthing)
2026-05-21 10:03:01 INF Loaded peer device configuration (device=MRIW7OK name=s2 address="[tcp://127.0.0.1:22002 quic://127.0.0.1:22002]" log.pkg=syncthing)
2026-05-21 10:03:01 INF Completed initial scan (folder.id=default folder.type=sendreceive log.pkg=model)
0xee9de1fe260
2026-05-21 10:03:02 INF Measured hashing performance (perf="2789.71 MB/s" log.pkg=syncthing)
Panic at 2026-05-21T10:03:02+02:00
runtime: marked free object in span 0x108b34d20, elemsize=8 freeindex=34 (bad use of unsafe.Pointer or having race conditions? try -d=checkptr or -race)
0xee9de1fe000 alloc marked
0xee9de1fe008 alloc marked
...
0xee9de1fe250 free unmarked
0xee9de1fe258 free unmarked
0xee9de1fe260 free marked zombie
7 6 5 4 3 2 1 0 f e d c b a 9 8 0123456789abcdef
00000ee9de1fe260: 00000000 00000000 ........
0xee9de1fe268 free unmarked
0xee9de1fe270 free unmarked
...
0xee9de1fff60 free unmarked
0xee9de1fff68 free unmarked
0xee9de1fff70 free unmarked
0xee9de1fff78 free unmarked
fatal error: found pointer to free object
runtime stack:
runtime.throw({0x105881781?, 0x8?})
runtime/panic.go:1229 +0x38 fp=0x16bf82bb0 sp=0x16bf82b80 pc=0x104f0ca48
runtime.(*mspan).reportZombies(0x108b34d20)
runtime/mgcsweep.go:893 +0x314 fp=0x16bf82c30 sp=0x16bf82bb0 pc=0x104ec10b4
runtime.(*sweepLocked).sweep(0x16bf82d88?, 0x0)
runtime/mgcsweep.go:673 +0xbd0 fp=0x16bf82d50 sp=0x16bf82c30 pc=0x104ec0840
runtime.(*mcentral).uncacheSpan(0x16bf82db8?, 0x104ea4954?)
runtime/mcentral.go:237 +0xbc fp=0x16bf82d80 sp=0x16bf82d50 pc=0x104eaac3c
runtime.(*mcache).releaseAll(0x1089a85f0)
runtime/mcache.go:322 +0x188 fp=0x16bf82df0 sp=0x16bf82d80 pc=0x104eaa4e8
runtime.(*mcache).prepareForSweep(0x1089a85f0)
runtime/mcache.go:366 +0x4c fp=0x16bf82e20 sp=0x16bf82df0 pc=0x104eaa61c
runtime.gcMarkTermination.func4(0xee9de005808)
runtime/mgc.go:1546 +0x24 fp=0x16bf82e50 sp=0x16bf82e20 pc=0x104f076e4
runtime.forEachPInternal(0x10656f798)
runtime/proc.go:2167 +0x178 fp=0x16bf82ee0 sp=0x16bf82e50 pc=0x104eda728
runtime.gcMarkTermination.forEachP.func7()
runtime/proc.go:2126 +0x40 fp=0x16bf82f10 sp=0x16bf82ee0 pc=0x104eb3130
runtime.systemstack(0x7fc000)
runtime/asm_arm64.s:399 +0x68 fp=0x16bf82f20 sp=0x16bf82f10 pc=0x104f12888
goroutine 84 gp=0xee9de45c1e0 m=3 mp=0xee9de019008 [flushing proc caches]:
runtime.systemstack_switch()
runtime/asm_arm64.s:347 +0x8 fp=0xee9de805c40 sp=0xee9de805c30 pc=0x104f12808
runtime.forEachP(...)
runtime/proc.go:2112
runtime.gcMarkTermination({0xc0?, 0x1331f928480ca?, 0xc?, 0x0?})
runtime/mgc.go:1545 +0x5f4 fp=0xee9de805e80 sp=0xee9de805c40 pc=0x104eb28c4
runtime.gcMarkDone()
runtime/mgc.go:1173 +0x364 fp=0xee9de805f20 sp=0xee9de805e80 pc=0x104eb1bc4
runtime.gcBgMarkWorker(0xee9de341810)
runtime/mgc.go:1912 +0x29c fp=0xee9de805fb0 sp=0xee9de805f20 pc=0x104eb372c
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1695 +0x20 fp=0xee9de805fd0 sp=0xee9de805fb0 pc=0x104eb3470
runtime.goexit({})
runtime/asm_arm64.s:1447 +0x4 fp=0xee9de805fd0 sp=0xee9de805fd0 pc=0x104f14a04
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1695 +0x134
+14 -4
View File
@@ -136,15 +136,25 @@ func (d *diskStore) Exists(path string) bool {
}
func (d *diskStore) clean() {
for len(d.currentFiles) > 0 && (len(d.currentFiles) > d.maxFiles || d.currentSize > d.maxBytes) {
f := d.currentFiles[0]
numDeleted := 0
for idx := range d.currentFiles {
if len(d.currentFiles)-numDeleted < d.maxFiles && d.currentSize < d.maxBytes {
break
}
f := d.currentFiles[idx]
log.Println("Removing", f.path)
if err := os.Remove(f.path); err != nil {
log.Println("Failed to remove file:", err)
}
d.currentFiles = d.currentFiles[1:]
d.currentSize -= f.size
numDeleted = idx + 1
}
// Compact currentFiles
copy(d.currentFiles, d.currentFiles[numDeleted:])
d.currentFiles = d.currentFiles[:len(d.currentFiles)-numDeleted]
var oldest time.Duration
if len(d.currentFiles) > 0 {
oldest = time.Since(time.Unix(d.currentFiles[0].mtime, 0)).Truncate(time.Minute)
@@ -158,7 +168,7 @@ func (d *diskStore) clean() {
}
func (d *diskStore) inventory() error {
d.currentFiles = nil
d.currentFiles = d.currentFiles[:0]
d.currentSize = 0
err := filepath.Walk(d.dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
+11 -8
View File
@@ -20,6 +20,7 @@ import (
"io"
"log"
"net/http"
"net/http/pprof"
"os"
"path/filepath"
"regexp"
@@ -29,7 +30,7 @@ import (
raven "github.com/getsentry/raven-go"
"github.com/prometheus/client_golang/prometheus/promhttp"
"github.com/syncthing/syncthing/lib/build"
"github.com/syncthing/syncthing/lib/ur"
"github.com/syncthing/syncthing/lib/ur/contract"
)
const maxRequestSize = 1 << 20 // 1 MiB
@@ -89,6 +90,7 @@ func main() {
if params.MetricsListen != "" {
mmux := http.NewServeMux()
mmux.Handle("/metrics", promhttp.Handler())
mmux.HandleFunc("/debug/pprof/", pprof.Index)
go func() {
if err := http.ListenAndServe(params.MetricsListen, mmux); err != nil {
log.Fatalln("HTTP serve metrics:", err)
@@ -123,12 +125,13 @@ func handleFailureFn(dsn, failureDir string, ignore *ignorePatterns) func(w http
return
}
if ignore.match(bs) {
if pat, ok := ignore.match(bs); ok {
metricIgnoreMatchesTotal.WithLabelValues(pat).Inc()
result = "ignored"
return
}
var reports []ur.FailureReport
var reports []contract.FailureReport
err = json.Unmarshal(bs, &reports)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
@@ -176,7 +179,7 @@ func handleFailureFn(dsn, failureDir string, ignore *ignorePatterns) func(w http
}
}
func saveFailureWithGoroutines(data ur.FailureData, failureDir string) (string, error) {
func saveFailureWithGoroutines(data contract.FailureData, failureDir string) (string, error) {
bs := make([]byte, len(data.Description)+len(data.Goroutines))
copy(bs, data.Description)
copy(bs[len(data.Description):], data.Goroutines)
@@ -216,14 +219,14 @@ func loadIgnorePatterns(path string) (*ignorePatterns, error) {
return &ignorePatterns{patterns: patterns}, nil
}
func (i *ignorePatterns) match(report []byte) bool {
func (i *ignorePatterns) match(report []byte) (string, bool) {
if i == nil {
return false
return "", false
}
for _, re := range i.patterns {
if re.Match(report) {
return true
return re.String(), true
}
}
return false
return "", false
}
+20
View File
@@ -37,4 +37,24 @@ var (
Subsystem: "crashreceiver",
Name: "diskstore_oldest_age_seconds",
})
metricSentryReportsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "crashreceiver",
Name: "sentry_reports_total",
}, []string{"result"})
metricIgnoreMatchesTotal = promauto.NewCounterVec(prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "crashreceiver",
Name: "ignore_matches_total",
}, []string{"pattern"})
metricSourceCodeLoadsTotal = promauto.NewCounterVec(prometheus.CounterOpts{
Namespace: "syncthing",
Subsystem: "crashreceiver",
Name: "source_code_loads_total",
}, []string{"result"})
metricSourceCodeCacheSize = promauto.NewGauge(prometheus.GaugeOpts{
Namespace: "syncthing",
Subsystem: "crashreceiver",
Name: "source_code_cache_size",
})
)
+8 -8
View File
@@ -10,6 +10,7 @@ import (
"bytes"
"context"
"errors"
"fmt"
"io"
"log"
"regexp"
@@ -52,11 +53,15 @@ func (s *sentryService) Serve(ctx context.Context) {
pkt, err := parseCrashReport(req.reportID, req.data)
if err != nil {
log.Println("Failed to parse crash report:", err)
metricSentryReportsTotal.WithLabelValues("parse_failure").Inc()
continue
}
if err := sendReport(s.dsn, pkt, req.userID); err != nil {
log.Println("Failed to send crash report:", err)
metricSentryReportsTotal.WithLabelValues("send_failure").Inc()
continue
}
metricSentryReportsTotal.WithLabelValues("success").Inc()
case <-ctx.Done():
return
@@ -69,6 +74,7 @@ func (s *sentryService) Send(reportID, userID string, data []byte) bool {
case s.inbox <- sentryRequest{reportID, userID, data}:
return true
default:
metricCrashReportsTotal.WithLabelValues("overflow").Inc()
return false
}
}
@@ -108,11 +114,10 @@ func parseCrashReport(path string, report []byte) (*raven.Packet, error) {
version, err := build.ParseVersion(string(parts[0]))
if err != nil {
return nil, err
return nil, fmt.Errorf("%w in %q", err, parts[0])
}
report = parts[1]
foundPanic := false
var subjectLine []byte
for {
parts = bytes.SplitN(report, []byte("\n"), 2)
@@ -123,14 +128,9 @@ func parseCrashReport(path string, report []byte) (*raven.Packet, error) {
line := parts[0]
report = parts[1]
if foundPanic {
// The previous line was our "Panic at ..." header. We are now
// at the beginning of the real panic trace and this is our
// subject line.
if bytes.HasPrefix(line, []byte("panic:")) || bytes.HasPrefix(line, []byte("fatal error:")) {
subjectLine = line
break
} else if bytes.HasPrefix(line, []byte("Panic at")) {
foundPanic = true
}
}
+18 -11
View File
@@ -9,26 +9,33 @@ package main
import (
"fmt"
"os"
"path/filepath"
"testing"
)
func TestParseReport(t *testing.T) {
bs, err := os.ReadFile("_testdata/panic.log")
files, err := filepath.Glob("_testdata/*.log")
if err != nil {
t.Fatal(err)
}
for _, file := range files {
bs, err := os.ReadFile(file)
if err != nil {
t.Fatal(err)
}
pkt, err := parseCrashReport("1/2/345", bs)
if err != nil {
t.Fatal(err)
pkt, err := parseCrashReport("1/2/345", bs)
if err != nil {
t.Fatal(err)
}
bs, err = pkt.JSON()
if err != nil {
t.Fatal(err)
}
fmt.Printf("%s\n", bs)
}
bs, err = pkt.JSON()
if err != nil {
t.Fatal(err)
}
fmt.Printf("%s\n", bs)
}
func TestCrashReportFingerprint(t *testing.T) {
+26 -11
View File
@@ -15,23 +15,33 @@ import (
"strings"
"sync"
"time"
lru "github.com/hashicorp/golang-lru/v2"
)
const (
urlPrefix = "https://raw.githubusercontent.com/syncthing/syncthing/"
httpTimeout = 10 * time.Second
urlPrefix = "https://raw.githubusercontent.com/syncthing/syncthing/"
httpTimeout = 10 * time.Second
maxCacheEntries = 1000
)
type cacheKey struct {
version string
file string
}
type githubSourceCodeLoader struct {
mut sync.Mutex
version string
cache map[string]map[string][][]byte // version -> file -> lines
client *http.Client
cache *lru.TwoQueueCache[cacheKey, [][]byte] // version & file -> lines
client *http.Client
}
func newGithubSourceCodeLoader() *githubSourceCodeLoader {
cache, _ := lru.New2Q[cacheKey, [][]byte](maxCacheEntries)
return &githubSourceCodeLoader{
cache: make(map[string]map[string][][]byte),
cache: cache,
client: &http.Client{Timeout: httpTimeout},
}
}
@@ -39,9 +49,6 @@ func newGithubSourceCodeLoader() *githubSourceCodeLoader {
func (l *githubSourceCodeLoader) LockWithVersion(version string) {
l.mut.Lock()
l.version = version
if _, ok := l.cache[version]; !ok {
l.cache[version] = make(map[string][][]byte)
}
}
func (l *githubSourceCodeLoader) Unlock() {
@@ -50,11 +57,13 @@ func (l *githubSourceCodeLoader) Unlock() {
func (l *githubSourceCodeLoader) Load(filename string, line, context int) ([][]byte, int) {
filename = filepath.ToSlash(filename)
lines, ok := l.cache[l.version][filename]
key := cacheKey{version: l.version, file: filename}
lines, ok := l.cache.Get(key)
if !ok {
// Cache whatever we managed to find (or nil if nothing, so we don't try again)
defer func() {
l.cache[l.version][filename] = lines
l.cache.Add(key, lines)
metricSourceCodeCacheSize.Set(float64(l.cache.Len()))
}()
knownPrefixes := []string{"/lib/", "/cmd/"}
@@ -73,19 +82,25 @@ func (l *githubSourceCodeLoader) Load(filename string, line, context int) ([][]b
resp, err := l.client.Get(url)
if err != nil {
fmt.Println("Loading source:", err)
metricSourceCodeLoadsTotal.WithLabelValues("failed").Inc()
return nil, 0
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
fmt.Println("Loading source:", resp.Status)
metricSourceCodeLoadsTotal.WithLabelValues("failed").Inc()
return nil, 0
}
data, err := io.ReadAll(resp.Body)
_ = resp.Body.Close()
if err != nil {
fmt.Println("Loading source:", err.Error())
metricSourceCodeLoadsTotal.WithLabelValues("failed").Inc()
return nil, 0
}
lines = bytes.Split(data, []byte{'\n'})
metricSourceCodeLoadsTotal.WithLabelValues("loaded").Inc()
} else {
metricSourceCodeLoadsTotal.WithLabelValues("cached").Inc()
}
return getLineFromLines(lines, line, context)
+10 -29
View File
@@ -7,21 +7,18 @@
package main
import (
"bytes"
"io"
"log"
"net/http"
"path"
"strings"
"sync"
)
type crashReceiver struct {
store *diskStore
sentry *sentryService
ignore *ignorePatterns
ignoredMut sync.RWMutex
ignored map[string]struct{}
}
func (r *crashReceiver) ServeHTTP(w http.ResponseWriter, req *http.Request) {
@@ -69,12 +66,6 @@ func (r *crashReceiver) serveGet(reportID string, w http.ResponseWriter, _ *http
// serveHead responds to HEAD requests by checking if the named report
// already exists in the system.
func (r *crashReceiver) serveHead(reportID string, w http.ResponseWriter, _ *http.Request) {
r.ignoredMut.RLock()
_, ignored := r.ignored[reportID]
r.ignoredMut.RUnlock()
if ignored {
return // found
}
if !r.store.Exists(reportID) {
http.Error(w, "Not found", http.StatusNotFound)
}
@@ -87,17 +78,7 @@ func (r *crashReceiver) servePut(reportID string, w http.ResponseWriter, req *ht
metricCrashReportsTotal.WithLabelValues(result).Inc()
}()
r.ignoredMut.RLock()
_, ignored := r.ignored[reportID]
r.ignoredMut.RUnlock()
if ignored {
result = "ignored_cached"
io.Copy(io.Discard, req.Body)
return // found
}
// Read at most maxRequestSize of report data.
log.Println("Receiving report", reportID)
lr := io.LimitReader(req.Body, maxRequestSize)
bs, err := io.ReadAll(lr)
if err != nil {
@@ -106,14 +87,12 @@ func (r *crashReceiver) servePut(reportID string, w http.ResponseWriter, req *ht
return
}
if r.ignore.match(bs) {
r.ignoredMut.Lock()
if r.ignored == nil {
r.ignored = make(map[string]struct{})
}
r.ignored[reportID] = struct{}{}
r.ignoredMut.Unlock()
first := string(bytes.TrimSpace(bytes.Split(bs, []byte("\n"))[0]))
if pat, ok := r.ignore.match(bs); ok {
metricIgnoreMatchesTotal.WithLabelValues(pat).Inc()
result = "ignored"
log.Printf("Ignored report %s, matched: %s (%s)", reportID[:8], pat, first)
return
}
@@ -121,13 +100,15 @@ func (r *crashReceiver) servePut(reportID string, w http.ResponseWriter, req *ht
// Store the report
if !r.store.Put(reportID, bs) {
log.Println("Failed to store report (queue full):", reportID)
log.Println("Failed to store report (queue full):", reportID[:8])
result = "queue_failure"
}
// Send the report to Sentry
if !r.sentry.Send(reportID, userIDFor(req), bs) {
log.Println("Failed to send report to sentry (queue full):", reportID)
log.Println("Failed to send report to sentry (queue full):", reportID[:8])
result = "sentry_failure"
}
log.Printf("Received report %s (%s)", reportID[:8], first)
}
+4 -4
View File
@@ -15,7 +15,7 @@ import (
"io"
"log/slog"
"os"
"path"
"path/filepath"
"runtime"
"slices"
"strings"
@@ -77,7 +77,7 @@ func newInMemoryStore(dir string, flushInterval time.Duration, blobs blob.Store)
slog.Error("Failed to find database in blob storage", "error", cerr)
return s
}
fd, cerr := os.Create(path.Join(s.dir, "records.db"))
fd, cerr := os.Create(filepath.Join(s.dir, "records.db"))
if cerr != nil {
slog.Error("Failed to create database file", "error", cerr)
return s
@@ -257,7 +257,7 @@ func (s *inMemoryStore) write() (err error) {
}
}()
dbf := path.Join(s.dir, "records.db")
dbf := filepath.Join(s.dir, "records.db")
fd, err := os.Create(dbf + ".tmp")
if err != nil {
return err
@@ -340,7 +340,7 @@ func (s *inMemoryStore) write() (err error) {
}
func (s *inMemoryStore) read() (int, error) {
fd, err := os.Open(path.Join(s.dir, "records.db"))
fd, err := os.Open(filepath.Join(s.dir, "records.db"))
if err != nil {
return 0, err
}
+27 -8
View File
@@ -7,6 +7,8 @@
package main
import (
"net/http"
"github.com/prometheus/client_golang/prometheus"
)
@@ -113,14 +115,11 @@ var (
)
const (
dbOpGet = "get"
dbOpPut = "put"
dbOpMerge = "merge"
dbOpDelete = "delete"
dbResSuccess = "success"
dbResNotFound = "not_found"
dbResError = "error"
dbResUnmarshalError = "unmarsh_err"
dbOpGet = "get"
dbOpPut = "put"
dbOpMerge = "merge"
dbResSuccess = "success"
dbResNotFound = "not_found"
)
func init() {
@@ -132,4 +131,24 @@ func init() {
databaseOperations, databaseOperationSeconds,
databaseWriteSeconds, databaseLastWritten,
retryAfterLevel)
// Prewarm important counters so they're available with zero values at
// startup
apiRequestsTotal.WithLabelValues(http.MethodGet, "200")
apiRequestsTotal.WithLabelValues(http.MethodGet, "404")
apiRequestsTotal.WithLabelValues(http.MethodPost, "204")
apiRequestsTotal.WithLabelValues(http.MethodPost, "400")
apiRequestsTotal.WithLabelValues(http.MethodPost, "403")
lookupRequestsTotal.WithLabelValues("success")
lookupRequestsTotal.WithLabelValues("not_found_ever")
lookupRequestsTotal.WithLabelValues("not_found_recent")
announceRequestsTotal.WithLabelValues("success")
announceRequestsTotal.WithLabelValues("bad_request")
announceRequestsTotal.WithLabelValues("no_certificate")
replicationSendsTotal.WithLabelValues("success")
replicationRecvsTotal.WithLabelValues("success")
}
+7 -3
View File
@@ -915,10 +915,14 @@ func (u upgradeCmd) Run() error {
case err != nil && !os.IsNotExist(err):
slog.Error("Failed to lock for upgrade", slogutil.Error(err))
os.Exit(1)
case locked:
err = upgradeViaRest()
default:
case locked || os.IsNotExist(err):
// We got the lock, or the config directory didn't exist, so we
// can do a direct upgrade
err = upgrade.To(release)
default:
// We didn't get the lock, because Syncthing was running, so
// upgrade via REST.
err = upgradeViaRest()
}
}
if err != nil {
+1 -1
View File
@@ -233,7 +233,7 @@ func copyStderr(stderr io.Reader, dst io.Writer) {
dst.Write([]byte(line))
if panicFd == nil && (strings.HasPrefix(line, "panic:") || strings.HasPrefix(line, "fatal error:")) {
if panicFd == nil && (strings.HasPrefix(line, "panic:") || strings.HasPrefix(line, "fatal error:") || strings.HasPrefix(line, "runtime:")) {
panicFd, err = os.Create(locations.GetTimestamped(locations.PanicLog))
if err != nil {
slog.Error("Failed to create panic log", slogutil.Error(err))
+3 -5
View File
@@ -42,8 +42,8 @@ require (
github.com/wlynxg/anet v0.0.5
golang.org/x/crypto v0.51.0
golang.org/x/exp v0.0.0-20260508232706-74f9aab9d74a
golang.org/x/net v0.54.0
golang.org/x/sys v0.44.0
golang.org/x/net v0.55.0
golang.org/x/sys v0.45.0
golang.org/x/text v0.37.0
golang.org/x/time v0.15.0
google.golang.org/protobuf v1.36.11
@@ -111,9 +111,7 @@ replace github.com/gobwas/glob v0.2.3 => github.com/calmh/glob v0.0.0-2022061508
replace github.com/jackpal/gateway v1.1.1 => github.com/marbens-arch/gateway v1.1.2-0.20260308173556-c567cc04e7d4
// https://github.com/mattn/go-sqlite3/pull/1338
// https://github.com/mattn/go-sqlite3/pull/1399
// https://github.com/mattn/go-sqlite3/pull/1400
replace github.com/mattn/go-sqlite3 v1.14.44 => github.com/calmh/go-sqlite3 v1.14.45-0.20260519121030-00c8bf368e65
replace github.com/mattn/go-sqlite3 v1.14.44 => github.com/calmh/go-sqlite3 v1.14.35-0.20260509063420-822b4765116d
tool (
github.com/calmh/xdr/cmd/genxdr
+6 -6
View File
@@ -19,8 +19,8 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/calmh/glob v0.0.0-20220615080505-1d823af5017b h1:Fjm4GuJ+TGMgqfGHN42IQArJb77CfD/mAwLbDUoJe6g=
github.com/calmh/glob v0.0.0-20220615080505-1d823af5017b/go.mod h1:91K7jfEsgJSyfSrX+gmrRfZMtntx6JsHolWubGXDopg=
github.com/calmh/go-sqlite3 v1.14.45-0.20260519121030-00c8bf368e65 h1:4tbv1D+AkdxV4si6cdT3+8lr/c/gJqBdW6yGi8I7z3E=
github.com/calmh/go-sqlite3 v1.14.45-0.20260519121030-00c8bf368e65/go.mod h1:pjEuOr8IwzLJP2MfGeTb0A35jauH+C2kbHKBr7yXKVQ=
github.com/calmh/go-sqlite3 v1.14.35-0.20260509063420-822b4765116d h1:liVtRMlDBqDyR4qKkMRIFTdUK6+aoG6Oh97DKnhrCHc=
github.com/calmh/go-sqlite3 v1.14.35-0.20260509063420-822b4765116d/go.mod h1:pjEuOr8IwzLJP2MfGeTb0A35jauH+C2kbHKBr7yXKVQ=
github.com/calmh/incontainer v1.0.0 h1:g2cTUtZuFGmMGX8GoykPkN1Judj2uw8/3/aEtq4Z/rg=
github.com/calmh/incontainer v1.0.0/go.mod h1:eOhqnw15c9X+4RNBe0W3HlUZFfX16O0EDsCOInTndHY=
github.com/calmh/xdr v1.2.0 h1:GaGSNH4ZDw9kNdYqle6+RcAENiaQ8/611Ok+jQbBEeU=
@@ -280,8 +280,8 @@ golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwY
golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk=
golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
golang.org/x/net v0.0.0-20220607020251-c690dde0001d/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.54.0 h1:2zJIZAxAHV/OHCDTCOHAYehQzLfSXuf/5SoL/Dv6w/w=
golang.org/x/net v0.54.0/go.mod h1:Sj4oj8jK6XmHpBZU/zWHw3BV3abl4Kvi+Ut7cQcY+cQ=
golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -309,8 +309,8 @@ golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ=
golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY=
golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6 h1:HjU6IWBiAgRIdAJ9/y1rwCn+UELEmwV+VsTLzj/W4sE=
golang.org/x/telemetry v0.0.0-20260508192327-42602be52be6/go.mod h1:Eqhaxk/wZsWEH8CRxLwj6xzEJbz7k1EFGqx7nyCoabE=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+1 -2
View File
@@ -13,7 +13,6 @@ import (
"errors"
"iter"
"os"
"path"
"path/filepath"
"sync"
"testing"
@@ -1168,7 +1167,7 @@ func TestOpenSpecialName(t *testing.T) {
// Create a "base" dir that is in the way if the path becomes
// incorrectly truncated in the next steps.
base := path.Join(dir, "test")
base := filepath.Join(dir, "test")
if err := os.Mkdir(base, 0o755); err != nil {
t.Fatal(err)
}
+1 -1
View File
@@ -16,7 +16,7 @@ import (
// or, somewhere along the way the "+" in the version tag disappeared:
// syncthing v1.23.7-dev.26.gdf7b56ae.dirty-stversionextra "Fermium Flea" (go1.20.5 darwin-arm64) jb@ok.kastelo.net 2023-07-12 06:55:26 UTC [Some Wrapper, purego, stnoupgrade]
var (
longVersionRE = regexp.MustCompile(`syncthing\s+(v[^\s]+)\s+"([^"]+)"\s\(([^\s]+)\s+([^-]+)-([^)]+)\)\s+([^\s]+)[^\[]*(?:\[(.+)\])?$`)
longVersionRE = regexp.MustCompile(`syncthing\s+(v[^\s]+)\s+"([^"]+)"\s\(([^\s]+)\s+([^-]+)-([^)]+)\)\s+([^\s]+)[^\[]*(?:\[(.+)\])?`)
gitExtraRE = regexp.MustCompile(`\.\d+\.g[0-9a-f]+`) // ".1.g6aaae618"
gitExtraSepRE = regexp.MustCompile(`[.-]`) // dot or dash
)
+14
View File
@@ -57,6 +57,20 @@ func TestParseVersion(t *testing.T) {
Extra: []string{"Some Wrapper", "purego", "stnoupgrade"},
},
},
{
longVersion: `2026-05-18 14:53:32 INF syncthing v2.0.3 "Hafnium Hornet" (go1.25.0 darwin-amd64) builder@github.syncthing.net 2025-08-22 07:00:05 UTC [stnoupgrade] (log.pkg=main)`,
parsed: VersionParts{
Version: "v2.0.3",
Tag: "v2.0.3",
Commit: "",
Codename: "Hafnium Hornet",
Runtime: "go1.25.0",
GOOS: "darwin",
GOARCH: "amd64",
Builder: "builder@github.syncthing.net",
Extra: []string{"stnoupgrade"},
},
},
}
for _, tc := range cases {
+5 -3
View File
@@ -13,6 +13,7 @@ import (
"log/slog"
"net"
"net/url"
"slices"
"sync"
"time"
@@ -185,9 +186,10 @@ func (t *tcpListener) WANAddresses() []*url.URL {
t.mut.RUnlock()
// If we support ReusePort, add an unspecified zero port address, which will be resolved by the discovery server
// in hopes that TCP punch through works.
if dialer.SupportsReusePort {
// If we support ReusePort, and we are already announcing an unspecified
// address, add an unspecified zero port address, which will be resolved
// by the discovery server in hopes that TCP punch through works.
if dialer.SupportsReusePort && slices.ContainsFunc(uris, func(u *url.URL) bool { return u.Hostname() == "0.0.0.0" }) {
uri := *t.uri
uri.Host = "0.0.0.0:0"
uris = append([]*url.URL{&uri}, uris...)
-1
View File
@@ -289,7 +289,6 @@ func (c *globalClient) sendAnnouncement(ctx context.Context, timer *time.Timer)
resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode > 299 {
slog.DebugContext(ctx, "announce POST", "server", c.server, "status", resp.Status)
c.setError(errors.New(resp.Status))
if h := resp.Header.Get("Retry-After"); h != "" {
+2 -2
View File
@@ -20,7 +20,7 @@ import (
"github.com/syncthing/syncthing/lib/events"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/svcutil"
"github.com/syncthing/syncthing/lib/ur"
"github.com/syncthing/syncthing/lib/ur/contract"
)
type indexHandler struct {
@@ -470,7 +470,7 @@ func (s *indexHandler) logSequenceAnomaly(msg string, extra map[string]any) {
extraStrs[k] = fmt.Sprint(v)
}
s.evLogger.Log(events.Failure, ur.FailureData{
s.evLogger.Log(events.Failure, contract.FailureData{
Description: msg,
Extra: extraStrs,
})
+13
View File
@@ -282,3 +282,16 @@ func clear(v interface{}, since int) error {
}
return nil
}
type FailureReport struct {
FailureData
Count int
Version string
}
type FailureData struct {
Description string
Goroutines string
Extra map[string]string
}
+14 -26
View File
@@ -23,6 +23,7 @@ import (
"github.com/syncthing/syncthing/lib/events"
"github.com/syncthing/syncthing/lib/svcutil"
"github.com/syncthing/syncthing/lib/tlsutil"
"github.com/syncthing/syncthing/lib/ur/contract"
"github.com/thejerf/suture/v4"
)
@@ -39,23 +40,10 @@ var (
invalidEventDataType = "failure event data is not a string"
)
type FailureReport struct {
FailureData
Count int
Version string
}
type FailureData struct {
Description string
Goroutines string
Extra map[string]string
}
func FailureDataWithGoroutines(description string) FailureData {
func FailureDataWithGoroutines(description string) contract.FailureData {
var buf strings.Builder
pprof.Lookup("goroutine").WriteTo(&buf, 1)
return FailureData{
return contract.FailureData{
Description: description,
Goroutines: buf.String(),
Extra: make(map[string]string),
@@ -86,7 +74,7 @@ type failureHandler struct {
type failureStat struct {
first, last time.Time
count int
data FailureData
data contract.FailureData
}
func (h *failureHandler) Serve(ctx context.Context) error {
@@ -105,24 +93,24 @@ func (h *failureHandler) Serve(ctx context.Context) error {
if !ok {
// Just to be safe - shouldn't ever happen, as
// evChan is set to nil when unsubscribing.
h.addReport(FailureData{Description: evChanClosed}, time.Now())
h.addReport(contract.FailureData{Description: evChanClosed}, time.Now())
evChan = nil
continue
}
var data FailureData
var data contract.FailureData
switch d := e.Data.(type) {
case string:
data.Description = d
case FailureData:
case contract.FailureData:
data = d
default:
// Same here, shouldn't ever happen.
h.addReport(FailureData{Description: invalidEventDataType}, time.Now())
h.addReport(contract.FailureData{Description: invalidEventDataType}, time.Now())
continue
}
h.addReport(data, e.Time)
case <-timer.C:
reports := make([]FailureReport, 0, len(h.buf))
reports := make([]contract.FailureReport, 0, len(h.buf))
now := time.Now()
for descr, stat := range h.buf {
if now.Sub(stat.last) > minDelay || now.Sub(stat.first) > maxDelay {
@@ -152,7 +140,7 @@ func (h *failureHandler) Serve(ctx context.Context) error {
if sub != nil {
sub.Unsubscribe()
if len(h.buf) > 0 {
reports := make([]FailureReport, 0, len(h.buf))
reports := make([]contract.FailureReport, 0, len(h.buf))
for _, stat := range h.buf {
reports = append(reports, newFailureReport(stat))
}
@@ -179,7 +167,7 @@ func (h *failureHandler) applyOpts(opts config.OptionsConfiguration, sub events.
return url, nil, nil
}
func (h *failureHandler) addReport(data FailureData, evTime time.Time) {
func (h *failureHandler) addReport(data contract.FailureData, evTime time.Time) {
if stat, ok := h.buf[data.Description]; ok {
stat.last = evTime
stat.count++
@@ -204,7 +192,7 @@ func (*failureHandler) String() string {
return "FailureHandler"
}
func sendFailureReports(ctx context.Context, reports []FailureReport, url string) {
func sendFailureReports(ctx context.Context, reports []contract.FailureReport, url string) {
var b bytes.Buffer
if err := json.NewEncoder(&b).Encode(reports); err != nil {
panic(err)
@@ -235,8 +223,8 @@ func sendFailureReports(ctx context.Context, reports []FailureReport, url string
resp.Body.Close()
}
func newFailureReport(stat *failureStat) FailureReport {
return FailureReport{
func newFailureReport(stat *failureStat) contract.FailureReport {
return contract.FailureReport{
FailureData: stat.data,
Count: stat.count,
Version: build.LongVersion,
+55
View File
@@ -15,6 +15,7 @@ import (
"github.com/syncthing/syncthing/lib/build"
"github.com/syncthing/syncthing/lib/config"
"github.com/syncthing/syncthing/lib/fs"
)
func TestTaggedFilename(t *testing.T) {
@@ -256,3 +257,57 @@ func TestArchiveFoldersCreationPermission(t *testing.T) {
t.Errorf("földer2 permissions %v, want %v", folder2VersionsInfo.Mode(), folder2Perms)
}
}
func TestDupDirTreeWritePermissions(t *testing.T) {
// The structure should be replicated, with user permission bits set along the way
srcFs := fs.NewFilesystem(fs.FilesystemTypeFake, "TestDupDirTreeWritePermissions/srcFs")
dstFs := fs.NewFilesystem(fs.FilesystemTypeFake, "TestDupDirTreeWritePermissions/dstFs")
// A source dir to duplicate
_ = srcFs.Mkdir("foo", 0o444)
_ = srcFs.Mkdir("foo/bar", 0o555)
_ = srcFs.Mkdir("foo/bar/baz", 0o000)
// Duplication should succeed
if err := dupDirTree(srcFs, dstFs, "foo/bar/baz"); err != nil {
t.Fatal(err)
}
// Permissions should be the same, but with read/write/execute bits for
// the user
if info, err := dstFs.Lstat("foo"); err != nil || info.Mode() != 0o744 {
t.Fatalf("foo: 0o%o", info.Mode())
}
if info, err := dstFs.Lstat("foo/bar"); err != nil || info.Mode() != 0o755 {
t.Fatalf("foo/bar: 0o%o", info.Mode())
}
if info, err := dstFs.Lstat("foo/bar/baz"); err != nil || info.Mode() != 0o700 {
t.Fatalf("foo/bar/baz: 0o%o", info.Mode())
}
}
func TestDupDirFastPath(t *testing.T) {
srcFs := fs.NewFilesystem(fs.FilesystemTypeFake, "TestDupDirFastPath/srcFs")
dstFs := fs.NewFilesystem(fs.FilesystemTypeFake, "TestDupDirFastPath/dstFs")
// A source dir to duplicate
_ = srcFs.Mkdir("foo", 0o444)
_ = srcFs.Mkdir("foo/bar", 0o555)
_ = srcFs.Mkdir("foo/bar/baz", 0o000)
// The destination exists, but with too few permission bits
_ = dstFs.MkdirAll("foo/bar/baz", 0o555)
// Duplication should succeed
if err := dupDirTree(srcFs, dstFs, "foo/bar/baz"); err != nil {
t.Fatal(err)
}
// Permissions for the destination should have been updated. (This
// differs from what would have been created by the duplication of the
// 0o000 dir in the src, because it already existed.)
if info, err := dstFs.Lstat("foo/bar/baz"); err != nil || info.Mode() != 0o755 {
t.Fatalf("foo/bar/baz: 0o%o", info.Mode())
}
}
+66 -36
View File
@@ -192,48 +192,78 @@ func archiveFile(method fs.CopyRangeMethod, srcFs, dstFs fs.Filesystem, filePath
return err
}
func dupDirTree(srcFs, dstFs fs.Filesystem, folderPath string) error {
// Return early if the folder already exists.
_, err := dstFs.Stat(folderPath)
if err == nil || !fs.IsNotExist(err) {
return err
// dupDirTree ensures folderPath exists in dstFs, copying permissions mostly
// from srcFs. Permissions are altered to have the user read, write, and
// execute bits set so that Syncthing file operations are possible within
// the destination directory.
//
// We want to retain the source group and other bits so that we do not
// inadvertently open up a directory for users who shouldn't have access to
// it, but we do not consider it a security issue to open up the permissions
// for the current user.
//
// This is based on os.MkdirAll with our srcFs adjustments.
func dupDirTree(srcFs, dstFs fs.Filesystem, path string) error {
const (
allPerms = 0o777
minDirPerms = 0o700
)
// Fast path: if we can tell whether path is a directory or file, stop with success or error.
if dir, err := dstFs.Lstat(path); err == nil {
if !dir.IsDir() {
return errors.New("destination exists but is not a directory")
}
if dir.Mode()&minDirPerms != minDirPerms {
// We want all the required permission bits set
_ = dstFs.Chmod(path, dir.Mode()&allPerms|minDirPerms)
}
return nil
}
hadParent := true
for i := range folderPath {
if os.IsPathSeparator(folderPath[i]) {
// If the parent folder didn't exist, then this folder doesn't exist
// so we can skip the check
if hadParent {
_, err := dstFs.Stat(folderPath[:i])
if err == nil {
continue
}
if !fs.IsNotExist(err) {
return err
}
}
hadParent = false
err := dupDirWithPerms(srcFs, dstFs, folderPath[:i])
if err != nil {
return err
}
// Slow path: make sure parent exists and then call Mkdir for path.
// Extract the parent folder from path by first removing any trailing
// path separator and then scanning backward until finding a path
// separator or reaching the beginning of the string.
i := len(path) - 1
for i >= 0 && os.IsPathSeparator(path[i]) {
i--
}
for i >= 0 && !os.IsPathSeparator(path[i]) {
i--
}
if i < 0 {
i = 0
}
// If there is a parent directory, and it is not the volume name,
// recurse to ensure parent directory exists.
if parent := path[:i]; len(parent) > len(filepath.VolumeName(path)) {
if err := dupDirTree(srcFs, dstFs, parent); err != nil {
return err
}
}
return dupDirWithPerms(srcFs, dstFs, folderPath)
}
func dupDirWithPerms(srcFs, dstFs fs.Filesystem, folderPath string) error {
srcStat, err := srcFs.Stat(folderPath)
if err != nil {
// Parent now exists; invoke Mkdir and use its result.
srcPerms := fs.FileMode(minDirPerms)
if srcDir, err := srcFs.Lstat(path); err == nil {
srcPerms = srcDir.Mode()&allPerms | minDirPerms
}
if err := dstFs.Mkdir(path, srcPerms); err != nil {
// Handle arguments like "foo/." by
// double-checking that directory doesn't exist.
dir, err1 := dstFs.Lstat(path)
if err1 == nil && dir.IsDir() {
return nil
}
return err
}
// If we call Mkdir with srcStat.Mode(), we won't get the expected perms because of umask
// So, we create the folder with 0700, and then change the perms to the srcStat.Mode()
err = dstFs.Mkdir(folderPath, 0o700)
if err != nil {
return err
}
return dstFs.Chmod(folderPath, srcStat.Mode())
// Extra chmod to ensure our permissions override umask
_ = dstFs.Chmod(path, srcPerms)
return nil
}
func restoreFile(method fs.CopyRangeMethod, src, dst fs.Filesystem, filePath string, versionTime time.Time, tagger fileTagger) error {