felixfoertsch/syncthing
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

feat: switch logging framework (#10220)

Browse Source 
This updates our logging framework from legacy freetext strings using
the `log` package to structured log entries using `log/slog`. I have
updated all INFO or higher level entries, but not yet DEBUG (😓)... So,
at a high level:

There is a slight change in log levels, effectively adding a new warning
level:

- DEBUG is still debug (ideally not for users but developers, though
this is something we need to work on)
- INFO is still info, though I've added more data here, effectively
making Syncthing more verbose by default (more on this below)
- WARNING is a new log level that is different from the _old_ WARNING
(more below)
- ERROR is what was WARNING before -- problems that must be dealt with,
and also bubbled as a popup in the GUI.

A new feature is that the logging level can be set per package to
something other than just debug or info, and hence I feel that we can
add a bit more things into INFO while moving some (in fact, most)
current INFO level warnings into WARNING. For example, I think it's
justified to get a log of synced files in INFO and sync failures in
WARNING. These are things that have historically been tricky to debug
properly, and having more information by default will be useful to many,
while still making it possible get close to told level of inscrutability
by setting the log level to WARNING. I'd like to get to a stage where
DEBUG is never necessary to just figure out what's going on, as opposed
to trying to narrow down a likely bug.

Code wise:

- Our logging object, generally known as `l` in each package, is now a
new adapter object that provides the old API on top of the newer one.
(This should go away once all old log entries are migrated.) This is
only for `l.Debugln` and `l.Debugf`.
- There is a new level tracker that keeps the log level for each
package.
- There is a nested setup of handlers, since the structure mandated by
`log/slog` is slightly convoluted (imho). We do this because we need to
do formatting at a "medium" level internally so we can buffer log lines
in text format but with separate timestamp and log level for the API/GUI
to consume.
- The `debug` API call becomes a `loglevels` API call, which can set the
log level to `DEBUG`, `INFO`, `WARNING` or `ERROR` per package. The GUI
is updated to handle this.
- Our custom `sync` package provided some debugging of mutexes quite
strongly integrated into the old logging framework, only turned on when
`STTRACE` was set to certain values at startup, etc. It's been a long
time since this has been useful; I removed it.
- The `STTRACE` env var remains and can be used the same way as before,
while additionally permitting specific log levels to be specified,
`STTRACE=model:WARN,scanner:DEBUG`.
- There is a new command line option `--log-level=INFO` to set the
default log level.
- The command line options `--log-flags` and `--verbose` go away, but
are currently retained as hidden & ignored options since we set them by
default in some of our startup examples and Syncthing would otherwise
fail to start.

Sample format messages:

```
2009-02-13 23:31:30 INF A basic info line (attr1="val with spaces" attr2=2 attr3="val\"quote" a=a log.pkg=slogutil)
2009-02-13 23:31:30 INF An info line with grouped values (attr1=val1 foo.attr2=2 foo.bar.attr3=3 a=a log.pkg=slogutil)
2009-02-13 23:31:30 INF An info line with grouped values via logger (foo.attr1=val1 foo.attr2=2 a=a log.pkg=slogutil)
2009-02-13 23:31:30 INF An info line with nested grouped values via logger (bar.foo.attr1=val1 bar.foo.attr2=2 a=a log.pkg=slogutil)
2009-02-13 23:31:30 WRN A warning entry (a=a log.pkg=slogutil)
2009-02-13 23:31:30 ERR An error (a=a log.pkg=slogutil)
```

---------

Co-authored-by: Ross Smith II <ross@smithii.com>
This commit is contained in:
Jakob Borg
2025-08-07 11:19:36 +02:00
committed by GitHub
co-authored by Ross Smith II
parent 49462448d0
commit 836045ee87
149 changed files with 1797 additions and 2641 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cmd/syncthing/blockprof.go
+5 -2
View File
@@ -8,11 +8,14 @@ package main
import (
"fmt"
"log/slog"
"os"
"runtime"
"runtime/pprof"
"syscall"
"time"
"github.com/syncthing/syncthing/internal/slogutil"
)
func startBlockProfiler() {
@@ -20,10 +23,10 @@ func startBlockProfiler() {
if profiler == nil {
panic("Couldn't find block profiler")
}
l.Debugln("Starting block profiling")
slog.Debug("Starting block profiling")
go func() {
err := saveBlockingProfiles(profiler) // Only returns on error
l.Warnln("Block profiler failed:", err)
slog.Error("Block profiler failed", slogutil.Error(err))
panic("Block profiler failed")
}()
}
cmd/syncthing/cli/utils.go
-9
View File
@@ -131,15 +131,6 @@ func prettyPrintResponse(response *http.Response) error {
return prettyPrintJSON(data)
}
func nulString(bs []byte) string {
for i := range bs {
if bs[i] == 0 {
return string(bs[:i])
}
}
return string(bs)
}
func normalizePath(path string) string {
return filepath.ToSlash(filepath.Clean(path))
}
cmd/syncthing/crash_reporting.go
+6 -3
View File
@@ -11,12 +11,15 @@ import (
"context"
"crypto/sha256"
"fmt"
"log/slog"
"net/http"
"os"
"path/filepath"
"slices"
"strings"
"time"
"github.com/syncthing/syncthing/internal/slogutil"
)
const (
@@ -33,7 +36,7 @@ const (
func uploadPanicLogs(ctx context.Context, urlBase, dir string) {
files, err := filepath.Glob(filepath.Join(dir, "panic-*.log"))
if err != nil {
l.Warnln("Failed to list panic logs:", err)
slog.ErrorContext(ctx, "Failed to list panic logs", slogutil.Error(err))
return
}
@@ -48,7 +51,7 @@ func uploadPanicLogs(ctx context.Context, urlBase, dir string) {
}
if err := uploadPanicLog(ctx, urlBase, file); err != nil {
l.Warnln("Reporting crash:", err)
slog.ErrorContext(ctx, "Reporting crash", slogutil.Error(err))
} else {
// Rename the log so we don't have to try to report it again. This
// succeeds, or it does not. There is no point complaining about it.
@@ -71,7 +74,7 @@ func uploadPanicLog(ctx context.Context, urlBase, file string) error {
data = filterLogLines(data)
hash := fmt.Sprintf("%x", sha256.Sum256(data))
l.Infof("Reporting crash found in %s (report ID %s) ...\n", filepath.Base(file), hash[:8])
slog.InfoContext(ctx, "Reporting crash", slogutil.FilePath(filepath.Base(file)), slog.String("id", hash[:8]))
url := fmt.Sprintf("%s/%s", urlBase, hash)
headReq, err := http.NewRequest(http.MethodHead, url, nil)
cmd/syncthing/debug.go
+2 -4
View File
@@ -6,8 +6,6 @@
package main
import (
"github.com/syncthing/syncthing/lib/logger"
)
import "github.com/syncthing/syncthing/internal/slogutil"
var l = logger.DefaultLogger.NewFacility("main", "Main package")
func init() { slogutil.RegisterPackage("Main package") }
cmd/syncthing/generate/generate.go
+10 -10
View File
@@ -18,9 +18,9 @@ import (
"github.com/syncthing/syncthing/lib/events"
"github.com/syncthing/syncthing/lib/fs"
"github.com/syncthing/syncthing/lib/locations"
"github.com/syncthing/syncthing/lib/logger"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/syncthing"
"golang.org/x/exp/slog"
)
type CLI struct {
@@ -29,7 +29,7 @@ type CLI struct {
NoPortProbing bool `help:"Don't try to find free ports for GUI and listen addresses on first startup" env:"STNOPORTPROBING"`
}
func (c *CLI) Run(l logger.Logger) error {
func (c *CLI) Run() error {
// Support reading the password from a pipe or similar
if c.GUIPassword == "-" {
reader := bufio.NewReader(os.Stdin)
@@ -40,13 +40,13 @@ func (c *CLI) Run(l logger.Logger) error {
c.GUIPassword = string(password)
}
if err := Generate(l, locations.GetBaseDir(locations.ConfigBaseDir), c.GUIUser, c.GUIPassword, c.NoPortProbing); err != nil {
if err := Generate(locations.GetBaseDir(locations.ConfigBaseDir), c.GUIUser, c.GUIPassword, c.NoPortProbing); err != nil {
return fmt.Errorf("failed to generate config and keys: %w", err)
}
return nil
}
func Generate(l logger.Logger, confDir, guiUser, guiPassword string, skipPortProbing bool) error {
func Generate(confDir, guiUser, guiPassword string, skipPortProbing bool) error {
dir, err := fs.ExpandTilde(confDir)
if err != nil {
return err
@@ -61,7 +61,7 @@ func Generate(l logger.Logger, confDir, guiUser, guiPassword string, skipPortPro
certFile, keyFile := locations.Get(locations.CertFile), locations.Get(locations.KeyFile)
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err == nil {
l.Warnln("Key exists; will not overwrite.")
slog.Warn("Key exists; will not overwrite.")
} else {
cert, err = syncthing.GenerateCertificate(certFile, keyFile)
if err != nil {
@@ -69,7 +69,7 @@ func Generate(l logger.Logger, confDir, guiUser, guiPassword string, skipPortPro
}
}
myID = protocol.NewDeviceID(cert.Certificate[0])
l.Infoln("Device ID:", myID)
slog.Info("Genereated new keypair", myID.LogAttr())
cfgFile := locations.Get(locations.ConfigFile)
cfg, _, err := config.Load(cfgFile, myID, events.NoopLogger)
@@ -87,7 +87,7 @@ func Generate(l logger.Logger, confDir, guiUser, guiPassword string, skipPortPro
var updateErr error
waiter, err := cfg.Modify(func(cfg *config.Configuration) {
updateErr = updateGUIAuthentication(l, &cfg.GUI, guiUser, guiPassword)
updateErr = updateGUIAuthentication(&cfg.GUI, guiUser, guiPassword)
})
if err != nil {
return fmt.Errorf("modify config: %w", err)
@@ -103,17 +103,17 @@ func Generate(l logger.Logger, confDir, guiUser, guiPassword string, skipPortPro
return nil
}
func updateGUIAuthentication(l logger.Logger, guiCfg *config.GUIConfiguration, guiUser, guiPassword string) error {
func updateGUIAuthentication(guiCfg *config.GUIConfiguration, guiUser, guiPassword string) error {
if guiUser != "" && guiCfg.User != guiUser {
guiCfg.User = guiUser
l.Infoln("Updated GUI authentication user name:", guiUser)
slog.Info("Updated GUI authentication user", "name", guiUser)
}
if guiPassword != "" && guiCfg.Password != guiPassword {
if err := guiCfg.SetPassword(guiPassword); err != nil {
return fmt.Errorf("failed to set GUI authentication password: %w", err)
}
l.Infoln("Updated GUI authentication password.")
slog.Info("Updated GUI authentication password")
}
return nil
}
cmd/syncthing/heapprof.go
+5 -2
View File
@@ -8,18 +8,21 @@ package main
import (
"fmt"
"log/slog"
"os"
"runtime"
"runtime/pprof"
"syscall"
"time"
"github.com/syncthing/syncthing/internal/slogutil"
)
func startHeapProfiler() {
l.Debugln("Starting heap profiling")
slog.Debug("Starting heap profiling")
go func() {
err := saveHeapProfiles(1) // Only returns on error
l.Warnln("Heap profiler failed:", err)
slog.Error("Heap profiler failed", slogutil.Error(err))
panic("Heap profiler failed")
}()
}
cmd/syncthing/main.go
+71 -94
View File
@@ -14,7 +14,8 @@ import (
"errors"
"fmt"
"io"
"log"
"log/slog"
"maps"
"net/http"
_ "net/http/pprof" // Need to import this to support STPROFILER.
"net/url"
@@ -40,6 +41,7 @@ import (
"github.com/syncthing/syncthing/cmd/syncthing/generate"
"github.com/syncthing/syncthing/internal/db"
"github.com/syncthing/syncthing/internal/db/sqlite"
"github.com/syncthing/syncthing/internal/slogutil"
_ "github.com/syncthing/syncthing/lib/automaxprocs"
"github.com/syncthing/syncthing/lib/build"
"github.com/syncthing/syncthing/lib/config"
@@ -47,7 +49,6 @@ import (
"github.com/syncthing/syncthing/lib/events"
"github.com/syncthing/syncthing/lib/fs"
"github.com/syncthing/syncthing/lib/locations"
"github.com/syncthing/syncthing/lib/logger"
"github.com/syncthing/syncthing/lib/osutil"
"github.com/syncthing/syncthing/lib/protocol"
"github.com/syncthing/syncthing/lib/svcutil"
@@ -61,20 +62,8 @@ const (
const (
extraUsage = `
The --logflags value is a sum of the following:
1 Date
2 Time
4 Microsecond time
8 Long filename
16 Short filename
I.e. to prefix each log line with time and filename, set --logflags=18 (2 + 16
from above). The value 0 is used to disable all of the above. The default is
to show date and time (3).
Logging always happens to the command line (stdout) and optionally to the
file at the path specified by --logfile=path. In addition to an path, the special
file at the path specified by --log-file=path. In addition to an path, the special
values "default" and "-" may be used. The former logs to DATADIR/syncthing.log
(see --data), which is the default on Windows, and the latter only to stdout,
no file, which is the default anywhere else.
@@ -87,11 +76,10 @@ The following environment variables modify Syncthing's behavior in ways that
are mostly useful for developers. Use with care. See also the --debug-* options
above.
STTRACE A comma separated string of facilities to trace. The valid
facility strings are listed below.
STLOCKTHRESHOLD Used for debugging internal deadlocks; sets debug
sensitivity. Use only under direction of a developer.
STTRACE A comma separated string of packages to trace or change log
level for. The valid package strings are listed below. A log
level (DEBUG, INFO, WARN or ERROR) can be added after each
package, separated by a colon. Ex: "model:WARN,nat:DEBUG".
STVERSIONEXTRA Add extra information to the version string in logs and the
version line in the GUI. Can be set to the name of a wrapper
@@ -106,8 +94,8 @@ above.
of CPU usage (i.e. performance).
Debugging Facilities
--------------------
Logging Facilities
------------------
The following are valid values for the STTRACE variable:
@@ -170,8 +158,9 @@ type serveCmd struct {
DBDeleteRetentionInterval time.Duration `help:"Database deleted item retention interval" default:"4320h" env:"STDBDELETERETENTIONINTERVAL"`
GUIAddress string `name:"gui-address" help:"Override GUI address (e.g. \"http://192.0.2.42:8443\")" placeholder:"URL" env:"STGUIADDRESS"`
GUIAPIKey string `name:"gui-apikey" help:"Override GUI API key" placeholder:"API-KEY" env:"STGUIAPIKEY"`
LogFile string `name:"logfile" help:"Log file name (see below)" default:"${logFile}" placeholder:"PATH" env:"STLOGFILE"`
LogFlags int `name:"logflags" help:"Select information in log line prefix (see below)" default:"${logFlags}" placeholder:"BITS" env:"STLOGFLAGS"`
LogFile string `name:"log-file" aliases:"logfile" help:"Log file name (see below)" default:"${logFile}" placeholder:"PATH" env:"STLOGFILE"`
LogFlags int `name:"logflags" help:"Deprecated option that does nothing, kept for compatibility" hidden:""`
LogLevel slog.Level `help:"Log level for all packages (DEBUG,INFO,WARN,ERROR)" env:"STLOGLEVEL" default:"INFO"`
LogMaxFiles int `name:"log-max-old-files" help:"Number of old files to keep (zero to keep only current)" default:"${logMaxFiles}" placeholder:"N" env:"STLOGMAXOLDFILES"`
LogMaxSize int `help:"Maximum size of any file (zero to disable log rotation)" default:"${logMaxSize}" placeholder:"BYTES" env:"STLOGMAXSIZE"`
NoBrowser bool `help:"Do not start browser" env:"STNOBROWSER"`
@@ -180,7 +169,6 @@ type serveCmd struct {
NoUpgrade bool `help:"Disable automatic upgrades" env:"STNOUPGRADE"`
Paused bool `help:"Start with all devices and folders paused" env:"STPAUSED"`
Unpaused bool `help:"Start with all devices and folders unpaused" env:"STUNPAUSED"`
Verbose bool `help:"Print verbose log output" env:"STVERBOSE"`
// Debug options below
DebugGUIAssetsDir string `help:"Directory to load GUI assets from" placeholder:"PATH" env:"STGUIASSETS"`
@@ -199,14 +187,9 @@ type serveCmd struct {
func defaultVars() kong.Vars {
vars := kong.Vars{}
vars["logFlags"] = strconv.Itoa(logger.DefaultFlags)
vars["logMaxSize"] = strconv.Itoa(10 << 20) // 10 MiB
vars["logMaxFiles"] = "3" // plus the current one
if os.Getenv("STTRACE") != "" {
vars["logFlags"] = strconv.Itoa(logger.DebugFlags)
}
// On non-Windows, we explicitly default to "-" which means stdout. On
// Windows, the "default" options.logFile will later be replaced with the
// default path, unless the user has manually specified "-" or
@@ -234,13 +217,13 @@ func main() {
defaultVars(),
)
if err != nil {
log.Fatal(err)
slog.Error("Parsing startup", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
kongplete.Complete(parser)
ctx, err := parser.Parse(os.Args[1:])
parser.FatalIfErrorf(err)
ctx.BindTo(l, (*logger.Logger)(nil)) // main logger available to subcommands
err = ctx.Run()
parser.FatalIfErrorf(err)
}
@@ -252,15 +235,13 @@ func helpHandler(options kong.HelpOptions, ctx *kong.Context) error {
if ctx.Command() == "serve" {
// Help was requested for `syncthing serve`, so we add our extra
// usage info afte the normal options output.
fmt.Printf(extraUsage, debugFacilities())
fmt.Printf(extraUsage, logPackages())
}
return nil
}
// serveCmd.Run() is the entrypoint for `syncthing serve`
func (c *serveCmd) Run() error {
l.SetFlags(c.LogFlags)
if c.GUIAddress != "" {
// The config picks this up from the environment.
os.Setenv("STGUIADDRESS", c.GUIAddress)
@@ -274,6 +255,9 @@ func (c *serveCmd) Run() error {
osutil.HideConsole()
}
// The default log level for all packages
slogutil.SetDefaultLevel(c.LogLevel)
// Treat an explicitly empty log file name as no log file
if c.LogFile == "" {
c.LogFile = "-"
@@ -281,7 +265,7 @@ func (c *serveCmd) Run() error {
if c.LogFile != "default" {
// We must set this *after* expandLocations above.
if err := locations.Set(locations.LogFile, c.LogFile); err != nil {
l.Warnln("Setting log file path:", err)
slog.Error("Failed to set log file path", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
}
@@ -290,7 +274,7 @@ func (c *serveCmd) Run() error {
// The asset dir is blank if STGUIASSETS wasn't set, in which case we
// should look for extra assets in the default place.
if err := locations.Set(locations.GUIAssets, c.DebugGUIAssetsDir); err != nil {
l.Warnln("Setting GUI assets path:", err)
slog.Error("Failed to set GUI assets path", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
}
@@ -298,7 +282,7 @@ func (c *serveCmd) Run() error {
// Ensure that our config and data directories exist.
for _, loc := range []locations.BaseDirEnum{locations.ConfigBaseDir, locations.DataBaseDir} {
if err := syncthing.EnsureDir(locations.GetBaseDir(loc), 0o700); err != nil {
l.Warnln("Failed to ensure directory exists:", err)
slog.Error("Failed to ensure directory exists", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
}
@@ -321,29 +305,27 @@ func openGUI() error {
return err
}
} else {
l.Warnln("Browser: GUI is currently disabled")
slog.Error("Browser: GUI is currently disabled")
}
return nil
}
func debugFacilities() string {
facilities := l.Facilities()
func logPackages() string {
packages := slogutil.PackageDescrs()
// Get a sorted list of names
var names []string
names := slices.Sorted(maps.Keys(packages))
maxLen := 0
for name := range facilities {
names = append(names, name)
for _, name := range names {
if len(name) > maxLen {
maxLen = len(name)
}
}
slices.Sort(names)
// Format the choices
b := new(bytes.Buffer)
for _, name := range names {
fmt.Fprintf(b, " %-*s - %s\n", maxLen, name, facilities[name])
fmt.Fprintf(b, " %-*s - %s\n", maxLen, name, packages[name])
}
return b.String()
}
@@ -371,7 +353,7 @@ func checkUpgrade() (upgrade.Release, error) {
return upgrade.Release{}, &errNoUpgrade{build.Version, release.Tag}
}
l.Infof("Upgrade available (current %q < latest %q)", build.Version, release.Tag)
slog.Info("Upgrade available", "current", build.Version, "latest", release.Tag)
return release, nil
}
@@ -428,13 +410,9 @@ func (c *serveCmd) syncthingMain() {
startPerfStats()
}
// Set a log prefix similar to the ID we will have later on, or early log
// lines look ugly.
l.SetPrefix("[start] ")
// Print our version information up front, so any crash that happens
// early etc. will have it available.
l.Infoln(build.LongVersion)
slog.Info(build.LongVersion) //nolint:sloglint
// Ensure that we have a certificate and key.
cert, err := syncthing.LoadOrGenerateCertificate(
@@ -442,7 +420,7 @@ func (c *serveCmd) syncthingMain() {
locations.Get(locations.KeyFile),
)
if err != nil {
l.Warnln("Failed to load/generate certificate:", err)
slog.Error("Failed to load/generate certificate", slogutil.Error(err))
os.Exit(1)
}
@@ -450,10 +428,10 @@ func (c *serveCmd) syncthingMain() {
lf := flock.New(locations.Get(locations.LockFile))
locked, err := lf.TryLock()
if err != nil {
l.Warnln("Failed to acquire lock:", err)
slog.Error("Failed to acquire lock", slogutil.Error(err))
os.Exit(1)
} else if !locked {
l.Warnln("Failed to acquire lock: is another Syncthing instance already running?")
slog.Error("Failed to acquire lock: is another Syncthing instance already running?")
os.Exit(1)
}
@@ -462,7 +440,7 @@ func (c *serveCmd) syncthingMain() {
// earlyService is a supervisor that runs the services needed for or
// before app startup; the event logger, and the config service.
spec := svcutil.SpecWithDebugLogger(l)
spec := svcutil.SpecWithDebugLogger()
earlyService := suture.New("early", spec)
earlyService.ServeBackground(ctx)
@@ -471,7 +449,7 @@ func (c *serveCmd) syncthingMain() {
cfgWrapper, err := syncthing.LoadConfigAtStartup(locations.Get(locations.ConfigFile), cert, evLogger, c.AllowNewerConfig, c.NoPortProbing)
if err != nil {
l.Warnln("Failed to initialize config:", err)
slog.Error("Failed to initialize config", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
earlyService.Add(cfgWrapper)
@@ -483,7 +461,7 @@ func (c *serveCmd) syncthingMain() {
if build.IsCandidate && !upgrade.DisabledByCompilation && !c.NoUpgrade {
cfgWrapper.Modify(func(cfg *config.Configuration) {
l.Infoln("Automatic upgrade is always enabled for candidate releases.")
slog.Info("Automatic upgrade is always enabled for candidate releases")
if cfg.Options.AutoUpgradeIntervalH == 0 || cfg.Options.AutoUpgradeIntervalH > 24 {
cfg.Options.AutoUpgradeIntervalH = 12
// Set the option into the config as well, as the auto upgrade
@@ -495,13 +473,13 @@ func (c *serveCmd) syncthingMain() {
}
if err := syncthing.TryMigrateDatabase(c.DBDeleteRetentionInterval); err != nil {
l.Warnln("Failed to migrate old-style database:", err)
slog.Error("Failed to migrate old-style database", slogutil.Error(err))
os.Exit(1)
}
sdb, err := syncthing.OpenDatabase(locations.Get(locations.Database), c.DBDeleteRetentionInterval)
if err != nil {
l.Warnln("Error opening database:", err)
slog.Error("Error opening database", slogutil.Error(err))
os.Exit(1)
}
@@ -518,12 +496,12 @@ func (c *serveCmd) syncthingMain() {
}
if err != nil {
if _, ok := err.(*errNoUpgrade); ok || err == errTooEarlyUpgradeCheck || err == errTooEarlyUpgrade {
l.Debugln("Initial automatic upgrade:", err)
slog.Debug("Initial automatic upgrade", slogutil.Error(err))
} else {
l.Infoln("Initial automatic upgrade:", err)
slog.Info("Initial automatic upgrade", slogutil.Error(err))
}
} else {
l.Infof("Upgraded to %q, should exit now.", release.Tag)
slog.Info("Upgraded, should exit now", "newVersion", release.Tag)
os.Exit(svcutil.ExitUpgrade.AsInt())
}
}
@@ -538,18 +516,17 @@ func (c *serveCmd) syncthingMain() {
NoUpgrade: c.NoUpgrade,
ProfilerAddr: c.DebugProfilerListen,
ResetDeltaIdxs: c.DebugResetDeltaIdxs,
Verbose: c.Verbose,
DBMaintenanceInterval: c.DBMaintenanceInterval,
}
if c.Audit || cfgWrapper.Options().AuditEnabled {
l.Infoln("Auditing is enabled.")
slog.Info("Auditing is enabled")
auditFile := cfgWrapper.Options().AuditFile
// Ignore config option if command-line option is set
if c.AuditFile != "" {
l.Debugln("Using the audit file from the command-line parameter.")
slog.Debug("Using the audit file from the command-line parameter", slogutil.FilePath(c.AuditFile))
auditFile = c.AuditFile
}
@@ -558,7 +535,7 @@ func (c *serveCmd) syncthingMain() {
app, err := syncthing.New(cfgWrapper, sdb, evLogger, cert, appOpts)
if err != nil {
l.Warnln("Failed to start Syncthing:", err)
slog.Error("Failed to start Syncthing", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
@@ -571,11 +548,11 @@ func (c *serveCmd) syncthingMain() {
if c.DebugProfileCPU {
f, err := os.Create(fmt.Sprintf("cpu-%d.pprof", os.Getpid()))
if err != nil {
l.Warnln("Creating profile:", err)
slog.Error("Failed to create profile", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
if err := pprof.StartCPUProfile(f); err != nil {
l.Warnln("Starting profile:", err)
slog.Error("Failed to start profile", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
}
@@ -595,7 +572,7 @@ func (c *serveCmd) syncthingMain() {
status := app.Wait()
if status == svcutil.ExitError {
l.Warnln("Syncthing stopped with error:", app.Error())
slog.Error("Syncthing stopped with error", slogutil.Error(app.Error()))
}
if c.DebugProfileCPU {
@@ -663,13 +640,13 @@ func auditWriter(auditFile string) io.Writer {
}
fd, err = os.OpenFile(auditFile, auditFlags, 0o600)
if err != nil {
l.Warnln("Audit:", err)
slog.Error("Failed to open audit file", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
auditDest = auditFile
}
l.Infoln("Audit log in", auditDest)
slog.Info("Writing audit log", slogutil.FilePath(auditDest))
return fd
}
@@ -679,7 +656,7 @@ func (c *serveCmd) autoUpgradePossible() bool {
return false
}
if c.NoUpgrade {
l.Infof("No automatic upgrades; STNOUPGRADE environment variable defined.")
slog.Info("No automatic upgrades; STNOUPGRADE environment variable defined")
return false
}
return true
@@ -696,7 +673,7 @@ func autoUpgrade(cfg config.Wrapper, app *syncthing.App, evLogger events.Logger)
continue
}
if cfg.Options().AutoUpgradeEnabled() {
l.Infof("Connected to device %s with a newer version (current %q < remote %q). Checking for upgrades.", data["id"], build.Version, data["clientVersion"])
slog.Info("Connected to device with a newer version; checking for upgrades", slog.String("device", data["id"]), slog.String("ourVersion", build.Version), slog.String("theirVersion", data["clientVersion"]))
}
case <-timer.C:
}
@@ -716,7 +693,7 @@ func autoUpgrade(cfg config.Wrapper, app *syncthing.App, evLogger events.Logger)
if err != nil {
// Don't complain too loudly here; we might simply not have
// internet connectivity, or the upgrade server might be down.
l.Infoln("Automatic upgrade:", err)
slog.Info("Automatic upgrade", slogutil.Error(err))
timer.Reset(checkInterval)
continue
}
@@ -727,15 +704,15 @@ func autoUpgrade(cfg config.Wrapper, app *syncthing.App, evLogger events.Logger)
continue
}
l.Infof("Automatic upgrade (current %q < latest %q)", build.Version, rel.Tag)
slog.Info("Automatic upgrade", "current", build.Version, "latest", rel.Tag)
err = upgrade.To(rel)
if err != nil {
l.Warnln("Automatic upgrade:", err)
slog.Error("Automatic upgrade failed", slogutil.Error(err))
timer.Reset(checkInterval)
continue
}
sub.Unsubscribe()
l.Warnf("Automatically upgraded to version %q. Restarting in 1 minute.", rel.Tag)
slog.Error("Automatically upgraded, restarting in 1 minute", slog.String("newVersion", rel.Tag))
time.Sleep(time.Minute)
app.Stop(svcutil.ExitUpgrade)
return
@@ -788,22 +765,22 @@ func cleanConfigDirectory() {
fs := fs.NewFilesystem(fs.FilesystemTypeBasic, locations.GetBaseDir(locations.ConfigBaseDir))
files, err := fs.Glob(pat)
if err != nil {
l.Infoln("Cleaning:", err)
slog.Warn("Failed to clean config directory", slogutil.Error(err))
continue
}
for _, file := range files {
info, err := fs.Lstat(file)
if err != nil {
l.Infoln("Cleaning:", err)
slog.Warn("Failed to clean config directory", slogutil.Error(err))
continue
}
if time.Since(info.ModTime()) > dur {
if err = fs.RemoveAll(file); err != nil {
l.Infoln("Cleaning:", err)
slog.Warn("Failed to clean config directory", slogutil.Error(err))
} else {
l.Infoln("Cleaned away old file", filepath.Base(file))
slog.Warn("Cleaned away old file", slogutil.FilePath(filepath.Base(file)))
}
}
}
@@ -820,7 +797,7 @@ func setPauseState(cfgWrapper config.Wrapper, paused bool) {
}
})
if err != nil {
l.Warnln("Cannot adjust paused state:", err)
slog.Error("Cannot adjust paused state", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
}
@@ -847,7 +824,7 @@ func (deviceIDCmd) Run() error {
locations.Get(locations.KeyFile),
)
if err != nil {
l.Warnln("Error reading device ID:", err)
slog.Error("Failed to read device ID", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
@@ -870,7 +847,7 @@ type upgradeCmd struct {
func (u upgradeCmd) Run() error {
if u.CheckOnly {
if _, err := checkUpgrade(); err != nil {
l.Warnln("Checking for upgrade:", err)
slog.Error("Failed to check for upgrade", slogutil.Error(err))
os.Exit(exitCodeForUpgrade(err))
}
return nil
@@ -879,10 +856,10 @@ func (u upgradeCmd) Run() error {
if u.From != "" {
err := upgrade.ToURL(u.From)
if err != nil {
l.Warnln("Error while Upgrading:", err)
slog.Error("Failed to upgrade", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
l.Infoln("Upgraded from", u.From)
slog.Info("Upgraded", "from", u.From)
return nil
}
@@ -892,7 +869,7 @@ func (u upgradeCmd) Run() error {
var locked bool
locked, err = lf.TryLock()
if err != nil {
l.Warnln("Upgrade:", err)
slog.Error("Failed to lock for upgrade", slogutil.Error(err))
os.Exit(1)
} else if locked {
err = upgradeViaRest()
@@ -901,10 +878,10 @@ func (u upgradeCmd) Run() error {
}
}
if err != nil {
l.Warnln("Upgrade:", err)
slog.Error("Failed to check for upgrade", slogutil.Error(err))
os.Exit(exitCodeForUpgrade(err))
}
l.Infof("Upgraded to %q", release.Tag)
slog.Info("Upgraded", "to", release.Tag)
os.Exit(svcutil.ExitUpgrade.AsInt())
return nil
}
@@ -913,7 +890,7 @@ type browserCmd struct{}
func (browserCmd) Run() error {
if err := openGUI(); err != nil {
l.Warnln("Failed to open web UI:", err)
slog.Error("Failed to open web UI", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
return nil
@@ -929,12 +906,12 @@ type debugCmd struct {
type resetDatabaseCmd struct{}
func (resetDatabaseCmd) Run() error {
l.Infoln("Removing database in", locations.Get(locations.Database))
slog.Info("Removing database", slogutil.FilePath(locations.Get(locations.Database)))
if err := os.RemoveAll(locations.Get(locations.Database)); err != nil {
l.Warnln("Resetting database:", err)
slog.Error("Failed to reset database", slogutil.Error(err))
os.Exit(svcutil.ExitError.AsInt())
}
l.Infoln("Successfully reset database - it will be rebuilt after next start.")
slog.Info("Reset database - it will be rebuilt after next start")
return nil
}
cmd/syncthing/monitor.go
+22 -39
View File
@@ -11,26 +11,28 @@ import (
"context"
"fmt"
"io"
"log/slog"
"os"
"os/exec"
"os/signal"
"path/filepath"
"strings"
"sync"
"syscall"
"time"
"github.com/syncthing/syncthing/internal/slogutil"
"github.com/syncthing/syncthing/lib/build"
"github.com/syncthing/syncthing/lib/fs"
"github.com/syncthing/syncthing/lib/locations"
"github.com/syncthing/syncthing/lib/osutil"
"github.com/syncthing/syncthing/lib/svcutil"
"github.com/syncthing/syncthing/lib/sync"
)
var (
stdoutFirstLines []string // The first 10 lines of stdout
stdoutLastLines []string // The last 50 lines of stdout
stdoutMut = sync.NewMutex()
stdoutMut sync.Mutex
)
const (
@@ -44,8 +46,6 @@ const (
)
func (c *serveCmd) monitorMain() {
l.SetPrefix("[monitor] ")
var dst io.Writer = os.Stdout
logFile := locations.Get(locations.LogFile)
@@ -64,7 +64,7 @@ func (c *serveCmd) monitorMain() {
fileDst, err = open(logFile)
}
if err != nil {
l.Warnln("Failed to set up logging to file, proceeding with logging to stdout only:", err)
slog.Error("Failed to set up logging to file, proceeding with logging to stdout only", slogutil.Error(err))
} else {
if build.IsWindows {
// Translate line breaks to Windows standard
@@ -78,14 +78,14 @@ func (c *serveCmd) monitorMain() {
// Log to both stdout and file.
dst = io.MultiWriter(dst, fileDst)
l.Infof(`Log output saved to file "%s"`, logFile)
slog.Info("Saved log output", slogutil.FilePath(logFile))
}
}
args := os.Args
binary, err := getBinary(args[0])
if err != nil {
l.Warnln("Error starting the main Syncthing process:", err)
slog.Error("Failed to start the main Syncthing process", slogutil.Error(err))
panic("Error starting the main Syncthing process")
}
var restarts [restartCounts]time.Time
@@ -102,7 +102,7 @@ func (c *serveCmd) monitorMain() {
maybeReportPanics()
if t := time.Since(restarts[0]); t < restartLoopThreshold {
l.Warnf("%d restarts in %v; not retrying further", restartCounts, t)
slog.Error("Too many restarts; not retrying further", slog.Int("count", restartCounts), slog.Any("interval", t))
os.Exit(svcutil.ExitError.AsInt())
}
@@ -122,10 +122,10 @@ func (c *serveCmd) monitorMain() {
panic(err)
}
l.Debugln("Starting syncthing")
slog.Debug("Starting syncthing")
err = cmd.Start()
if err != nil {
l.Warnln("Error starting the main Syncthing process:", err)
slog.Error("Failed to start the main Syncthing process", slogutil.Error(err))
panic("Error starting the main Syncthing process")
}
@@ -134,7 +134,7 @@ func (c *serveCmd) monitorMain() {
stdoutLastLines = make([]string, 0, 50)
stdoutMut.Unlock()
wg := sync.NewWaitGroup()
var wg sync.WaitGroup
wg.Add(1)
go func() {
@@ -158,13 +158,13 @@ func (c *serveCmd) monitorMain() {
stopped := false
select {
case s := <-stopSign:
l.Infof("Signal %d received; exiting", s)
slog.Info("Received signal; exiting", "signal", s)
cmd.Process.Signal(sigTerm)
err = <-exit
stopped = true
case s := <-restartSign:
l.Infof("Signal %d received; restarting", s)
slog.Info("Received signal; restarting", "signal", s)
cmd.Process.Signal(sigHup)
err = <-exit
@@ -184,9 +184,9 @@ func (c *serveCmd) monitorMain() {
if exitCode == svcutil.ExitUpgrade.AsInt() {
// Restart the monitor process to release the .old
// binary as part of the upgrade process.
l.Infoln("Restarting monitor...")
slog.Info("Restarting monitor...")
if err = restartMonitor(binary, args); err != nil {
l.Warnln("Restart:", err)
slog.Error("Failed to restart monitor", slogutil.Error(err))
}
os.Exit(exitCode)
}
@@ -196,7 +196,7 @@ func (c *serveCmd) monitorMain() {
os.Exit(svcutil.ExitError.AsInt())
}
l.Infoln("Syncthing exited:", err)
slog.Info("Syncthing exited", slogutil.Error(err))
time.Sleep(restartPause)
if first {
@@ -243,29 +243,13 @@ func copyStderr(stderr io.Reader, dst io.Writer) {
if panicFd == nil && (strings.HasPrefix(line, "panic:") || strings.HasPrefix(line, "fatal error:")) {
panicFd, err = os.Create(locations.GetTimestamped(locations.PanicLog))
if err != nil {
l.Warnln("Create panic log:", err)
slog.Error("Failed to create panic log", slogutil.Error(err))
continue
}
l.Warnf("Panic detected, writing to \"%s\"", panicFd.Name())
if strings.Contains(line, "leveldb") && strings.Contains(line, "corrupt") {
l.Warnln(`
*********************************************************************************
* Crash due to corrupt database. *
* *
* This crash usually occurs due to one of the following reasons: *
* - Syncthing being stopped abruptly (killed/loss of power) *
* - Bad hardware (memory/disk issues) *
* - Software that affects disk writes (SSD caching software and similar) *
* *
* Please see the following URL for instructions on how to recover: *
* https://docs.syncthing.net/users/faq.html#my-syncthing-database-is-corrupt *
*********************************************************************************
`)
} else {
l.Warnln("Please check for existing issues with similar panic message at https://github.com/syncthing/syncthing/issues/")
l.Warnln("If no issue with similar panic message exists, please create a new issue with the panic log attached")
}
slog.Error("Panic detected, writing to file", slogutil.FilePath(panicFd.Name()))
slog.Info("Please check for existing issues with similar panic message at https://github.com/syncthing/syncthing/issues/")
slog.Info("If no issue with similar panic message exists, please create a new issue with the panic log attached")
stdoutMut.Lock()
for _, line := range stdoutFirstLines {
@@ -446,7 +430,6 @@ func newAutoclosedFile(name string, closeDelay, maxOpenTime time.Duration) (*aut
name: name,
closeDelay: closeDelay,
maxOpenTime: maxOpenTime,
mut: sync.NewMutex(),
closed: make(chan struct{}),
closeTimer: time.NewTimer(time.Minute),
}
@@ -554,7 +537,7 @@ func maybeReportPanics() {
// Try to get a config to see if/where panics should be reported.
cfg, err := loadOrDefaultConfig()
if err != nil {
l.Warnln("Couldn't load config; not reporting crash")
slog.Error("Couldn't load config; not reporting crash")
return
}
@@ -574,7 +557,7 @@ func maybeReportPanics() {
case <-ctx.Done():
return
case <-time.After(panicUploadNoticeWait):
l.Warnln("Uploading crash reports is taking a while, please wait...")
slog.Warn("Uploading crash reports is taking a while, please wait")
}
}()