felixfoertsch/syncthing
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity
7,850 Commits 9 Branches 685 Tags
ff88430efb74975a4fd431dfe2b83be40947a5c8
Commit Graph
57 Commits
Author SHA1 Message Date
Jakob Borg ff88430efb feat: add debug commands for folder counts and files (#10206) 
This adds two debugging commands that print information directly from
the database; one for folder counts, and one for file metadata for files
matching a pattern in a folder. E.g.,

```
% syncthing debug database-counts p3jms-73gps
DEVICE   TYPE       FLAGS    DELETED  COUNT  SIZE
-local-  FILE       -------  ---      0      0
-local-  FILE       --G----  ---      2473   70094796496
-local-  DIRECTORY  -------  ---      0      0
-local-  DIRECTORY  --G----  ---      19     2432
PSEUDOP  FILE       -------  ---      2473   70094796496
PSEUDOP  FILE       -nG----  ---      0      0
PSEUDOP  DIRECTORY  -------  ---      19     2432
PSEUDOP  DIRECTORY  -nG----  ---      0      0
```

```
% syncthing debug database-file p3jms-73gps 20240929-DSCF1387
DSCF1387
DEVICE   TYPE  NAME                          SEQUENCE  DELETED  MODIFIED                        SIZE      FLAGS    VERSION             BLOCKLIST
-local-  FILE  Austin/20240929-DSCF1387.raf  1204      ---      2024-09-29T01:10:54Z            48911888  --G----  HX2ELNU:1744213700  fsQdMvUL
PSEUDOP  FILE  Austin/20240929-DSCF1387.raf  22279     ---      2024-09-29T01:10:54Z            48911888  -------  HX2ELNU:1744213700  fsQdMvUL
-local-  FILE  Austin/20240929-DSCF1387.xmp  1196      ---      2024-10-16T08:08:35.137501751Z  5579      --G----  HX2ELNU:1744213700  xDGMnepi
PSEUDOP  FILE  Austin/20240929-DSCF1387.xmp  19910     ---      2024-10-16T08:08:35.137501751Z  5579      -------  HX2ELNU:1744213700  xDGMnepi
```

The local flag bits get a string representation for the bitmask,

```
	FlagLocalUnsupported:   "u",
	FlagLocalIgnored:       "i",
	FlagLocalMustRescan:    "r",
	FlagLocalReceiveOnly:   "e",
	FlagLocalGlobal:        "G",
	FlagLocalNeeded:        "n",
	FlagLocalRemoteInvalid: "v",
```
 2025-07-04 15:46:24 +02:00
Jakob Borg c5a29b5b26 fix(model): don't clobber local flags when receiving index (#10190) 2025-06-20 07:08:06 +00:00
Simon Frei 71c8a2c36f fix(db): remove invalid member from FileMetadata (#10180) 2025-06-15 09:12:25 +02:00
Simon FreiandJakob Borg 8b978d4712 chore: add migration for remote invalid local flag (#10174) 
Follow to resp. migration for the change in this commit:

7b319111d3

---------

Co-authored-by: Jakob Borg <jakob@kastelo.net>
 2025-06-13 21:28:07 +00:00
Simon Frei 7b319111d3 fix: track invalid files in LocalFlags to fix global count (#10170) 
Move the "invalid" bit to a local flag, making it easier to track in counts etc.
 2025-06-13 07:33:31 +02:00
Jakob Borg e25de22705 chore(internal): linter complaints 2025-06-06 13:45:44 +02:00
Jakob Borg ef6d561c66 chore(sqlite): linter complaints 2025-06-06 13:45:44 +02:00
Jakob Borg 15f693d93c refactor(sqlite): move deleted flag into logical order in schema 2025-05-31 08:39:11 +02:00
Jakob Borg 54f6b5c2ee Merge branch 'main' into v2 
* main:
  build: use specific token for pushing release tags
  fix(gui): update `uncamel()` to handle strings like 'IDs' (fixes #10128) (#10131)
  refactor: use slices package for sort (#10132)
  build: process for automatic release tags (#10133)
  chore(gui, man, authors): update docs, translations, and contributors
 2025-05-26 14:22:30 +02:00
Jakob Borg 085455d72e feat: add syncthing debug database-statistics command (#10117) 
This adds a command that shows database statistics. Currently it
requires a fork of the sqlite package to add the dbstats virtual table;
the modernc variant already has it.

This also provides the canonical mapping between folder ID and database
file, for tinkerers...

```
% ./bin/syncthing debug database-statistics
DATABASE                 FOLDER ID    TABLE                                  SIZE     FILL
========                 ====== ==    =====                                  ====     ====
main.db                  -            folders                               4 KiB    8.4 %
main.db                  -            folders_database_name                 4 KiB    6.0 %
main.db                  -            kv                                    4 KiB   41.1 %
main.db                  -            schemamigrations                      4 KiB    3.9 %
main.db                  -            sqlite_autoindex_folders_1            4 KiB    3.7 %
...
folder.0007-txpxsvyd.db  w3ejt-fn4dm  indexids                              4 KiB    1.5 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  kv                                    4 KiB    0.8 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  mtimes                              608 KiB   81.5 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  schemamigrations                      4 KiB    3.9 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_autoindex_blocklists_1      4108 KiB   89.5 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_autoindex_blocks_1        700020 KiB   88.1 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_autoindex_devices_1            4 KiB    3.6 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_autoindex_kv_1                 4 KiB    0.6 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_schema                        12 KiB   45.9 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_sequence                       4 KiB    1.0 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_stat1                          4 KiB   12.2 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_stat4                          4 KiB    0.2 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  (total)                         1906020 KiB   92.8 %
main.db + children       -            (total)                         2205888 KiB   92.0 %
```
 2025-05-20 14:27:08 +02:00
Simon Frei 6b94599467 chore(db, model): simplify per hash DB lookup in copier (#10080) 
This is a draft because I haven't adjusted all the tests yet, I'd like
to get feedback on the change overall first, before spending time on
that.

In my opinion the main win of this change is in it's lower complexity
resp. fewer moving parts. It should also be faster as it only does one
query instead of two, but I have no idea if that's practically
relevant.

This also mirrors the v1 DB, where a block map key had the name
appended. Not that this is an argument for the change, it was mostly
reassuring me that I might not be missing something key here
conceptually (I might still be of course, please tell me :) ).

And the change isn't mainly intrinsically motivated, instead it came
up while fixing a bug in the copier. And the nested nature of that code
makes the fix harder, and "un-nesting" it required me to understand
what's happening. This change fell out of that.
 2025-05-01 13:44:25 -05:00
Jakob Borg d28be1b711 fix: handle null database name in getGolderDB 2025-04-30 14:34:53 -05:00
Jakob Borg 47e3147d0b fix: don't hold main database update lock when tidying folder databases 2025-04-30 14:28:05 -05:00
Jakob Borg ed252ed6d7 fix(sqlite): hold update lock while generating folder idx 2025-04-27 23:19:07 +05:30
Jakob Borg ddea2e449c fix(db): version vector serialisation :( (#10050) 
ffs
 2025-04-09 17:46:49 +02:00
Jakob Borg 7cfa871d58 fix(db): skip invalid files as remote need when local is deleted 2025-04-09 16:24:17 +02:00
Tommy van der Vorst e0c1abc5fe fix(sqlite): apply options (#10049) 
@calmh it seems `sqlite.Open` forgets to apply options supplied to it
(currently only the delete retention interval).
 2025-04-09 06:29:46 +02:00
Jakob Borg d5aa991b73 chore(db): use pseudo random naming for folder databases 2025-04-07 11:35:31 +02:00
Jakob Borg 05210d0325 fix(db): wrong prepare method 2025-04-07 10:58:14 +02:00
Jakob Borgandbt90 cf1cf85ce6 chore(db): use one SQLite database per folder (#10042) 
This changes the database structure to use one database per folder, with
a small main database to coordinate. Reverts the prior change to buffer
all files in memory when pulling, meaning there is now a phase where the
WAL file will grow significantly, at least for initial sync of folders
with many directories.

---------

Co-authored-by: bt90 <btom1990@googlemail.com>
 2025-04-06 14:30:43 +02:00
Jakob Borg fa3b9acca3 chore(db): buffer pulled files for smaller WAL (#10036) 
We can't hold a long select open while pulling.
 2025-04-04 08:15:59 +02:00
bt90 bae976905c chore(db): fix debug logging (#10033) 
Looks like a copy&paste error
 2025-04-03 20:21:39 +02:00
Jakob Borg 8a2d8ebf81 chore: configurable delete retention interval (#10030) 
Command line flag, as it also needs to be able to take effect during
migration.
 2025-04-03 09:55:19 +02:00
Jakob Borg 82a0dd8eaa chore(db): use shorter read transactions and periodic checkpoint for smaller WAL (#10027) 
Also make sure our journal size limit is in effect when checkpointing.
 2025-04-02 22:19:34 +02:00
Jakob Borg 4096a35b86 fix(db): handle large numbers of blocks in update (#10025) 
Avoid failure when inserting file with very large block list
 2025-04-02 19:35:37 +02:00
Jakob Borg 86cbc2486f chore: forget deleted files older than six months (fixes #6284) (#10023) 
This reduces the number of file entries we carry in the database,
sometimes significantly. The downside is that if a file is deleted while
a device is offline, and that device comes back more than the cutoff
interval (six months) later, those files will get resurrected at some
point.
 2025-04-02 14:58:59 +02:00
bt90 0bcc31d058 chore(db): increase journal limit to 64MiB (#10022) 
The current limit is far too low for our workloads. Perhaps we should
aim even higher than the 64MiB this patch proposes?

Citing the sqlite docs:

> [...] after committing a transaction the rollback journal file may
remain in the file-system. **This increases performance for subsequent
transactions since overwriting an existing file is faster than append to
a file**, but it also consumes file-system space.

tl;dr: if the limit is too low, we're shooting ourselves in the foot in
terms of performance.
 2025-04-02 14:52:44 +02:00
Jakob BorgTommy van der Vorstbt90
 		025905fcdf chore: switch database engine to sqlite (fixes #9954) (#9965) 
Switch the database from LevelDB to SQLite, for greater stability and
simpler code.

Co-authored-by: Tommy van der Vorst <tommy@pixelspark.nl>
Co-authored-by: bt90 <btom1990@googlemail.com>
 2025-03-29 13:50:08 +01:00
Jakob Borg 7705a6c1f1 mv internal lib 2015-08-09 09:35:26 +02:00
Jakob Borg d3972b88f2 Remove set.ReplaceWithDelete (dead code) 2015-07-28 21:09:43 +04:00
Jakob Borg da4ebb6535 Determine conflict winner based on change type and modification time (fixes #1848) 2015-07-21 15:39:20 +02:00
Audrius Butkevicius 12a3086a9e Add label next to "Last file received" (fixes #1952) 2015-06-16 12:12:34 +01:00
Jakob Borg 76ad925842 Refactor config commit stuff to support restartless updates better 
Includes restartless updates of the GUI settings (listening port etc) as
a proof of concept.
 2015-06-09 15:41:22 +02:00
Jakob Borg c0d02a65c3 Reduce allocations during iteration 
Reuses key byte slices to reduce allocations:

	benchmark                    old allocs     new allocs     delta
	Benchmark10kReplace          178112         168615         -5.33%
	Benchmark10kUpdateChg        567954         561557         -1.13%
	Benchmark10kUpdateSme        238691         228680         -4.19%
	Benchmark10kNeed2k           105382         103383         -1.90%
	Benchmark10kHaveFullList     60230          60230          +0.00%
	Benchmark10kGlobal           237484         227493         -4.21%

	benchmark                    old bytes     new bytes     delta
	Benchmark10kReplace          8725368       7661788       -12.19%
	Benchmark10kUpdateChg        58155152      57441025      -1.23%
	Benchmark10kUpdateSme        16130875      14996717      -7.03%
	Benchmark10kNeed2k           6561685       6338283       -3.40%
	Benchmark10kHaveFullList     7611112       7611135       +0.00%
	Benchmark10kGlobal           21415056      20300723      -5.20%
 2015-05-24 13:08:14 +02:00
Jakob Borg 9d150bef9f Refactor GetMtime for early return 2015-05-14 08:26:21 +02:00
Chris Howie aa96f7b660 Virtual mtime support for environments that don't support altering mtimes (fixes #831) 2015-05-13 14:57:29 +00:00
Jakob Borg 0b9441e1a4 Reset a namespaced kv-store 2015-05-13 14:41:47 +02:00
Jakob Borg 9308c42cff Skip boring concurrency test in internal/db 2015-05-07 09:57:49 +02:00
Jakob Borg 60fcaebfdb Run vet and lint. Make us lint clean. 2015-04-29 10:38:02 +02:00
Audrius Butkevicius 433b923ea7 Add mutex logging 2015-04-23 10:54:14 +01:00
Jakob Borg d1cc1828b8 Improve ItemStarted/ItemFinished events 
- Remove full details from ItemStarted (unnecessary, incorrect CamelCase)

 - Add "type" ("file" or "dir") to both events

 - Add "action" (what we tried to do - "delete" or "update") to both
   events.
 2015-04-14 23:31:39 +09:00
Jakob Borg f5030f1c2c Update XDR dependency (fixes #1606) 2015-04-08 14:49:29 +02:00
Jakob Borg 6da7f17c4a Implement version vectors 2015-03-25 23:10:34 +01:00
Jakob Borg b4f45d1e79 Update tests for version vectors 2015-03-25 23:10:33 +01:00
Jakob Borg 32425c5561 MPLv2 2015-03-17 16:02:27 +01:00
Audrius Butkevicius 9d1e2d9f46 Add /rest/tree API call 2015-03-04 23:39:27 +00:00
Jakob Borg 6f8ac2b61c Refactor: add and use db.NamespacedKV 2015-02-26 09:56:11 +01:00
Jakob Borg 6f2b4b96cf Refactor: use leveldb/util.BytesPrefix 2015-02-26 09:56:11 +01:00
Jakob Borg 88c44b303d Simplify FileInfoTruncated 2015-02-15 12:50:03 +01:00
Jakob Borg 9bb928bb38 Reduce memory usage by writing smaller batches 2015-02-10 20:24:25 +01:00