Commit Graph
55 Commits
Author SHA1 Message Date
Simon Frei 71c8a2c36f fix(db): remove invalid member from FileMetadata (#10180) 2025-06-15 09:12:25 +02:00
Simon FreiandJakob Borg 8b978d4712 chore: add migration for remote invalid local flag (#10174)
Follow to resp. migration for the change in this commit:

7b319111d3

---------

Co-authored-by: Jakob Borg <jakob@kastelo.net>
2025-06-13 21:28:07 +00:00
Simon Frei 7b319111d3 fix: track invalid files in LocalFlags to fix global count (#10170)
Move the "invalid" bit to a local flag, making it easier to track in counts etc.
2025-06-13 07:33:31 +02:00
Jakob Borg e25de22705 chore(internal): linter complaints 2025-06-06 13:45:44 +02:00
Jakob Borg ef6d561c66 chore(sqlite): linter complaints 2025-06-06 13:45:44 +02:00
Jakob Borg 15f693d93c refactor(sqlite): move deleted flag into logical order in schema 2025-05-31 08:39:11 +02:00
Jakob Borg 54f6b5c2ee Merge branch 'main' into v2
* main:
  build: use specific token for pushing release tags
  fix(gui): update `uncamel()` to handle strings like 'IDs' (fixes #10128) (#10131)
  refactor: use slices package for sort (#10132)
  build: process for automatic release tags (#10133)
  chore(gui, man, authors): update docs, translations, and contributors
2025-05-26 14:22:30 +02:00
Jakob Borg 085455d72e feat: add syncthing debug database-statistics command (#10117)
This adds a command that shows database statistics. Currently it
requires a fork of the sqlite package to add the dbstats virtual table;
the modernc variant already has it.

This also provides the canonical mapping between folder ID and database
file, for tinkerers...

```
% ./bin/syncthing debug database-statistics
DATABASE                 FOLDER ID    TABLE                                  SIZE     FILL
========                 ====== ==    =====                                  ====     ====
main.db                  -            folders                               4 KiB    8.4 %
main.db                  -            folders_database_name                 4 KiB    6.0 %
main.db                  -            kv                                    4 KiB   41.1 %
main.db                  -            schemamigrations                      4 KiB    3.9 %
main.db                  -            sqlite_autoindex_folders_1            4 KiB    3.7 %
...
folder.0007-txpxsvyd.db  w3ejt-fn4dm  indexids                              4 KiB    1.5 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  kv                                    4 KiB    0.8 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  mtimes                              608 KiB   81.5 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  schemamigrations                      4 KiB    3.9 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_autoindex_blocklists_1      4108 KiB   89.5 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_autoindex_blocks_1        700020 KiB   88.1 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_autoindex_devices_1            4 KiB    3.6 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_autoindex_kv_1                 4 KiB    0.6 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_schema                        12 KiB   45.9 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_sequence                       4 KiB    1.0 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_stat1                          4 KiB   12.2 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  sqlite_stat4                          4 KiB    0.2 %
folder.0007-txpxsvyd.db  w3ejt-fn4dm  (total)                         1906020 KiB   92.8 %
main.db + children       -            (total)                         2205888 KiB   92.0 %
```
2025-05-20 14:27:08 +02:00
Simon Frei 6b94599467 chore(db, model): simplify per hash DB lookup in copier (#10080)
This is a draft because I haven't adjusted all the tests yet, I'd like
to get feedback on the change overall first, before spending time on
that.

In my opinion the main win of this change is in it's lower complexity
resp. fewer moving parts. It should also be faster as it only does one
query instead of two, but I have no idea if that's practically
relevant.

This also mirrors the v1 DB, where a block map key had the name
appended. Not that this is an argument for the change, it was mostly
reassuring me that I might not be missing something key here
conceptually (I might still be of course, please tell me :) ).

And the change isn't mainly intrinsically motivated, instead it came
up while fixing a bug in the copier. And the nested nature of that code
makes the fix harder, and "un-nesting" it required me to understand
what's happening. This change fell out of that.
2025-05-01 13:44:25 -05:00
Jakob Borg d28be1b711 fix: handle null database name in getGolderDB 2025-04-30 14:34:53 -05:00
Jakob Borg 47e3147d0b fix: don't hold main database update lock when tidying folder databases 2025-04-30 14:28:05 -05:00
Jakob Borg ed252ed6d7 fix(sqlite): hold update lock while generating folder idx 2025-04-27 23:19:07 +05:30
Jakob Borg ddea2e449c fix(db): version vector serialisation :( (#10050)
ffs
2025-04-09 17:46:49 +02:00
Jakob Borg 7cfa871d58 fix(db): skip invalid files as remote need when local is deleted 2025-04-09 16:24:17 +02:00
Tommy van der Vorst e0c1abc5fe fix(sqlite): apply options (#10049)
@calmh it seems `sqlite.Open` forgets to apply options supplied to it
(currently only the delete retention interval).
2025-04-09 06:29:46 +02:00
Jakob Borg d5aa991b73 chore(db): use pseudo random naming for folder databases 2025-04-07 11:35:31 +02:00
Jakob Borg 05210d0325 fix(db): wrong prepare method 2025-04-07 10:58:14 +02:00
Jakob Borgandbt90 cf1cf85ce6 chore(db): use one SQLite database per folder (#10042)
This changes the database structure to use one database per folder, with
a small main database to coordinate. Reverts the prior change to buffer
all files in memory when pulling, meaning there is now a phase where the
WAL file will grow significantly, at least for initial sync of folders
with many directories.

---------

Co-authored-by: bt90 <btom1990@googlemail.com>
2025-04-06 14:30:43 +02:00
Jakob Borg fa3b9acca3 chore(db): buffer pulled files for smaller WAL (#10036)
We can't hold a long select open while pulling.
2025-04-04 08:15:59 +02:00
bt90 bae976905c chore(db): fix debug logging (#10033)
Looks like a copy&paste error
2025-04-03 20:21:39 +02:00
Jakob Borg 8a2d8ebf81 chore: configurable delete retention interval (#10030)
Command line flag, as it also needs to be able to take effect during
migration.
2025-04-03 09:55:19 +02:00
Jakob Borg 82a0dd8eaa chore(db): use shorter read transactions and periodic checkpoint for smaller WAL (#10027)
Also make sure our journal size limit is in effect when checkpointing.
2025-04-02 22:19:34 +02:00
Jakob Borg 4096a35b86 fix(db): handle large numbers of blocks in update (#10025)
Avoid failure when inserting file with very large block list
2025-04-02 19:35:37 +02:00
Jakob Borg 86cbc2486f chore: forget deleted files older than six months (fixes #6284) (#10023)
This reduces the number of file entries we carry in the database,
sometimes significantly. The downside is that if a file is deleted while
a device is offline, and that device comes back more than the cutoff
interval (six months) later, those files will get resurrected at some
point.
2025-04-02 14:58:59 +02:00
bt90 0bcc31d058 chore(db): increase journal limit to 64MiB (#10022)
The current limit is far too low for our workloads. Perhaps we should
aim even higher than the 64MiB this patch proposes?

Citing the sqlite docs:

> [...] after committing a transaction the rollback journal file may
remain in the file-system. **This increases performance for subsequent
transactions since overwriting an existing file is faster than append to
a file**, but it also consumes file-system space.

tl;dr: if the limit is too low, we're shooting ourselves in the foot in
terms of performance.
2025-04-02 14:52:44 +02:00
025905fcdf chore: switch database engine to sqlite (fixes #9954) (#9965)
Switch the database from LevelDB to SQLite, for greater stability and
simpler code.

Co-authored-by: Tommy van der Vorst <tommy@pixelspark.nl>
Co-authored-by: bt90 <btom1990@googlemail.com>
2025-03-29 13:50:08 +01:00
Jakob Borg 7705a6c1f1 mv internal lib 2015-08-09 09:35:26 +02:00
Jakob Borg d3972b88f2 Remove set.ReplaceWithDelete (dead code) 2015-07-28 21:09:43 +04:00
Jakob Borg da4ebb6535 Determine conflict winner based on change type and modification time (fixes #1848) 2015-07-21 15:39:20 +02:00
Audrius Butkevicius 12a3086a9e Add label next to "Last file received" (fixes #1952) 2015-06-16 12:12:34 +01:00
Jakob Borg 76ad925842 Refactor config commit stuff to support restartless updates better
Includes restartless updates of the GUI settings (listening port etc) as
a proof of concept.
2015-06-09 15:41:22 +02:00
Jakob Borg c0d02a65c3 Reduce allocations during iteration
Reuses key byte slices to reduce allocations:

	benchmark                    old allocs     new allocs     delta
	Benchmark10kReplace          178112         168615         -5.33%
	Benchmark10kUpdateChg        567954         561557         -1.13%
	Benchmark10kUpdateSme        238691         228680         -4.19%
	Benchmark10kNeed2k           105382         103383         -1.90%
	Benchmark10kHaveFullList     60230          60230          +0.00%
	Benchmark10kGlobal           237484         227493         -4.21%

	benchmark                    old bytes     new bytes     delta
	Benchmark10kReplace          8725368       7661788       -12.19%
	Benchmark10kUpdateChg        58155152      57441025      -1.23%
	Benchmark10kUpdateSme        16130875      14996717      -7.03%
	Benchmark10kNeed2k           6561685       6338283       -3.40%
	Benchmark10kHaveFullList     7611112       7611135       +0.00%
	Benchmark10kGlobal           21415056      20300723      -5.20%
2015-05-24 13:08:14 +02:00
Jakob Borg 9d150bef9f Refactor GetMtime for early return 2015-05-14 08:26:21 +02:00
Chris Howie aa96f7b660 Virtual mtime support for environments that don't support altering mtimes (fixes #831) 2015-05-13 14:57:29 +00:00
Jakob Borg 0b9441e1a4 Reset a namespaced kv-store 2015-05-13 14:41:47 +02:00
Jakob Borg 9308c42cff Skip boring concurrency test in internal/db 2015-05-07 09:57:49 +02:00
Jakob Borg 60fcaebfdb Run vet and lint. Make us lint clean. 2015-04-29 10:38:02 +02:00
Audrius Butkevicius 433b923ea7 Add mutex logging 2015-04-23 10:54:14 +01:00
Jakob Borg d1cc1828b8 Improve ItemStarted/ItemFinished events
- Remove full details from ItemStarted (unnecessary, incorrect CamelCase)

 - Add "type" ("file" or "dir") to both events

 - Add "action" (what we tried to do - "delete" or "update") to both
   events.
2015-04-14 23:31:39 +09:00
Jakob Borg f5030f1c2c Update XDR dependency (fixes #1606) 2015-04-08 14:49:29 +02:00
Jakob Borg 6da7f17c4a Implement version vectors 2015-03-25 23:10:34 +01:00
Jakob Borg b4f45d1e79 Update tests for version vectors 2015-03-25 23:10:33 +01:00
Jakob Borg 32425c5561 MPLv2 2015-03-17 16:02:27 +01:00
Audrius Butkevicius 9d1e2d9f46 Add /rest/tree API call 2015-03-04 23:39:27 +00:00
Jakob Borg 6f8ac2b61c Refactor: add and use db.NamespacedKV 2015-02-26 09:56:11 +01:00
Jakob Borg 6f2b4b96cf Refactor: use leveldb/util.BytesPrefix 2015-02-26 09:56:11 +01:00
Jakob Borg 88c44b303d Simplify FileInfoTruncated 2015-02-15 12:50:03 +01:00
Jakob Borg 9bb928bb38 Reduce memory usage by writing smaller batches 2015-02-10 20:24:25 +01:00
Audrius Butkevicius 38eaefcabd Add ItemFinished event (fixes #1258) 2015-02-01 18:59:29 +00:00
Jakob Borg 2c8b627008 Integer type policy
Integers are for numbers, enabling arithmetic like subtractions and for
loops without getting shot in the foot. Unsigneds are for bitfields.

- "int" for numbers that will always be laughably smaller than four
  billion, and where we don't care about the serialization format.

- "int32" for numbers that will always be laughably smaller than four
  billion, and will be serialized to four bytes.

- "int64" for numbers that may approach four billion or will be
  serialized to eight bytes.

- "uint32" and "uint64" for bitfields, depending on required number of
  bits and serialization format. Likewise "uint8" and "uint16", although
  rare in this project since they don't exist in XDR.

- "int8", "int16" and plain "uint" are almost never useful.
2015-01-19 10:34:36 -08:00