From f5ec46e66dffd2712e1394d95a66f619d81ec3f4 Mon Sep 17 00:00:00 2001 From: Jakob Borg Date: Fri, 29 Apr 2016 15:34:25 +0200 Subject: [PATCH] users/syncing: Add article describing the underlying mechanisms --- draft/syncing.rst | 53 ----------------------------- index.rst | 2 ++ users/syncing.rst | 86 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 88 insertions(+), 53 deletions(-) delete mode 100644 draft/syncing.rst create mode 100644 users/syncing.rst diff --git a/draft/syncing.rst b/draft/syncing.rst deleted file mode 100644 index f67cdb69d..000000000 --- a/draft/syncing.rst +++ /dev/null @@ -1,53 +0,0 @@ -.. _syncing: - -Understanding Synchronization -============================= - -Blocks ------- - -Files are divided into *blocks*. The blocks are currently fixed size, 128 KiB, -except the last one in the file which may be smaller. Each file is sliced into -a number of these blocks, and the SHA256 hash of each block is computed. This -results in a *block list* containing the offset, size and hash of all blocks -in the file. - -To update a file, Syncthing compares the block list of the current version of -the file to the block list of the desired version of the file. It then tries -to find a source for each block that differs. This might be locally, if -another file already has a block with the same hash, or it may be from another -device in the cluster. In the first case the block is simply copied on disk, -in the second case it is requested over the network from the other device. - -When a block is copied or recieved from another device, its SHA256 hash is -computed and compared with the expected value. If it matches the block is -written to a temporary copy of the file, otherwise it is discarded and -Syncthing tries to find another source for the block. - -Scanning --------- - -Syncthing detects changes to files by scanning. By default this happens every -60 seconds, but this can be changed per folder. Increasing the scan interval -uses less resources and is useful for example on large folders that changes -infrequently. Syncthing-inotify can also be used, which tells Syncthing to -scan changed files when changes are detected, thus reducing the need for -periodic rescans. - -During a rescan the existing files are checked for changes to their -modification time or size. The file is "rehashed" if a change is detected -based on those attributes, that is a new block list is calculated for the -file. It is not possible to know which parts of a file changed without reading -the file and computing new SHA256 hashes for each block. - -Changes that were detected and hashed are transmitted to the other devices -after a rescan. - -Temporary Files ---------------- - -Syncthing never writes directly to a file. Instead all changes are made to a -temporary copy which is then moved in place over the old version. If an error -occurrs during the copying or syncing, such as a necessary block not being -available, the temporary file is kept around for up to a day. This is to avoid -needlessly requesting data over the network. diff --git a/index.rst b/index.rst index 94290966b..59750522b 100644 --- a/index.rst +++ b/index.rst @@ -63,6 +63,8 @@ can use the headings in the left sidebar to navigate the site. users/advanced users/foldermaster + users/syncing + users/firewall users/relaying users/proxying diff --git a/users/syncing.rst b/users/syncing.rst new file mode 100644 index 000000000..eb6887b78 --- /dev/null +++ b/users/syncing.rst @@ -0,0 +1,86 @@ +.. _syncing: + +Understanding Synchronization +============================= + +This article describes the mechanisms Syncthing uses to bring files in sync +on a high level. + +Blocks +------ + +Files are divided into *blocks*. The blocks are currently fixed size, 128 +KiB, except the last one in the file which may be smaller. Each file is +sliced into a number of these blocks, and the SHA256 hash of each block is +computed. This results in a *block list* containing the offset, size and +hash of all blocks in the file. + +To update a file, Syncthing compares the block list of the current version +of the file to the block list of the desired version of the file. It then +tries to find a source for each block that differs. This might be locally, +if another file already has a block with the same hash, or it may be from +another device in the cluster. In the first case the block is simply copied +on disk, in the second case it is requested over the network from the other +device. + +When a block is copied or recieved from another device, its SHA256 hash is +computed and compared with the expected value. If it matches the block is +written to a temporary copy of the file, otherwise it is discarded and +Syncthing tries to find another source for the block. + +Scanning +-------- + +Syncthing detects changes to files by scanning. By default this happens +every 60 seconds, but this can be changed per folder. Increasing the scan +interval uses less resources and is useful for example on large folders that +changes infrequently. ``syncthing-inotify`` can also be used, which tells +Syncthing to scan changed files when changes are detected, thus reducing the +need for periodic scans. + +During a rescan the existing files are checked for changes to their +modification time, size or permission bits. The file is "rehashed" if a +change is detected based on those attributes, that is a new block list is +calculated for the file. It is not possible to know which parts of a file +have changed without reading the file and computing new SHA256 hashes for +each block. + +Changes that were detected and hashed are transmitted to the other devices +after each rescan. + +Syncing +------- + +Syncthing keeps track of several version of each file - the version that it +currently has on disk, called the *local* version, the versions announced by +all other connected devices, and the "best" (usually the most recent) +version of the file. This version is called the *global* version and is the +one that each device strives to be up to date with. + +This information is kept in the *index database*, which is stored in the +configuration directory and called ``index-vx.y.z.db`` (for some version +x.y.z which may not be exactly the version of Syncthing you're running). + +When new index data is received from other devices Syncthing recalculates +which version for each file should be the global version, and compares this +to the current local version. When the two differ, Syncthing needs to +syncrhonize the file. The block lists are compared to build a list of needed +blocks, which are then requested from the network or copied locally, as +described above. + +Temporary Files +--------------- + +Syncthing never writes directly to a destination file. Instead all changes +are made to a temporary copy which is then moved in place over the old +version. If an error occurs during the copying or syncing, such as a +necessary block not being available, the temporary file is kept around for +up to a day. This is to avoid needlessly requesting data over the network. + +The temporary files are named ``.syncthing.original-filename.ext.tmp`` or, +on Windows, ``~syncthing~original-filename.ext.tmp`` where +``original-filename.ext`` is the destination filename. The temporary file is +normally hidden. If the temporary file name would be too long due to the +addition of the prefix and extra extension, a hash of the original file name +is used instead of the actual original file name. +