From a9b3fef697ed0c546ee10594878fa61be0579e69 Mon Sep 17 00:00:00 2001 From: Jakob Borg Date: Mon, 4 Jul 2016 11:56:58 +0000 Subject: [PATCH] specs: Update specs for protocol buffer formats GitHub-Pull-Request: https://github.com/syncthing/docs/pull/190 --- specs/bep-v1.rst | 1332 +++++++++++++-------------------------- specs/localdisco-v3.rst | 120 ---- specs/localdisco-v4.rst | 77 +++ 3 files changed, 508 insertions(+), 1021 deletions(-) delete mode 100644 specs/localdisco-v3.rst create mode 100644 specs/localdisco-v4.rst diff --git a/specs/bep-v1.rst b/specs/bep-v1.rst index c86fd0629..e86d1d9c8 100644 --- a/specs/bep-v1.rst +++ b/specs/bep-v1.rst @@ -35,7 +35,7 @@ level protocols providing encryption and authentication. |-----------------------------| | Encryption & Auth (TLS 1.2) | |-----------------------------| - | TCP | + | Reliable Transport | |-----------------------------| v ... v @@ -62,382 +62,264 @@ another. The underlying transport protocol MUST guarantee reliable packet delivery. +In this document, in diagrams and text, "bit 0" refers to the *most +significant* bit of a word; "bit 15" is thus the least significant bit of a +16 bit word (int16) and "bit 31" is the least significant bit of a 32 bit +word (int32). Non protocol buffer integers are always represented in network +byte order (i.e., big endian) and are signed unless stated otherwise, but +when describing message lengths negative values do not make sense and the +most significant bit MUST be zero. + +The protocol buffer schemas in this document are in ``proto3`` syntax. This +means, among other things, that all fields are optional and will assume +their default value when missing. This does not nececessarily mean that a +message is *valid* with all fields empty - for example, an index entry for a +file that does not have a name is not useful and MAY be rejected by the +implementation. However the folder label is for human consumption only so an +empty label should be accepted - the implementation will have to choose some +way to represent the folder, perhaps by using the ID in it's place or +automatically generating a label. + Pre-authentication messages --------------------------- AFTER establishing a connection, but BEFORE performing any authentication, -*devices* MUST exchange Hello messages. +devices MUST exchange Hello messages. -Hello messages are used to carry additional information about the peer, which -might be of interest to the user even if the peer is not permitted to -communicate due to failing authentication. +Hello messages are used to carry additional information about the peer, +which might be of interest to the user even if the peer is not permitted to +communicate due to failing authentication. Note that the certificate based +authentication may be considered part of the TLS handshake that precedes the +Hello message exchange, but even in the case that a connection is rejected a +Hello message must be sent before the connection is terminated. -Hello messages MUST be prefixed with a magic number **0x9F79BC40** -represented in network byte order (BE), followed by 4 bytes representing the -size of the message in network byte order (BE), followed by the content of -the Hello message itself. The size of the contents of Hello message MUST be -less or equal to 1024 bytes. +Hello messages MUST be prefixed with an int32 containing the magic number +**0x2EA7D90B**, followed by an int16 representing the size of the message, +followed by the contents of the Hello message itself. -:: +.. code-block:: none - Prefix Structure: + 0 1 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Magic | + | (32 bits) | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Length | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + / / + \ Hello \ + / / + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Magic | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Content of HelloMessage \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +The Hello message itself is in protocol buffer format with the following schema: - HelloMessage Structure: +.. code-block:: proto - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Device Name (length + padded data) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Client Name (length + padded data) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Client Version (length + padded data) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + message Hello { + string device_name = 1; + string client_name = 2; + string client_version = 3; + } +Fields (Hello message) +^^^^^^^^^^^^^^^^^^^^^^ -Fields (HelloMessage) -^^^^^^^^^^^^^^^^^^^^^ - -The **Device Name** is a human readable (configured or auto detected) device +The **device_name** is a human readable (configured or auto detected) device name or host name, for the remote device. -The **Client Name** and **Client Version** identifies the implementation. The +The **client_name** and **client_version** identifies the implementation. The values SHOULD be simple strings identifying the implementation name, as a user would expect to see it, and the version string in the same manner. An -example Client Name is "syncthing" and an example Client Version is "v0.7.2". -The Client Version field SHOULD follow the patterns laid out in the `Semantic +example client name is "syncthing" and an example client version is "v0.7.2". +The client version field SHOULD follow the patterns laid out in the `Semantic Versioning `__ standard. -XDR -^^^ - -:: - - struct HelloMessage { - string DeviceName<64>; - string ClientName<64>; - string ClientVersion<64>; - }; - -Immediately after exchanging Hello messages, the connection should be -dropped if device does not pass authentication. +Immediately after exchanging Hello messages, the connection MUST be dropped +if the remote device does not pass authentication. Post-authentication Messages ---------------------------- -Every message starts with one 32 bit word indicating the message version, type -and ID, followed by the length of the message. The header is in network byte -order, i.e. big endian. In this document, in diagrams and text, "bit 0" refers -to the *most significant* bit of a word; "bit 31" is thus the least -significant bit of a 32 bit word. +Every message post authentication is made up of several parts: -:: +- A header length word +- A **Header** +- A message length word +- A **Message** - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Ver | Message ID | Type | Reserved |C| - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +.. code-block:: none -For BEP v1 the **Version** field is set to zero. Future versions with -incompatible message formats will increment the Version field. A message -with an unknown version is a protocol error and MUST result in the -connection being terminated. A client supporting multiple versions MAY -retry with a different protocol version upon disconnection. + 0 1 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Header Length | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + / / + \ Header \ + / / + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Message Length | + | (32 bits) | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + / / + \ Message \ + / / + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -The **Message ID** is set to a unique value for each transmitted Request -message. In Response messages it is set to the Message ID of the corresponding -Request message. The uniqueness requirement implies that no more than 4096 -request messages may be outstanding at any given moment. For message types -that do not have a corresponding response (Cluster Configuration, Index, etc.) -the Message ID field is irrelevant and SHOULD be set to zero. +The header length word is 16 bits. It indicates the length of the following +**Header** message. The Header is in protocol buffer format. The Header +describes the type and compression status of the following message. -The **Type** field indicates the type of data following the message header -and is one of the integers defined below. A message of an unknown type -is a protocol error and MUST result in the connection being terminated. +The message is preceded by the 32 bit message length word and is one of the +concrete BEP messages described below, identified by the **type** field of +the Header. -The **Compression** bit "C" indicates the compression used for the message. +As always, the length words are in network byte order (big endian). -For C=0: +.. code-block:: proto -- The Length field contains the length, in bytes, of the uncompressed - message data. + message Header { + MessageType type = 1; + MessageCompression compression = 2; + } -- The message is not compressed. + enum MessageType { + CLUSTER_CONFIG = 0; + INDEX = 1; + INDEX_UPDATE = 2; + REQUEST = 3; + RESPONSE = 4; + DOWNLOAD_PROGRESS = 5; + PING = 6; + CLOSE = 7; + } -For C=1: + enum MessageCompression { + NONE = 0; + LZ4 = 1; + } -- The Length field contains the length, in bytes, of the compressed - message data plus a four byte uncompressed length field. +When the **compression** field is **NONE**, the message is directly in +protocol buffer format. -- The compressed message data is preceded by a 32 bit field denoting - the length of the uncompressed message. +When the compression field is **LZ4**, the message consists of a 32 bit +integer describing the uncompressed message length followed by a single LZ4 +block. After decompressing the LZ4 block it should be interpreted as a +protocol buffer message just as in the uncompressed case. -- The message data is compressed using the LZ4 format and algorithm - described in http://www.lz4.org/. +Message Subtypes +---------------- -All data within the message (post decompression, if compression is in -use) MUST be in XDR (RFC 1014) encoding. All fields shorter than 32 bits -and all variable length data MUST be padded to a multiple of 32 bits. -The actual data types in use by BEP, in XDR naming convention, are the -following: - -(unsigned) int: - (unsigned) 32 bit integer - -(unsigned) hyper: - (unsigned) 64 bit integer - -opaque<> - variable length opaque data - -string<> - variable length string - -The transmitted length of string and opaque data is the length of actual -data, excluding any added padding. The encoding of opaque<> and string<> -are identical, the distinction being solely one of interpretation. -Opaque data should not be interpreted but can be compared bytewise to -other opaque data. All strings MUST use the Unicode UTF-8 encoding, -normalization form C. - -Cluster Config (Type = 0) -^^^^^^^^^^^^^^^^^^^^^^^^^ +Cluster Config +^^^^^^^^^^^^^^ .. Documentation note: the structure of a message section is always: 1. A short description of the message - 2. ASCII art overview of the message formats + 2. Protocol buffer schema of the message 3. Description of the fields in the message. - 4. XDR syntax field descriptions. This informational message provides information about the cluster configuration as it pertains to the current connection. A Cluster Config -message MUST be the first message sent on a BEP connection. Additional -Cluster Config messages MUST NOT be sent after the initial exchange. +message MUST be the first post authentication message sent on a BEP +connection. Additional Cluster Config messages MUST NOT be sent after the +initial exchange. -Graphical Representation -~~~~~~~~~~~~~~~~~~~~~~~~ +Protocol Buffer Schema +~~~~~~~~~~~~~~~~~~~~~~ -:: +.. code-block:: proto - ClusterConfigMessage Structure: + message ClusterConfig { + repeated Folder folders = 1; + } - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Folders | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Folder Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Options | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Option Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + message Folder { + string id = 1; + string label = 2; + bool read_only = 3; + bool ignore_permissions = 4; + bool ignore_delete = 5; + bool disable_temp_indexes = 6; - Folder Structure: + repeated Device devices = 16; + } - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of ID | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ ID (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Label (length + padded data) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Devices | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Device Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Flags | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Options | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Option Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + message Device { + bytes id = 1; + string name = 2; + repeated string addresses = 3; + Compression compression = 4; + string cert_name = 5; + int64 max_local_version = 6; + bool introducer = 7; + } - Device Structure: + enum Compression { + METADATA = 0; + NEVER = 1; + ALWAYS = 2; + } - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of ID | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ ID (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Name | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Name (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Addresses | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Address | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Address (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Compression | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Cert Name | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Cert Name (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | | - + Max Local Version (64 bits) + - | | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Flags | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Options | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Option Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +Fields (Cluster Config Message) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Option Structure: +.. Documentation note: the first time a field is mentioned it is put in + **bold text**. We use the space separated names in running text and + snake_case in the protocol buffer schema. - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Key | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Key (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Value | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Value (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -Fields (ClusterConfigMessage) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. Documentation note: the first time a field is mentioned it is put in **bold - text**. We use the Space Separated names in running text and ASCII art - diagrams, and CamelCase in the XDR syntax block at the end. - -The **Folders** field contains the list of folders that will be synchronized +The **folders** field contains the list of folders that will be synchronized over the current connection. -The **Options** field is a list of options that apply to the current -connection. The options are used in an implementation specific manner. The -options list is conceptually a map of keys to values, although it is -transmitted in the form of a list of key and value pairs, both of string type. -Key ID:s are implementation specific. An implementation MUST ignore unknown -keys. An implementation MAY impose limits on the length keys and values. The -options list may be used to inform devices of relevant local configuration -options such as rate limiting or make recommendations about request -parallelism, device priorities, etc. An empty options list is valid for -devices not having any such information to share. Devices MAY NOT make any -assumptions about peers acting in a specific manner as a result of sent -options. +Fields (Folder Message) +~~~~~~~~~~~~~~~~~~~~~~~ +The **id** field contains the folder ID, which is the unique identifier of +the folder. -Fields (Folder Structure) -~~~~~~~~~~~~~~~~~~~~~~~~~ +The **label** field contains the folder label, the human readable name of +the folder. -The **ID** field contains the folder ID, as a human readable string. +The **read only** field is set for folders that the device will accept no +updates from the network for. -The **Label** field contains the folder label, as human readable name for the folder. +The **ignore permissions** field is set for folders that the device will not +accept or announce file permissions for. -The **Devices** field is list of devices participating in sharing this folder. +The **ignore delete** field is set for folders that the device will ignore +deletes for. -The **Flags** field contains flags that affect the behavior of the folder. The -folder Flags field contains the following single bit flags: +The **disable temp indexes** field is set for folders that will not dispatch +and do not wish to receive progress updates about partially downloaded files +via Download Progress messages. -:: +The **devices** field is a list of devices participating in sharing this +folder. - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Reserved |T|D|P|R| - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +Fields (Device Message) +~~~~~~~~~~~~~~~~~~~~~~~ -:Bit 31 ("R", Read Only): - is set for folders that the device will accept no updates from the network - for. - -:Bit 30 ("P", Ignore Permissions): - is set for folders that the device will not accept or announce file - permissions for. - -:Bit 29 ("D", Ignore Deletes): - is set for folders that the device will ignore deletes for. - -:Bit 28 ("T", Disable Temporary Indexes): - is set for folders that will not dispatch and do not wish to receive - progress updates about partially downloaded files via DownloadProgress - messages. - -The **Options** field contains a list of options that apply to the folder. - -Fields (Device Structure) -~~~~~~~~~~~~~~~~~~~~~~~~~ - -The device **ID** field is a 32 byte number that uniquely identifies the +The device **id** field is a 32 byte number that uniquely identifies the device. For instance, the reference implementation uses the SHA-256 of the device X.509 certificate. -The **Name** field is a human readable name assigned to the described device +The **name** field is a human readable name assigned to the described device by the sending device. It MAY be empty and it need not be unique. -The list of **Addressess** is that used by the sending device to connect to +The list of **addresses** is that used by the sending device to connect to the described device. -The **Compression** field indicates the compression mode in use for this +The **compression** field indicates the compression mode in use for this device and folder. The following values are valid: :0: Compress metadata. This enables compression of metadata messages such as Index. :1: Compression disabled. No compression is used on any message. :2: Compress always. Metadata messages as well as Response messages are compressed. -The **Cert Name** field indicates the expected certificate name for this +The **cert name** field indicates the expected certificate name for this device. It is commonly blank, indicating to use the implementation default. -The **Max Local Version** field contains the highest local file +The **max local version** field contains the highest local file version number of the files already known to be in the index sent by this device. If nothing is known about the index of a given device, this field MUST be set to zero. When receiving a Cluster Config message with @@ -445,85 +327,11 @@ a non-zero Max Local Version for the local device ID, a device MAY elect to send an Index Update message containing only files with higher local version numbers in place of the initial Index message. -The **Flags** field indicates the sharing mode of the folder and other device -& folder specific settings. See the discussion on Sharing Modes. The Device -Flags field contains the following single bit flags: +The **introducer** field is set for devices that are trusted as cluster +introducers. -:: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Reserved |Pri| Reserved |I|R|T| - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -:Bit 31 ("T", Trusted): is set for devices that participate in trusted - mode. - -:Bit 30 ("R", Read Only): is set for devices that participate in read - only mode. - -:Bit 29 ("I", Introducer): is set for devices that are trusted as - cluster introducers. - -:Bits 16 through 28: are reserved and MUST be set to zero. - -:Bits 14-15 ("Pri", Priority): indicate the device's upload priority for this - folder. Possible values are: - - :00: The default. Normal priority. - - :01: High priority. Other devices SHOULD favour requesting files - from this device over devices with normal or low priority. - - :10: Low priority. Other devices SHOULD avoid requesting files from - this device when they are available from other devices. - - :11: Sharing disabled. Other devices SHOULD NOT request files from - this device. - -:Bits 0 through 14: are reserved and MUST be set to zero. - -Exactly one of the T and R bits MUST be set. - -The **Options** field contains a list of options that apply to the device. - -XDR -~~~ - -:: - - struct ClusterConfigMessage { - Folder Folders<1000000>; - Option Options<64>; - }; - - struct Folder { - string ID<256>; - string Label<256>; - Device Devices<1000000>; - unsigned int Flags; - Option Options<64>; - }; - - struct Device { - opaque ID<32>; - string Name<64>; - string Addresses<64>; - unsigned int Compression; - string CertName<64>; - hyper MaxLocalVersion; - unsigned int Flags; - Option Options<64>; - }; - - struct Option { - string Key<64>; - string Value<1024>; - }; - -Index (Type = 1) and Index Update (Type = 6) -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Index and Index Update +^^^^^^^^^^^^^^^^^^^^^^ The Index and Index Update messages define the contents of the senders folder. An Index message represents the full contents of the folder and @@ -533,363 +341,206 @@ the message. An Index Update MAY NOT be sent unless preceded by an Index, unless a non-zero Max Local Version has been announced for the given folder by the peer device. -Graphical Representation -~~~~~~~~~~~~~~~~~~~~~~~~ +The Index and Index Update messages are currently identical in format, +although this is not guaranteed to be the case in the future. -:: +Protocol Buffer Schema +~~~~~~~~~~~~~~~~~~~~~~ - IndexMessage Structure: +.. code-block:: proto - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Folder | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Folder (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Files | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more FileInfo Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Flags | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Options | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Option Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + message Index { + string folder = 1; + repeated FileInfo files = 2; + } - FileInfo Structure: + message IndexUpdate { + string folder = 1; + repeated FileInfo files = 2; + } - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Name | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Name (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Flags | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | | - + Modified (64 bits) + - | | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Version (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | | - + Local Version (64 bits) + - | | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Blocks | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more BlockInfo Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + message FileInfo { + string name = 1; + FileInfoType type = 2; + int64 size = 3; + uint32 permissions = 4; + int64 modified = 5; + bool deleted = 6; + bool invalid = 7; + bool no_permissions = 8; + Vector version = 9; + int64 local_version = 10; - Vector Structure: + repeated BlockInfo Blocks = 16; + } - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Counters | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Counter Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + enum FileInfoType { + FILE = 0; + DIRECTORY = 1; + SYMLINK_FILE = 2; + SYMLINK_DIRECTORY = 3; + SYMLINK_UNKNOWN = 4; + } - Counter Structure: + message BlockInfo { + int64 offset = 1; + int32 size = 2; + bytes hash = 3; + } - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | | - + ID (64 bits) + - | | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | | - + Value (64 bits) + - | | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + message Vector { + repeated Counter counters = 1; + } - - BlockInfo Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Size | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Hash | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Hash (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + message Counter { + uint64 id = 1; + uint64 value = 2; + } Fields (Index Message) ~~~~~~~~~~~~~~~~~~~~~~ -The **Folder** field identifies the folder that the index message pertains to. +The **folder** field identifies the folder that the index message pertains to. -**Files** +The **files** field is a list of files making up the index information. -The **Flags** field is reserved for future use and MUST currently be set to -zero. +Fields (FileInfo Message) +~~~~~~~~~~~~~~~~~~~~~~~~~ -The **Options** list is implementation defined and as described in the -ClusterConfig message section. - -Fields (FileInfo Structure) -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The **Name** is the file name path relative to the folder root. Like all +The **name** is the file name path relative to the folder root. Like all strings in BEP, the Name is always in UTF-8 NFC regardless of operating -system or file system specific conventions. The Name field uses the -slash character ("/") as path separator, regardless of the -implementation's operating system conventions. The combination of Folder -and Name uniquely identifies each file in a cluster. +system or file system specific conventions. The name field uses the slash +character ("/") as path separator, regardless of the implementation's +operating system conventions. The combination of folder and name uniquely +identifies each file in a cluster. -The **Flags** field is made up of the following single bit flags: +The **type** field contains the type of the described item. The type is one +of **file (0)**, **directory (1)**, **symlink to file (2)**, **symlink to +directory (3)**, or **symlink to unknown target (4)**. The distinction +between the various types of symlinks is not required on all operating +systems - the implementation SHOULD nonetheless indicate the target type +when possible. -:: +The **size** field contains the size of the file, in bytes. For directories +the size is zero. For symlinks the size is the length of the target name. - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Reserved |U|S|P|D|I|R| Unix Perm. & Mode | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +The **permissions** field holds the common Unix permission bits. An +implementation MAY ignore or interpret these as is suitable on the host +operating system. -:The lower 12 bits: hold the common Unix permission and mode bits. An - implementation MAY ignore or interpret these as is suitable on the - host operating system. - -:Bit 19 ("R"): is set when the file has been deleted. The block list - SHALL be of length zero and the modification time indicates the time - of deletion or, if the time of deletion is not reliably determinable, - the last known modification time. - -:Bit 18 ("I"): is set when the file is invalid and unavailable for - synchronization. A peer MAY set this bit to indicate that it can - temporarily not serve data for the file. - -:Bit 17 ("D"): is set when the item represents a directory. The block - list SHALL be of length zero. - -:Bit 16 ("P"): is set when there is no permission information for the - file. This is the case when it originates on a file system which - does not support permissions. Changes to only permission bits SHOULD - be disregarded on files with this bit set. The permissions bits MUST - be set to the octal value 0666. - -:Bit 15 ("S"): is set when the file is a symbolic link. The block list - SHALL be of one or more blocks since the target of the symlink is - stored within the blocks of the file. - -:Bit 14 ("U"): is set when the symbolic links target does not exist. On - systems where symbolic links have types, this bit being means that - the default file symlink SHALL be used. If this bit is unset bit 19 - will decide the type of symlink to be created. - -:Bit 0 through 13: are reserved for future use and SHALL be set to - zero. - -The **Modified** time is expressed as the number of seconds since the Unix +The **modified** time is expressed as the number of seconds since the Unix Epoch (1970-01-01 00:00:00 UTC). -The **Version** field is a version vector describing the updates performed -to a file by all members in the cluster. Each counter in the version -vector is an ID-Value tuple. The ID is used the first 64 bits of the -device ID. The Value is a simple incrementing counter, starting at zero. -The combination of Folder, Name and Version uniquely identifies the -contents of a file at a given point in time. +The **deleted** field is set when the file has been deleted. The block list +SHALL be of length zero and the modification time indicates the time of +deletion or, if the time of deletion is not reliably determinable, the last +known modification time. -The **Local Version** field is the value of a device local monotonic clock +The **invalid** field is set when the file is invalid and unavailable for +synchronization. A peer MAY set this bit to indicate that it can temporarily +not serve data for the file. + +The **no permissions** field is set when there is no permission information +for the file. This is the case when it originates on a file system which +does not support permissions. Changes to only permission bits SHOULD be +disregarded on files with this bit set. The permissions bits MUST be set to +the octal value 0666. + +The **version** field is a version vector describing the updates performed +to a file by all members in the cluster. Each counter in the version vector +is an ID-Value tuple. The ID is the first 64 bits of the device ID. The +Value is a simple incrementing counter, starting at zero. The combination of +Folder, Name and Version uniquely identifies the contents of a file at a +given point in time. + +The **local version** field is the value of a device local monotonic clock at the time of last local database update to a file. The clock ticks on every local database update. -The **Blocks** list contains the size and hash for each block in the file. -Each block represents a 128 KiB slice of the file, except for the last -block which may represent a smaller amount of data. +The **blocks** list contains the size and hash for each block in the file. +Each block represents a 128 KiB slice of the file, except for the last block +which may represent a smaller amount of data. -The hash algorithm is implied by the **Hash** length. Currently, the hash -MUST be 32 bytes long and computed by SHA256. - -XDR -~~~ - -:: - - struct IndexMessage { - string Folder<256>; - FileInfo Files<1000000>; - unsigned int Flags; - Option Options<64>; - }; - - struct FileInfo { - string Name<8192>; - unsigned int Flags; - hyper Modified; - Vector Version; - hyper LocalVersion; - BlockInfo Blocks<10000000>; - }; - - struct Vector { - Counter Counters<>; - }; - - struct Counter { - unsigned hyper ID; - unsigned hyper Value; - }; - - struct BlockInfo { - unsigned int Size; - opaque Hash<64>; - }; - -Request (Type = 2) -^^^^^^^^^^^^^^^^^^ +Request +^^^^^^^ The Request message expresses the desire to receive a data block corresponding to a part of a certain file in the peer's folder. -Graphical Representation -~~~~~~~~~~~~~~~~~~~~~~~~ +Protocol Buffer Schema +~~~~~~~~~~~~~~~~~~~~~~ -:: +.. code-block:: proto - RequestMessage Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Folder | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Folder (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Name | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Name (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | | - + Offset (64 bits) + - | | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Size | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Hash | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Hash (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Flags | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Options | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Option Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + message Request { + int32 id = 1; + string folder = 2; + string name = 3; + int64 offset = 4; + int32 size = 5; + bytes hash = 6; + bool from_temporary = 7; + } Fields ~~~~~~ -The Folder and Name fields are as documented for the Index message. The -Offset and Size fields specify the region of the file to be transferred. -This SHOULD equate to exactly one block as seen in an Index message. +The **id** is the request identifier. It will be matched in the +corresponding **Request** message. Each outstanding request must have a +unique ID. -The Hash field MAY be set to the expected hash value of the block, or -may be left empty (zero length). If set, the other device SHOULD ensure -that the transmitted block matches the requested hash. The other device -MAY reuse a block from a different file and offset having the same size -and hash, if one exists. +The **folder** and **name** fields are as documented for the Index message. +The **offset** and **size** fields specify the region of the file to be +transferred. This SHOULD equate to exactly one block as seen in an Index +message. -The **Flags** field is made up of the following single bit flags: -:: +The *hash* field MAY be set to the expected hash value of the block. If set, +the other device SHOULD ensure that the transmitted block matches the +requested hash. The other device MAY reuse a block from a different file and +offset having the same size and hash, if one exists. - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Reserved |T| - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +The **from temporary** field is set to indicate that the read should be +performed from the temporary file (converting name to it's temporary form) +and falling back to the non temporary file if any error occurs. Knowledge of +contents of temporary files comes from DownloadProgress messages. -:Bit 31 ("T", Temporary): is set to indicate that the read should be performed - from the temporary file (converting Name to it's temporary form) and falling - back to the non temporary file if any error occurs. Knowledge of content - inside temporary files comes from DownloadProgress messages. - -The Options list is implementation defined and as described in the -ClusterConfig message section. - -XDR -~~~ - -:: - - struct RequestMessage { - string Folder<64>; - string Name<8192>; - hyper Offset; - int Size; - opaque Hash<64>; - unsigned int Flags; - Option Options<64>; - }; - -Response (Type = 3) -^^^^^^^^^^^^^^^^^^^ +Response +^^^^^^^^ The Response message is sent in response to a Request message. -Graphical Representation -~~~~~~~~~~~~~~~~~~~~~~~~ +Protocol Buffer Schema +~~~~~~~~~~~~~~~~~~~~~~ -ResponseMessage Structure: +.. code-block:: proto -:: + message Response { + int32 id = 1; + bytes data = 2; + ErrorCode code = 3; + } - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Data | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Data (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Code | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + enum ErrorCode { + NO_ERROR = 0; + GENERIC = 1; + NO_SUCH_FILE = 2; + INVALID_FILE = 3; + } Fields ~~~~~~ -The **Data** field contains either a full 128 KiB block, a shorter block in -the case of the last block in a file, or is empty (zero length) if the -requested block is not available. +The **id** field is the request identifier. It must match the ID of the +**Request** that is being responded to. -The **Code** field contains an error code describing the reason a Request -could not be fulfilled, in the case where a zero length Data was -returned. The following values are defined: +The **data** field contains either the requested data block or is empty if +the requested block is not available. -:0: No Error (Data should be present) +The **code** field contains an error code describing the reason a Request +could not be fulfilled, in the case where zero length data was returned. The +following values are defined: + +:0: No Error (data should be present) :1: Generic Error @@ -899,207 +550,126 @@ returned. The following values are defined: :3: Invalid (file exists but has invalid bit set or is otherwise unavailable) -XDR -~~~ - -:: - - struct ResponseMessage { - opaque Data<>; - int Code; - } - -DownloadProgress (Type = 8) -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +DownloadProgress +^^^^^^^^^^^^^^^^ The DownloadProgress message is used to notify remote devices about partial availability of files. By default, these messages are sent every 5 seconds, and only in the cases where progress or state changes have been detected. Each DownloadProgress message is addressed to a specific folder and MUST -contain zero or more FileDownloadProgressUpdate structures. +contain zero or more FileDownloadProgressUpdate messages. -Graphical Representation -~~~~~~~~~~~~~~~~~~~~~~~~ +Protocol Buffer Schema +~~~~~~~~~~~~~~~~~~~~~~ -:: +.. code-block:: proto - DownloadProgressMessage Structure: + message DownloadProgress { + string folder = 1; + repeated FileDownloadProgressUpdate updates = 2; + } - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Folder (length + padded data) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Updates | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more FileDownloadProgressUpdate Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Flags | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Options | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Option Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + message FileDownloadProgressUpdate { + FileDownloadProgressUpdateType update_type = 1; + string name = 2; + Vector version = 3; + repeated int32 block_indexes = 4; + } - FileDownloadProgressUpdate Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Update Type | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Name (length + padded data) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Version Structure \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Block Indexes | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - | Block Indexes (n items) | - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - - -Each + enum FileDownloadProgressUpdateType { + APPEND = 0; + FORGET = 1; + } Fields (DownloadProgress Message) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -**Folder** represents the ID of the folder for which the update is being -provided. -The **Flags** field is reserved for future use and MUST currently be set to -zero. The **Options** field contains a list of options that apply to the update. +The **folder** field represents the ID of the folder for which the update is +being provided. -Fields (FileDownloadProgressUpdate Structure) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The **updates** field is a list of progress update messages. -The **Update Type** field is made up of the following single bit flags: -:: +Fields (FileDownloadProgressUpdate Message) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Reserved |F| - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +The **update type** indicates whether the update is of type **append (0)** +(new blocks are available) or **forget (1)** (the file transfer has +completed or failed). -:Bit 31 ("F", Forget): is set to notify that the file that was previously - advertised is no longer available (at least as a temporary file). +The **name** field defines the file name from the global index for which +this update is being sent. -The **Name** field defines the file name from the global index for which this +The **version** message defines the version of the file for which this update is being sent. -The **Version** structure defines the version of the file for which this update -is being sent. +The **block indexes** field is a list of positive integers, where each +integer represents the index of the block in the FileInfo message Blocks +array that has become available for download. -**Block Indexes** is a list of positive integers, where each integer represents -the index of the block in the FileInfo structure Blocks array that has become -available for download. -For example an integer with with value 3 represents that the data defined in the -fourth BlockInfo structure of the FileInfo structure of that file is now available. -Please note that matching should be done on **Name** AND **Version**. -Furthermore, each update received is incremental, for example the initial update -structure might contain indexes 0, 1, 2, an update 5 seconds later might contain -indexes 3, 4, 5 which should be appended to the original list, which implies -that blocks 0-5 are currently available. +For example an integer with value 3 represents that the data defined in the +fourth BlockInfo message of the FileInfo message of that file is now +available. Please note that matching should be done on **name** AND +**version**. Furthermore, each update received is incremental, for example +the initial update message might contain indexes 0, 1, 2, an update 5 +seconds later might contain indexes 3, 4, 5 which should be appended to the +original list, which implies that blocks 0-5 are currently available. -Block indexes MAY be added in any order. -An implementation MUST NOT assume that block indexes are added in any specific -order. +Block indexes MAY be added in any order. An implementation MUST NOT assume +that block indexes are added in any specific order. -**Forget** bit being set implies that the file that was previously advertised -is no longer available, therefore the list of block indexes should be truncated. +The **forget** field being set implies that previously advertised file is no +longer available, therefore the list of block indexes should be truncated. -Messages with **Forget** bit set MUST NOT have any block indexes. +Messages with the **forget** field set MUST NOT have any block indexes. -Any update message which is being sent for a different **Version** of the same -file name must be preceded with an update message for the old version of that -file with the **Forget** bit set. +Any update message which is being sent for a different **version** of the +same file name must be preceded with an update message for the old version +of that file with the **forget** field set. -As a safeguard on the receiving side, value of **Version** changing between -update messages implies that the file has changed, and that any indexes -previously advertised are no longer available. The list of available block -indexes MUST be replaced (rather than appended) with the indexes specified in -this message. +As a safeguard on the receiving side, the value of **version** changing +between update messages implies that the file has changed and that any +indexes previously advertised are no longer available. The list of available +block indexes MUST be replaced (rather than appended) with the indexes +specified in this message. -XDR -~~~ +Ping +^^^^ -:: +The Ping message is used to determine that a connection is alive, and to +keep connections alive through state tracking network elements such as +firewalls and NAT gateways. A Ping message is sent every 90 seconds, if no +other message has been sent in the preceding 90 seconds. - struct DownloadProgressMessage { - string Folder<64>; - FileDownloadProgressUpdate Updates<1000000>; - unsigned int Flags; - Option Options<64>; - } +Protocol Buffer Schema +~~~~~~~~~~~~~~~~~~~~~~ - struct FileDownloadProgressUpdate { - unsigned int UpdateType; - string Name<8192>; - Vector Version; - int BlockIndexes<1000000>; +.. code-block:: proto + + message Ping { } -Ping (Type = 4) -^^^^^^^^^^^^^^^ +Close +^^^^^ -The Ping message is used to determine that a connection is alive, and to keep -connections alive through state tracking network elements such as firewalls -and NAT gateways. The Ping message has no contents. A Ping message is sent -every 90 seconds, if no other message has been sent in the preceding 90 -seconds. +The Close message MAY be sent to indicate that the connection will be torn +down due to an error condition. A Close message MUST NOT be followed by +further messages. -Close (Type = 7) -^^^^^^^^^^^^^^^^ +Protocol Buffer Schema +~~~~~~~~~~~~~~~~~~~~~~ -The Close message MAY be sent to indicate that the connection will be -torn down due to an error condition. A Close message MUST NOT be -followed by further messages. +.. code-block:: proto -Graphical Representation -~~~~~~~~~~~~~~~~~~~~~~~~ - -:: - - CloseMessage Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of Reason | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Reason (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Code | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + message Close { + string reason = 1; + } Fields ~~~~~~ -The **Reason** field contains a human description of the error condition, -suitable for consumption by a human. The **Code** field is for a machine -readable error code. Codes are reserved for future use and MUST -currently be set to zero. - -:: - - struct CloseMessage { - string Reason<1024>; - int Code; - } +The **reason** field contains a human readable description of the error +condition. Sharing Modes ------------- @@ -1121,10 +691,10 @@ directions. Read Only ^^^^^^^^^ -In read only mode, a device does not apply any updates from the cluster, -but publishes changes of its local folder to the cluster as usual. -The local folder can be seen as a "master copy" that is never affected -by the actions of other cluster devices. +In read only mode, a device does not apply any updates from the cluster, but +publishes changes of its local folder to the cluster as usual. The local +folder can be seen as a "master copy" that is never affected by the actions +of other cluster devices. :: @@ -1137,59 +707,19 @@ by the actions of other cluster devices. Message Limits -------------- -An implementation MAY impose reasonable limits on the length of messages -and message fields to aid robustness in the face of corruption or broken -implementations. These limits, if imposed, SHOULD NOT be more -restrictive than the following. An implementation should strive to keep -messages short and to the point, favouring more and smaller messages -over fewer and larger. For example, favour a smaller Index message -followed by one or more Index Update messages rather than sending a very -large Index message. +An implementation MAY impose reasonable limits on the length of messages and +message fields to aid robustness in the face of corruption or broken +implementations. An implementation should strive to keep messages short +and to the point, favouring more and smaller messages over fewer and larger. +For example, favour a smaller Index message followed by one or more Index +Update messages rather than sending a very large Index message. -=================== =================== ============= -Message Type Field Limit -=================== =================== ============= -**All Messages** ------------------------------------------------------ -| Total length 512 MiB - -**Index and Index Update Messages** ------------------------------------------------------ -| Folder 64 bytes -| Number of Files 1.000.000 -| Name 8192 bytes -| Number of Blocks 10.000.000 -| Hash 64 bytes -| Number of Counters 1.000.000 - -**Request Messages** ------------------------------------------------------ -| Folder 64 bytes -| Name 8192 bytes - -**Response Messages** ------------------------------------------------------ -| Data 256 KiB - -**Cluster Config Message** ------------------------------------------------------ -| Number of Folders 1.000.000 -| Number of Devices 1.000.000 -| Number of Options 64 -| Key 64 bytes -| Value 1024 bytes - -**Download Progress Messages** ------------------------------------------------------ -| Folder 64 bytes -| Number of Updates 1.000.000 -| Name 8192 bytes -| Number of Indexes 1.000.000 -=================== =================== ============= - -The currently defined values allow maximum file size of 1220 GiB -(10.000.000 x 128 KiB). The maximum message size covers an Index message -for the maximum file. +The Syncthing implementation imposes a hard limit of 500,000,000 bytes on +all messages. Attempting to send or receive a larger message will result in +a connection close. This size was chosen to accomodate Index messages +containing a large block list. It's intended that the limit may be further +reduced in a future protocol update supporting variable block sizes (and +thus shorter block lists for large files). Example Exchange ---------------- @@ -1216,15 +746,15 @@ Example Exchange === ======================= ====================== The connection is established and at 1. both peers send ClusterConfiguration -messages and then Index records. The Index records are received and both peers -recompute their knowledge of the data in the cluster. In this example, peer A -has four missing or outdated blocks. At 5 through 8 peer A sends requests for -these blocks. The requests are received by peer B, who retrieves the data from -the folder and transmits Response records (9 through 12). Device A updates -their folder contents and transmits an Index Update message (13). Both peers -enter idle state after 13. At some later time 14, the ping timer on device B -expires and a Ping message is sent. The same process occurs for device A at -15. +messages and then Index records. The Index records are received and both +peers recompute their knowledge of the data in the cluster. In this example, +peer A has four missing or outdated blocks. At 5 through 8 peer A sends +requests for these blocks. The requests are received by peer B, who +retrieves the data from the folder and transmits Response records (9 through +12). Device A updates their folder contents and transmits an Index Update +message (13). Both peers enter idle state after 13. At some later time 14, +the ping timer on device B expires and a Ping message is sent. The same +process occurs for device A at 15. Examples of Strong Cipher Suites -------------------------------- diff --git a/specs/localdisco-v3.rst b/specs/localdisco-v3.rst deleted file mode 100644 index ffa026d3d..000000000 --- a/specs/localdisco-v3.rst +++ /dev/null @@ -1,120 +0,0 @@ -.. _localdisco-v3: - -Local Discovery Protocol v3 -=========================== - -Mode of Operation ------------------ - -Each participating device periodically sends an Announcement packet. It also -keeps a table of the announcements it has seen. There is no way to solicit a -reply; the only message type is Announcement. - -On multihomed hosts the announcement packets should be sent on each interface -on which Syncthing will accept connections. - -For IPv4, the Announcement packet is broadcast either to the link-specific -broadcast address, or to the generic link-local broadcast address -``255.255.255.255``, with destination port 21027. - -For IPv6, the Announcement packet is multicast to the transient link-local -multicast address ``[ff12::8384]``, with destination port 21027. - -It is recommended that local discovery Announcement packets be sent on a 30 to -60 second interval, possibly with immediate transmissions when a previously -unknown device is discovered. - -Device ID ---------- - -The device ID is the SHA-256 (32 bytes) of the device X.509 certificate. See -:ref:`device-ids` in the Syncthing documentation. - -Announcement packet -------------------- - -The Announcement packet has the following structure:: - - Announce Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Magic | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Device Structure \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Extra Devices | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Device Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - - Device Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of ID | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ ID (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Number of Addresses | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ Zero or more Address Structures \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - - Address Structure: - - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - | Length of URL | - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - / / - \ URL (variable length) \ - / / - +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - -The corresponding XDR representation is as follows (see -`RFC4506 `__ for the XDR format): - -:: - - struct Announcement { - unsigned int Magic; - Device This; - Device Extra<>; - } - - struct Device { - opaque ID<32>; - Address Addresses<16>; - } - - struct Address { - string URL<2083>; - } - - -In the ``Announce`` structure field ``Magic`` is used to ensure -a correct datagram was received and MUST be equal to ``0x7D79BC40``. - -The first Device structure contains information about the sending -device. The following zero or more Extra devices contain information -about other devices known to the sending device. - -In the ``Device`` structure, field ``DeviceID`` is the SHA-256 (32 -bytes) of the device X.509 certificate, as explained in section *Device -ID*. - -For each ``Address`` the ``URL`` field contains the actual target address. -Direct connections will typically have the ``tcp://`` scheme. Relay connections -will typically use the ``relay://`` scheme. diff --git a/specs/localdisco-v4.rst b/specs/localdisco-v4.rst new file mode 100644 index 000000000..0b0d0fc1e --- /dev/null +++ b/specs/localdisco-v4.rst @@ -0,0 +1,77 @@ +.. _localdisco-v3: + +Local Discovery Protocol v4 +=========================== + +Mode of Operation +----------------- + +Each participating device periodically sends an Announcement packet. It also +keeps a table of the announcements it has seen. There is no way to solicit a +reply; the only message type is Announcement. + +On multihomed hosts the announcement packets should be sent on each interface +on which Syncthing will accept connections. + +The announcement packet is sent over UDP. + +For IPv4, the Announcement packet is broadcast either to the link-specific +broadcast address, or to the generic link-local broadcast address +``255.255.255.255``, with destination port 21027. + +For IPv6, the Announcement packet is multicast to the transient link-local +multicast address ``ff12::8384``, with destination port 21027. + +It is recommended that local discovery Announcement packets be sent on a 30 to +60 second interval, possibly with immediate transmissions when a previously +unknown device is discovered. + +Device ID +--------- + +The device ID is the SHA-256 (32 bytes) of the device X.509 certificate. See +:ref:`device-ids` in the Syncthing documentation. + +Announcement packet +------------------- + +The Announcement packet has the following structure: + +.. code-block:: none + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Magic | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + / / + \ Announce Message \ + / / + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + +There is no explicit length field as the length is given by the length of +the discovery announcement packet itself. + +The Magic field is a 32 bit word representing 0x2EA7D90B in network (big +endian) byte order. It identifies the packet as being a Syncthing discovery +protocol packet. + +The Announce Message contents are in protocol buffer format using the +following schema: + +.. code-block:: proto + + message Announce { + bytes id = 1; + repeated string addresses = 2; + } + +The ``id`` field contains the Device ID of the sending device. + +The ``addresses`` field contains a list of addresses where the device can be +contacted. Direct connections will typically have the ``tcp://`` scheme. +Relay connections will typically use the ``relay://`` scheme. + +When interpreting addresses with an unspecified address, e.g., +``tcp://0.0.0.0:22000`` or ``tcp://:42424``, the source address of the +discovery announcement is to be used.