Document temporary indexes

GitHub-Pull-Request: https://github.com/syncthing/docs/pull/161

specs/bep-v1.rst

@@ -222,7 +222,7 @@ other opaque data. All strings MUST use the Unicode UTF-8 encoding,
 normalization form C.
 
 Cluster Config (Type = 0)
-~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. Documentation note: the structure of a message section is always:
    1. A short description of the message
@@ -236,7 +236,7 @@ message MUST be the first message sent on a BEP connection. Additional
 Cluster Config messages MUST NOT be sent after the initial exchange.
 
 Graphical Representation
-^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~
 
 ::
 
@@ -353,7 +353,7 @@ Graphical Representation
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 
 Fields (ClusterConfigMessage)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. Documentation note: the first time a field is mentioned it is put in **bold
    text**. We use the Space Separated names in running text and ASCII art
@@ -377,7 +377,7 @@ options.
 
 
 Fields (Folder Structure)
-^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The **ID** field contains the folder ID, as a human readable string.
 
@@ -393,7 +393,7 @@ folder Flags field contains the following single bit flags:
      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-    |                           Reserved                      |D|P|R|
+    |                           Reserved                    |T|D|P|R|
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 
 :Bit 31 ("R", Read Only):
@@ -407,10 +407,15 @@ folder Flags field contains the following single bit flags:
 :Bit 29 ("D", Ignore Deletes):
     is set for folders that the device will ignore deletes for.
 
+:Bit 28 ("T", Disable Temporary Indexes):
+    is set for folders that will not dispatch and do not wish to receive
+    progress updates about partially downloaded files via DownloadProgress
+	messages.
+
 The **Options** field contains a list of options that apply to the folder.
 
 Fields (Device Structure)
-^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The device **ID** field is a 32 byte number that uniquely identifies the
 device. For instance, the reference implementation uses the SHA-256 of the
@@ -463,7 +468,7 @@ Flags field contains the following single bit flags:
 
 :Bits 16 through 28: are reserved and MUST be set to zero.
 
-:Bits 14-15 ("Pri): indicate the device's upload priority for this
+:Bits 14-15 ("Pri", Priority): indicate the device's upload priority for this
    folder. Possible values are:
 
    :00: The default. Normal priority.
@@ -484,7 +489,7 @@ Exactly one of the T and R bits MUST be set.
 The **Options** field contains a list of options that apply to the device.
 
 XDR
-^^^
+~~~
 
 ::
 
@@ -518,7 +523,7 @@ XDR
     };
 
 Index (Type = 1) and Index Update (Type = 6)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 The Index and Index Update messages define the contents of the senders
 folder. An Index message represents the full contents of the folder and
@@ -529,7 +534,7 @@ Index, unless a non-zero Max Local Version has been announced for the
 given folder by the peer device.
 
 Graphical Representation
-^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~
 
 ::
 
@@ -633,7 +638,7 @@ Graphical Representation
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 
 Fields (Index Message)
-^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~
 
 The **Folder** field identifies the folder that the index message pertains to.
 
@@ -646,7 +651,7 @@ The **Options** list is implementation defined and as described in the
 ClusterConfig message section.
 
 Fields (FileInfo Structure)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The **Name** is the file name path relative to the folder root. Like all
 strings in BEP, the Name is always in UTF-8 NFC regardless of operating
@@ -721,7 +726,7 @@ The hash algorithm is implied by the **Hash** length. Currently, the hash
 MUST be 32 bytes long and computed by SHA256.
 
 XDR
-^^^
+~~~
 
 ::
 
@@ -756,13 +761,13 @@ XDR
     };
 
 Request (Type = 2)
-~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^
 
 The Request message expresses the desire to receive a data block
 corresponding to a part of a certain file in the peer's folder.
 
 Graphical Representation
-^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~
 
 ::
 
@@ -805,7 +810,7 @@ Graphical Representation
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 
 Fields
-^^^^^^
+~~~~~~
 
 The Folder and Name fields are as documented for the Index message. The
 Offset and Size fields specify the region of the file to be transferred.
@@ -817,12 +822,25 @@ that the transmitted block matches the requested hash. The other device
 MAY reuse a block from a different file and offset having the same size
 and hash, if one exists.
 
-The Flags field is reserved for future use and MUST currently be set to
+The **Flags** field is made up of the following single bit flags:
-zero. The Options list is implementation defined and as described in the
+::
+
+     0                   1                   2                   3
+     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    |                           Reserved                          |T|
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+:Bit 31 ("T", Temporary): is set to indicate that the read should be performed
+    from the temporary file (converting Name to it's temporary form) and falling
+    back to the non temporary file if any error occurs. Knowledge of content
+	inside temporary files comes from DownloadProgress messages.
+
+The Options list is implementation defined and as described in the
 ClusterConfig message section.
 
 XDR
-^^^
+~~~
 
 ::
 
@@ -837,12 +855,12 @@ XDR
     };
 
 Response (Type = 3)
-~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^
 
 The Response message is sent in response to a Request message.
 
 Graphical Representation
-^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~
 
 ResponseMessage Structure:
 
@@ -861,7 +879,7 @@ ResponseMessage Structure:
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 
 Fields
-^^^^^^
+~~~~~~
 
 The **Data** field contains either a full 128 KiB block, a shorter block in
 the case of the last block in a file, or is empty (zero length) if the
@@ -882,7 +900,7 @@ returned. The following values are defined:
    unavailable)
 
 XDR
-^^^
+~~~
 
 ::
 
@@ -891,8 +909,150 @@ XDR
         int Code;
     }
 
+DownloadProgress (Type = 8)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The DownloadProgress message is used to notify remote devices about partial
+availability of files. By default, these messages are sent every 5 seconds,
+and only in the cases where progress or state changes have been detected.
+Each DownloadProgress message is addressed to a specific folder and MUST
+contain zero or more FileDownloadProgressUpdate structures.
+
+Graphical Representation
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+::
+
+    DownloadProgressMessage Structure:
+
+     0                   1                   2                   3
+     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    /                                                               /
+    \                 Folder (length + padded data)                 \
+    /                                                               /
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    |                       Number of Updates                       |
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    /                                                               /
+    \      Zero or more FileDownloadProgressUpdate Structures       \
+    /                                                               /
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    |                             Flags                             |
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    |                       Number of Options                       |
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    /                                                               /
+    \                Zero or more Option Structures                 \
+    /                                                               /
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+    FileDownloadProgressUpdate Structure:
+
+     0                   1                   2                   3
+     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    |                          Update Type                          |
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    /                                                               /
+    \                  Name (length + padded data)                  \
+    /                                                               /
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    /                                                               /
+    \                      Version Structure                        \
+    /                                                               /
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    |                    Number of Block Indexes                    |
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    /                                                               /
+    |                    Block Indexes (n items)                    |
+    /                                                               /
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+
+Each
+
+Fields (DownloadProgress Message)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+**Folder** represents the ID of the folder for which the update is being
+provided.
+
+The **Flags** field is reserved for future use and MUST currently be set to
+zero. The **Options** field contains a list of options that apply to the update.
+
+Fields (FileDownloadProgressUpdate Structure)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The **Update Type** field is made up of the following single bit flags:
+::
+
+     0                   1                   2                   3
+     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+    |                           Reserved                          |F|
+    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
+:Bit 31 ("F", Forget): is set to notify that the file that was previously
+    advertised is no longer available (at least as a temporary file).
+
+The **Name** field defines the file name from the global index for which this
+update is being sent.
+
+The **Version** structure defines the version of the file for which this update
+is being sent.
+
+**Block Indexes** is a list of positive integers, where each integer represents
+the index of the block in the FileInfo structure Blocks array that has become
+available for download.
+For example an integer with with value 3 represents that the data defined in the
+fourth BlockInfo structure of the FileInfo structure of that file is now available.
+Please note that matching should be done on **Name** AND **Version**.
+Furthermore, each update received is incremental, for example the initial update
+structure might contain indexes 0, 1, 2, an update 5 seconds later might contain
+indexes 3, 4, 5 which should be appended to the original list, which implies
+that blocks 0-5 are currently available.
+
+Block indexes MAY be added in any order.
+An implementation MUST NOT assume that block indexes are added in any specific
+order.
+
+**Forget** bit being set implies that the file that was previously advertised
+is no longer available, therefore the list of block indexes should be truncated.
+
+Messages with **Forget** bit set MUST NOT have any block indexes.
+
+Any update message which is being sent for a different **Version** of the same
+file name must be preceeded with an update message for the old version of that
+file with the **Forget** bit set.
+
+As a safeguard on the receiving side, value of **Version** changing between
+update messages implies that the file has changed, and that any indexes
+previously advertised are no longer available. The list of available block
+indexes MUST be replaced (rather than appended) with the indexes specified in
+this message.
+
+XDR
+~~~
+
+::
+
+    struct DownloadProgressMessage {
+        string Folder<64>;
+        FileDownloadProgressUpdate Updates<1000000>;
+        unsigned int Flags;
+        Option Options<64>;
+    }
+
+    struct FileDownloadProgressUpdate {
+        unsigned int UpdateType;
+        string Name<8192>;
+        Vector Version;
+        int BlockIndexes<1000000>;
+    }
+
+
 Ping (Type = 4)
-~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^
 
 The Ping message is used to determine that a connection is alive, and to keep
 connections alive through state tracking network elements such as firewalls
@@ -901,14 +1061,14 @@ every 90 seconds, if no other message has been sent in the preceding 90
 seconds.
 
 Close (Type = 7)
-~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^
 
 The Close message MAY be sent to indicate that the connection will be
 torn down due to an error condition. A Close message MUST NOT be
 followed by further messages.
 
 Graphical Representation
-^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~
 
 ::
 
@@ -927,7 +1087,7 @@ Graphical Representation
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 
 Fields
-^^^^^^
+~~~~~~
 
 The **Reason** field contains a human description of the error condition,
 suitable for consumption by a human. The **Code** field is for a machine
@@ -945,7 +1105,7 @@ Sharing Modes
 -------------
 
 Trusted
-~~~~~~~
+^^^^^^^
 
 Trusted mode is the default sharing mode. Updates are exchanged in both
 directions.
@@ -959,7 +1119,7 @@ directions.
     +------------+     Updates      \---------/
 
 Read Only
-~~~~~~~~~
+^^^^^^^^^
 
 In read only mode, a device does not apply any updates from the cluster,
 but publishes changes of its local folder to the cluster as usual.
@@ -1018,6 +1178,13 @@ Message Type        Field               Limit
 |                   Number of Options   64
 |                   Key                 64 bytes
 |                   Value               1024 bytes
+
+**Download Progress Messages**
+-----------------------------------------------------
+|                   Folder              64 bytes
+|                   Number of Updates   1.000.000
+|                   Name                8192 bytes
+|                   Number of Indexes   1.000.000
 =================== =================== =============
 
 The currently defined values allow maximum file size of 1220 GiB

specs/relay-v1.rst

@@ -65,7 +65,7 @@ which is connected to the relay using the permanent protocol submode.
 
 
 Permanent protocol submode
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 A permanent protocol submode begins with the client sending a JoinRelayRequest
 message, which the relay responds to with either a ResponseSuccess or
@@ -86,7 +86,7 @@ If the client fails to send a message (even if its a ping message) every minute
 (by default), the connection is terminated.
 
 Temporary protocol submode
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 A temporary protocol submode begins with ConnectRequest message, to which the
 relay responds with either ResponseNotFound if the device the client it is after
@@ -135,7 +135,7 @@ After the successful response, all the bytes written and received will be
 relayed between the two devices in the session directly.
 
 Example Exchange
-~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^
 
 Client A - Permanent protocol mode
 Client B - Temporary protocol mode
@@ -167,7 +167,7 @@ magic value 0x9E79BC40, message type integer, and message length.
 
 
 Header structure
-~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^
 
 ::
 
@@ -189,7 +189,7 @@ Header structure
 	}
 
 Ping message (Type = 0)
-~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^
 
 ::
 
@@ -202,7 +202,7 @@ Ping message (Type = 0)
 	}
 
 Pong message (Type = 1)
-~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^
 
 ::
 
@@ -215,7 +215,7 @@ Pong message (Type = 1)
 	}
 
 JoinRelayRequest message (Type = 2)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 ::
 
@@ -228,7 +228,7 @@ JoinRelayRequest message (Type = 2)
 	}
 
 JoinSessionRequest message (Type = 3)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 ::
 
@@ -253,7 +253,7 @@ JoinSessionRequest message (Type = 3)
 
 
 Response message (Type = 4)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 ::
 
@@ -282,13 +282,13 @@ Response message (Type = 4)
 
 .. Protocol defined responses:
 	1. ResponseSuccess           = Response{0, "success"}
-	2 .ResponseNotFound          = Response{1, "not found"}
+	2. ResponseNotFound          = Response{1, "not found"}
 	3. ResponseAlreadyConnected  = Response{2, "already connected"}
 	4. ResponseInternalError     = Response{99, "internal error"}
 	5. ResponseUnexpectedMessage = Response{100, "unexpected message"}
 
 ConnectRequest message (Type = 5)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 ::
 
@@ -312,7 +312,7 @@ ConnectRequest message (Type = 5)
 
 
 SessionInvitation message (Type = 6)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 ::