refactor: use modern Protobuf encoder (#9817)

At a high level, this is what I've done and why: - I'm moving the protobuf generation for the `protocol`, `discovery` and `db` packages to the modern alternatives, and using `buf` to generate because it's nice and simple. - After trying various approaches on how to integrate the new types with the existing code, I opted for splitting off our own data model types from the on-the-wire generated types. This means we can have a `FileInfo` type with nicer ergonomics and lots of methods, while the protobuf generated type stays clean and close to the wire protocol. It does mean copying between the two when required, which certainly adds a small amount of inefficiency. If we want to walk this back in the future and use the raw generated type throughout, that's possible, this however makes the refactor smaller (!) as it doesn't change everything about the type for everyone at the same time. - I have simply removed in cold blood a significant number of old database migrations. These depended on previous generations of generated messages of various kinds and were annoying to support in the new fashion. The oldest supported database version now is the one from Syncthing 1.9.0 from Sep 7, 2020. - I changed config structs to be regular manually defined structs. For the sake of discussion, some things I tried that turned out not to work... ### Embedding / wrapping Embedding the protobuf generated structs in our existing types as a data container and keeping our methods and stuff: ``` package protocol type FileInfo struct { *generated.FileInfo } ``` This generates a lot of problems because the internal shape of the generated struct is quite different (different names, different types, more pointers), because initializing it doesn't work like you'd expect (i.e., you end up with an embedded nil pointer and a panic), and because the types of child types don't get wrapped. That is, even if we also have a similar wrapper around a `Vector`, that's not the type you get when accessing `someFileInfo.Version`, you get the `*generated.Vector` that doesn't have methods, etc. ### Aliasing ``` package protocol type FileInfo = generated.FileInfo ``` Doesn't help because you can't attach methods to it, plus all the above. ### Generating the types into the target package like we do now and attaching methods This fails because of the different shape of the generated type (as in the embedding case above) plus the generated struct already has a bunch of methods that we can't necessarily override properly (like `String()` and a bunch of getters). ### Methods to functions I considered just moving all the methods we attach to functions in a specific package, so that for example ``` package protocol func (f FileInfo) Equal(other FileInfo) bool ``` would become ``` package fileinfos func Equal(a, b *generated.FileInfo) bool ``` and this would mostly work, but becomes quite verbose and cumbersome, and somewhat limits discoverability (you can't see what methods are available on the type in auto completions, etc). In the end I did this in some cases, like in the database layer where a lot of things like `func (fv *FileVersion) IsEmpty() bool` becomes `func fvIsEmpty(fv *generated.FileVersion)` because they were anyway just internal methods. Fixes #8247
2024-12-01 16:50:17 +01:00
parent 2b8ee4c7a5
commit 77970d5113
203 changed files with 7437 additions and 28636 deletions
@@ -1,19 +0,0 @@
-Copyright (C) 2014-2015 The Protocol Authors
-
-Permission is hereby granted, free of charge, to any person obtaining a copy of
-this software and associated documentation files (the "Software"), to deal in
-the Software without restriction, including without limitation the rights to
-use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
-of the Software, and to permit persons to whom the Software is furnished to do
-so, subject to the following conditions:
-
- The above copyright notice and this permission notice shall be included in
-  all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
@@ -1,4 +1,8 @@
-// Copyright (C) 2016 The Protocol Authors.
+// Copyright (C) 2016 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -0,0 +1,147 @@
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+package protocol
+
+import (
+	"fmt"
+
+	"github.com/syncthing/syncthing/internal/gen/bep"
+)
+
+type Compression = bep.Compression
+
+const (
+	CompressionMetadata = bep.Compression_COMPRESSION_METADATA
+	CompressionNever    = bep.Compression_COMPRESSION_NEVER
+	CompressionAlways   = bep.Compression_COMPRESSION_ALWAYS
+)
+
+type ClusterConfig struct {
+	Folders   []Folder
+	Secondary bool
+}
+
+func (c *ClusterConfig) toWire() *bep.ClusterConfig {
+	folders := make([]*bep.Folder, len(c.Folders))
+	for i, f := range c.Folders {
+		folders[i] = f.toWire()
+	}
+	return &bep.ClusterConfig{
+		Folders:   folders,
+		Secondary: c.Secondary,
+	}
+}
+
+func clusterConfigFromWire(w *bep.ClusterConfig) *ClusterConfig {
+	if w == nil {
+		return nil
+	}
+	c := &ClusterConfig{
+		Secondary: w.Secondary,
+	}
+	c.Folders = make([]Folder, len(w.Folders))
+	for i, f := range w.Folders {
+		c.Folders[i] = folderFromWire(f)
+	}
+	return c
+}
+
+type Folder struct {
+	ID                 string
+	Label              string
+	ReadOnly           bool
+	IgnorePermissions  bool
+	IgnoreDelete       bool
+	DisableTempIndexes bool
+	Paused             bool
+	Devices            []Device
+}
+
+func (f *Folder) toWire() *bep.Folder {
+	devices := make([]*bep.Device, len(f.Devices))
+	for i, d := range f.Devices {
+		devices[i] = d.toWire()
+	}
+	return &bep.Folder{
+		Id:                 f.ID,
+		Label:              f.Label,
+		ReadOnly:           f.ReadOnly,
+		IgnorePermissions:  f.IgnorePermissions,
+		IgnoreDelete:       f.IgnoreDelete,
+		DisableTempIndexes: f.DisableTempIndexes,
+		Paused:             f.Paused,
+		Devices:            devices,
+	}
+}
+
+func folderFromWire(w *bep.Folder) Folder {
+	devices := make([]Device, len(w.Devices))
+	for i, d := range w.Devices {
+		devices[i] = deviceFromWire(d)
+	}
+	return Folder{
+		ID:                 w.Id,
+		Label:              w.Label,
+		ReadOnly:           w.ReadOnly,
+		IgnorePermissions:  w.IgnorePermissions,
+		IgnoreDelete:       w.IgnoreDelete,
+		DisableTempIndexes: w.DisableTempIndexes,
+		Paused:             w.Paused,
+		Devices:            devices,
+	}
+}
+
+func (f Folder) Description() string {
+	// used by logging stuff
+	if f.Label == "" {
+		return f.ID
+	}
+	return fmt.Sprintf("%q (%s)", f.Label, f.ID)
+}
+
+type Device struct {
+	ID                       DeviceID
+	Name                     string
+	Addresses                []string
+	Compression              Compression
+	CertName                 string
+	MaxSequence              int64
+	Introducer               bool
+	IndexID                  IndexID
+	SkipIntroductionRemovals bool
+	EncryptionPasswordToken  []byte
+}
+
+func (d *Device) toWire() *bep.Device {
+	return &bep.Device{
+		Id:                       d.ID[:],
+		Name:                     d.Name,
+		Addresses:                d.Addresses,
+		Compression:              d.Compression,
+		CertName:                 d.CertName,
+		MaxSequence:              d.MaxSequence,
+		Introducer:               d.Introducer,
+		IndexId:                  uint64(d.IndexID),
+		SkipIntroductionRemovals: d.SkipIntroductionRemovals,
+		EncryptionPasswordToken:  d.EncryptionPasswordToken,
+	}
+}
+
+func deviceFromWire(w *bep.Device) Device {
+	return Device{
+		ID:                       DeviceID(w.Id),
+		Name:                     w.Name,
+		Addresses:                w.Addresses,
+		Compression:              w.Compression,
+		CertName:                 w.CertName,
+		MaxSequence:              w.MaxSequence,
+		Introducer:               w.Introducer,
+		IndexID:                  IndexID(w.IndexId),
+		SkipIntroductionRemovals: w.SkipIntroductionRemovals,
+		EncryptionPasswordToken:  w.EncryptionPasswordToken,
+	}
+}
@@ -0,0 +1,79 @@
+// Copyright (C) 2016 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+package protocol
+
+import "github.com/syncthing/syncthing/internal/gen/bep"
+
+type FileDownloadProgressUpdateType = bep.FileDownloadProgressUpdateType
+
+const (
+	FileDownloadProgressUpdateTypeAppend = bep.FileDownloadProgressUpdateType_FILE_DOWNLOAD_PROGRESS_UPDATE_TYPE_APPEND
+	FileDownloadProgressUpdateTypeForget = bep.FileDownloadProgressUpdateType_FILE_DOWNLOAD_PROGRESS_UPDATE_TYPE_FORGET
+)
+
+type DownloadProgress struct {
+	Folder  string
+	Updates []FileDownloadProgressUpdate
+}
+
+func (d *DownloadProgress) toWire() *bep.DownloadProgress {
+	updates := make([]*bep.FileDownloadProgressUpdate, len(d.Updates))
+	for i, u := range d.Updates {
+		updates[i] = u.toWire()
+	}
+	return &bep.DownloadProgress{
+		Folder:  d.Folder,
+		Updates: updates,
+	}
+}
+
+func downloadProgressFromWire(w *bep.DownloadProgress) *DownloadProgress {
+	dp := &DownloadProgress{
+		Folder:  w.Folder,
+		Updates: make([]FileDownloadProgressUpdate, len(w.Updates)),
+	}
+	for i, u := range w.Updates {
+		dp.Updates[i] = fileDownloadProgressUpdateFromWire(u)
+	}
+	return dp
+}
+
+type FileDownloadProgressUpdate struct {
+	UpdateType   FileDownloadProgressUpdateType
+	Name         string
+	Version      Vector
+	BlockIndexes []int
+	BlockSize    int
+}
+
+func (f *FileDownloadProgressUpdate) toWire() *bep.FileDownloadProgressUpdate {
+	bidxs := make([]int32, len(f.BlockIndexes))
+	for i, b := range f.BlockIndexes {
+		bidxs[i] = int32(b)
+	}
+	return &bep.FileDownloadProgressUpdate{
+		UpdateType:   f.UpdateType,
+		Name:         f.Name,
+		Version:      f.Version.ToWire(),
+		BlockIndexes: bidxs,
+		BlockSize:    int32(f.BlockSize),
+	}
+}
+
+func fileDownloadProgressUpdateFromWire(w *bep.FileDownloadProgressUpdate) FileDownloadProgressUpdate {
+	bidxs := make([]int, len(w.BlockIndexes))
+	for i, b := range w.BlockIndexes {
+		bidxs[i] = int(b)
+	}
+	return FileDownloadProgressUpdate{
+		UpdateType:   w.UpdateType,
+		Name:         w.Name,
+		Version:      VectorFromWire(w.Version),
+		BlockIndexes: bidxs,
+		BlockSize:    int(w.BlockSize),
+	}
+}
@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -6,51 +10,240 @@ import (
 	"bytes"
 	"crypto/sha256"
 	"encoding/binary"
-	"encoding/json"
-	"errors"
 	"fmt"
 	"time"

+	"github.com/syncthing/syncthing/internal/gen/bep"
 	"github.com/syncthing/syncthing/lib/build"
-	"github.com/syncthing/syncthing/lib/rand"
 )

+// FileInfo.LocalFlags flags
 const (
-	SyntheticDirectorySize        = 128
-	HelloMessageMagic      uint32 = 0x2EA7D90B
-	Version13HelloMagic    uint32 = 0x9F79BC40 // old
+	FlagLocalUnsupported = 1 << 0 // The kind is unsupported, e.g. symlinks on Windows
+	FlagLocalIgnored     = 1 << 1 // Matches local ignore patterns
+	FlagLocalMustRescan  = 1 << 2 // Doesn't match content on disk, must be rechecked fully
+	FlagLocalReceiveOnly = 1 << 3 // Change detected on receive only folder
+
+	// Flags that should result in the Invalid bit on outgoing updates
+	LocalInvalidFlags = FlagLocalUnsupported | FlagLocalIgnored | FlagLocalMustRescan | FlagLocalReceiveOnly
+
+	// Flags that should result in a file being in conflict with its
+	// successor, due to us not having an up to date picture of its state on
+	// disk.
+	LocalConflictFlags = FlagLocalUnsupported | FlagLocalIgnored | FlagLocalReceiveOnly
+
+	LocalAllFlags = FlagLocalUnsupported | FlagLocalIgnored | FlagLocalMustRescan | FlagLocalReceiveOnly
 )

-// FileIntf is the set of methods implemented by both FileInfo and
-// db.FileInfoTruncated.
-type FileIntf interface {
-	FileSize() int64
-	FileName() string
-	FileLocalFlags() uint32
-	IsDeleted() bool
-	IsInvalid() bool
-	IsIgnored() bool
-	IsUnsupported() bool
-	MustRescan() bool
-	IsReceiveOnlyChanged() bool
-	IsDirectory() bool
-	IsSymlink() bool
-	ShouldConflict() bool
-	HasPermissionBits() bool
-	SequenceNo() int64
-	BlockSize() int
-	FileVersion() Vector
-	FileType() FileInfoType
-	FilePermissions() uint32
-	FileModifiedBy() ShortID
-	ModTime() time.Time
-	PlatformData() PlatformData
-	InodeChangeTime() time.Time
-	FileBlocksHash() []byte
+// BlockSizes is the list of valid block sizes, from min to max
+var BlockSizes []int
+
+func init() {
+	for blockSize := MinBlockSize; blockSize <= MaxBlockSize; blockSize *= 2 {
+		BlockSizes = append(BlockSizes, blockSize)
+		if _, ok := sha256OfEmptyBlock[blockSize]; !ok {
+			panic("missing hard coded value for sha256 of empty block")
+		}
+	}
+	BufferPool = newBufferPool() // must happen after BlockSizes is initialized
 }

-func (Hello) Magic() uint32 {
-	return HelloMessageMagic
+type FileInfoType = bep.FileInfoType
+
+const (
+	FileInfoTypeFile             = bep.FileInfoType_FILE_INFO_TYPE_FILE
+	FileInfoTypeDirectory        = bep.FileInfoType_FILE_INFO_TYPE_DIRECTORY
+	FileInfoTypeSymlinkFile      = bep.FileInfoType_FILE_INFO_TYPE_SYMLINK_FILE
+	FileInfoTypeSymlinkDirectory = bep.FileInfoType_FILE_INFO_TYPE_SYMLINK_DIRECTORY
+	FileInfoTypeSymlink          = bep.FileInfoType_FILE_INFO_TYPE_SYMLINK
+)
+
+type FileInfo struct {
+	Name          string
+	Size          int64
+	ModifiedS     int64
+	ModifiedBy    ShortID
+	Version       Vector
+	Sequence      int64
+	Blocks        []BlockInfo
+	SymlinkTarget string
+	BlocksHash    []byte
+	Encrypted     []byte
+	Platform      PlatformData
+
+	Type         FileInfoType
+	Permissions  uint32
+	ModifiedNs   int32
+	RawBlockSize int32
+
+	// The local_flags fields stores flags that are relevant to the local
+	// host only. It is not part of the protocol, doesn't get sent or
+	// received (we make sure to zero it), nonetheless we need it on our
+	// struct and to be able to serialize it to/from the database.
+	LocalFlags uint32
+
+	// The version_hash is an implementation detail and not part of the wire
+	// format.
+	VersionHash []byte
+
+	// The time when the inode was last changed (i.e., permissions, xattrs
+	// etc changed). This is host-local, not sent over the wire.
+	InodeChangeNs int64
+
+	// The size of the data appended to the encrypted file on disk. This is
+	// host-local, not sent over the wire.
+	EncryptionTrailerSize int
+
+	Deleted       bool
+	RawInvalid    bool
+	NoPermissions bool
+
+	truncated bool // was created from a truncated file info without blocks
+}
+
+func (f *FileInfo) ToWire(withInternalFields bool) *bep.FileInfo {
+	if f.truncated && !(f.IsDeleted() || f.IsInvalid() || f.IsIgnored()) {
+		panic("bug: must not serialize truncated file info")
+	}
+	blocks := make([]*bep.BlockInfo, len(f.Blocks))
+	for j, b := range f.Blocks {
+		blocks[j] = b.ToWire()
+	}
+	w := &bep.FileInfo{
+		Name:          f.Name,
+		Size:          f.Size,
+		ModifiedS:     f.ModifiedS,
+		ModifiedBy:    uint64(f.ModifiedBy),
+		Version:       f.Version.ToWire(),
+		Sequence:      f.Sequence,
+		Blocks:        blocks,
+		SymlinkTarget: f.SymlinkTarget,
+		BlocksHash:    f.BlocksHash,
+		Encrypted:     f.Encrypted,
+		Type:          f.Type,
+		Permissions:   f.Permissions,
+		ModifiedNs:    f.ModifiedNs,
+		BlockSize:     f.RawBlockSize,
+		Platform:      f.Platform.toWire(),
+		Deleted:       f.Deleted,
+		Invalid:       f.RawInvalid,
+		NoPermissions: f.NoPermissions,
+	}
+	if withInternalFields {
+		w.LocalFlags = f.LocalFlags
+		w.VersionHash = f.VersionHash
+		w.InodeChangeNs = f.InodeChangeNs
+		w.EncryptionTrailerSize = int32(f.EncryptionTrailerSize)
+	}
+	return w
+}
+
+// WinsConflict returns true if "f" is the one to choose when it is in
+// conflict with "other".
+func (f *FileInfo) WinsConflict(other FileInfo) bool {
+	// If only one of the files is invalid, that one loses.
+	if f.IsInvalid() != other.IsInvalid() {
+		return !f.IsInvalid()
+	}
+
+	// If a modification is in conflict with a delete, we pick the
+	// modification.
+	if !f.IsDeleted() && other.IsDeleted() {
+		return true
+	}
+	if f.IsDeleted() && !other.IsDeleted() {
+		return false
+	}
+
+	// The one with the newer modification time wins.
+	if f.ModTime().After(other.ModTime()) {
+		return true
+	}
+	if f.ModTime().Before(other.ModTime()) {
+		return false
+	}
+
+	// The modification times were equal. Use the device ID in the version
+	// vector as tie breaker.
+	return f.FileVersion().Compare(other.FileVersion()) == ConcurrentGreater
+}
+
+func FileInfoFromWire(w *bep.FileInfo) FileInfo {
+	var blocks []BlockInfo
+	if len(w.Blocks) > 0 {
+		blocks = make([]BlockInfo, len(w.Blocks))
+		for j, b := range w.Blocks {
+			blocks[j] = BlockInfoFromWire(b)
+		}
+	}
+	return fileInfoFromWireWithBlocks(w, blocks)
+}
+
+type FileInfoWithoutBlocks interface {
+	GetName() string
+	GetSize() int64
+	GetModifiedS() int64
+	GetModifiedBy() uint64
+	GetVersion() *bep.Vector
+	GetSequence() int64
+	// GetBlocks() []*bep.BlockInfo // not included
+	GetSymlinkTarget() string
+	GetBlocksHash() []byte
+	GetEncrypted() []byte
+	GetType() FileInfoType
+	GetPermissions() uint32
+	GetModifiedNs() int32
+	GetBlockSize() int32
+	GetPlatform() *bep.PlatformData
+	GetLocalFlags() uint32
+	GetVersionHash() []byte
+	GetInodeChangeNs() int64
+	GetEncryptionTrailerSize() int32
+	GetDeleted() bool
+	GetInvalid() bool
+	GetNoPermissions() bool
+}
+
+func fileInfoFromWireWithBlocks(w FileInfoWithoutBlocks, blocks []BlockInfo) FileInfo {
+	return FileInfo{
+		Name:          w.GetName(),
+		Size:          w.GetSize(),
+		ModifiedS:     w.GetModifiedS(),
+		ModifiedBy:    ShortID(w.GetModifiedBy()),
+		Version:       VectorFromWire(w.GetVersion()),
+		Sequence:      w.GetSequence(),
+		Blocks:        blocks,
+		SymlinkTarget: w.GetSymlinkTarget(),
+		BlocksHash:    w.GetBlocksHash(),
+		Encrypted:     w.GetEncrypted(),
+		Type:          w.GetType(),
+		Permissions:   w.GetPermissions(),
+		ModifiedNs:    w.GetModifiedNs(),
+		RawBlockSize:  w.GetBlockSize(),
+		Platform:      platformDataFromWire(w.GetPlatform()),
+		Deleted:       w.GetDeleted(),
+		RawInvalid:    w.GetInvalid(),
+		NoPermissions: w.GetNoPermissions(),
+	}
+}
+
+func FileInfoFromDB(w *bep.FileInfo) FileInfo {
+	f := FileInfoFromWire(w)
+	f.LocalFlags = w.LocalFlags
+	f.VersionHash = w.VersionHash
+	f.InodeChangeNs = w.InodeChangeNs
+	f.EncryptionTrailerSize = int(w.EncryptionTrailerSize)
+	return f
+}
+
+func FileInfoFromDBTruncated(w FileInfoWithoutBlocks) FileInfo {
+	f := fileInfoFromWireWithBlocks(w, nil)
+	f.LocalFlags = w.GetLocalFlags()
+	f.VersionHash = w.GetVersionHash()
+	f.InodeChangeNs = w.GetInodeChangeNs()
+	f.EncryptionTrailerSize = int(w.GetEncryptionTrailerSize())
+	f.truncated = true
+	return f
 }

 func (f FileInfo) String() string {
@@ -128,7 +321,19 @@ func (f FileInfo) BlockSize() int {
 	if f.RawBlockSize < MinBlockSize {
 		return MinBlockSize
 	}
-	return f.RawBlockSize
+	return int(f.RawBlockSize)
+}
+
+// BlockSize returns the block size to use for the given file size
+func BlockSize(fileSize int64) int {
+	var blockSize int
+	for _, blockSize = range BlockSizes {
+		if fileSize < DesiredPerFileBlocks*int64(blockSize) {
+			break
+		}
+	}
+
+	return blockSize
 }

 func (f FileInfo) FileName() string {
@@ -175,36 +380,6 @@ func (f FileInfo) FileBlocksHash() []byte {
 	return f.BlocksHash
 }

-// WinsConflict returns true if "f" is the one to choose when it is in
-// conflict with "other".
-func WinsConflict(f, other FileIntf) bool {
-	// If only one of the files is invalid, that one loses.
-	if f.IsInvalid() != other.IsInvalid() {
-		return !f.IsInvalid()
-	}
-
-	// If a modification is in conflict with a delete, we pick the
-	// modification.
-	if !f.IsDeleted() && other.IsDeleted() {
-		return true
-	}
-	if f.IsDeleted() && !other.IsDeleted() {
-		return false
-	}
-
-	// The one with the newer modification time wins.
-	if f.ModTime().After(other.ModTime()) {
-		return true
-	}
-	if f.ModTime().Before(other.ModTime()) {
-		return false
-	}
-
-	// The modification times were equal. Use the device ID in the version
-	// vector as tie breaker.
-	return f.FileVersion().Compare(other.FileVersion()) == ConcurrentGreater
-}
-
 type FileInfoComparison struct {
 	ModTimeWindow   time.Duration
 	IgnorePerms     bool
@@ -336,18 +511,121 @@ func (f FileInfo) BlocksEqual(other FileInfo) bool {
 	return blocksEqual(f.Blocks, other.Blocks)
 }

+func (f *FileInfo) SetMustRescan() {
+	f.setLocalFlags(FlagLocalMustRescan)
+}
+
+func (f *FileInfo) SetIgnored() {
+	f.setLocalFlags(FlagLocalIgnored)
+}
+
+func (f *FileInfo) SetUnsupported() {
+	f.setLocalFlags(FlagLocalUnsupported)
+}
+
+func (f *FileInfo) SetDeleted(by ShortID) {
+	f.ModifiedBy = by
+	f.Deleted = true
+	f.Version = f.Version.Update(by)
+	f.ModifiedS = time.Now().Unix()
+	f.setNoContent()
+}
+
+func (f *FileInfo) setLocalFlags(flags uint32) {
+	f.RawInvalid = false
+	f.LocalFlags = flags
+	f.setNoContent()
+}
+
+func (f *FileInfo) setNoContent() {
+	f.Blocks = nil
+	f.BlocksHash = nil
+	f.Size = 0
+}
+
+type BlockInfo struct {
+	Hash     []byte
+	Offset   int64
+	Size     int
+	WeakHash uint32
+}
+
+func (b BlockInfo) ToWire() *bep.BlockInfo {
+	return &bep.BlockInfo{
+		Hash:     b.Hash,
+		Offset:   b.Offset,
+		Size:     int32(b.Size),
+		WeakHash: b.WeakHash,
+	}
+}
+
+func BlockInfoFromWire(w *bep.BlockInfo) BlockInfo {
+	return BlockInfo{
+		Hash:     w.Hash,
+		Offset:   w.Offset,
+		Size:     int(w.Size),
+		WeakHash: w.WeakHash,
+	}
+}
+
+func (b BlockInfo) String() string {
+	return fmt.Sprintf("Block{%d/%d/%d/%x}", b.Offset, b.Size, b.WeakHash, b.Hash)
+}
+
+// For each block size, the hash of a block of all zeroes
+var sha256OfEmptyBlock = map[int][sha256.Size]byte{
+	128 << KiB: {0xfa, 0x43, 0x23, 0x9b, 0xce, 0xe7, 0xb9, 0x7c, 0xa6, 0x2f, 0x0, 0x7c, 0xc6, 0x84, 0x87, 0x56, 0xa, 0x39, 0xe1, 0x9f, 0x74, 0xf3, 0xdd, 0xe7, 0x48, 0x6d, 0xb3, 0xf9, 0x8d, 0xf8, 0xe4, 0x71},
+	256 << KiB: {0x8a, 0x39, 0xd2, 0xab, 0xd3, 0x99, 0x9a, 0xb7, 0x3c, 0x34, 0xdb, 0x24, 0x76, 0x84, 0x9c, 0xdd, 0xf3, 0x3, 0xce, 0x38, 0x9b, 0x35, 0x82, 0x68, 0x50, 0xf9, 0xa7, 0x0, 0x58, 0x9b, 0x4a, 0x90},
+	512 << KiB: {0x7, 0x85, 0x4d, 0x2f, 0xef, 0x29, 0x7a, 0x6, 0xba, 0x81, 0x68, 0x5e, 0x66, 0xc, 0x33, 0x2d, 0xe3, 0x6d, 0x5d, 0x18, 0xd5, 0x46, 0x92, 0x7d, 0x30, 0xda, 0xad, 0x6d, 0x7f, 0xda, 0x15, 0x41},
+	1 << MiB:   {0x30, 0xe1, 0x49, 0x55, 0xeb, 0xf1, 0x35, 0x22, 0x66, 0xdc, 0x2f, 0xf8, 0x6, 0x7e, 0x68, 0x10, 0x46, 0x7, 0xe7, 0x50, 0xab, 0xb9, 0xd3, 0xb3, 0x65, 0x82, 0xb8, 0xaf, 0x90, 0x9f, 0xcb, 0x58},
+	2 << MiB:   {0x56, 0x47, 0xf0, 0x5e, 0xc1, 0x89, 0x58, 0x94, 0x7d, 0x32, 0x87, 0x4e, 0xeb, 0x78, 0x8f, 0xa3, 0x96, 0xa0, 0x5d, 0xb, 0xab, 0x7c, 0x1b, 0x71, 0xf1, 0x12, 0xce, 0xb7, 0xe9, 0xb3, 0x1e, 0xee},
+	4 << MiB:   {0xbb, 0x9f, 0x8d, 0xf6, 0x14, 0x74, 0xd2, 0x5e, 0x71, 0xfa, 0x0, 0x72, 0x23, 0x18, 0xcd, 0x38, 0x73, 0x96, 0xca, 0x17, 0x36, 0x60, 0x5e, 0x12, 0x48, 0x82, 0x1c, 0xc0, 0xde, 0x3d, 0x3a, 0xf8},
+	8 << MiB:   {0x2d, 0xae, 0xb1, 0xf3, 0x60, 0x95, 0xb4, 0x4b, 0x31, 0x84, 0x10, 0xb3, 0xf4, 0xe8, 0xb5, 0xd9, 0x89, 0xdc, 0xc7, 0xbb, 0x2, 0x3d, 0x14, 0x26, 0xc4, 0x92, 0xda, 0xb0, 0xa3, 0x5, 0x3e, 0x74},
+	16 << MiB:  {0x8, 0xa, 0xcf, 0x35, 0xa5, 0x7, 0xac, 0x98, 0x49, 0xcf, 0xcb, 0xa4, 0x7d, 0xc2, 0xad, 0x83, 0xe0, 0x1b, 0x75, 0x66, 0x3a, 0x51, 0x62, 0x79, 0xc8, 0xb9, 0xd2, 0x43, 0xb7, 0x19, 0x64, 0x3e},
+}
+
+// IsEmpty returns true if the block is a full block of zeroes.
+func (b BlockInfo) IsEmpty() bool {
+	if v, ok := sha256OfEmptyBlock[b.Size]; ok {
+		return bytes.Equal(b.Hash, v[:])
+	}
+	return false
+}
+
+func BlocksHash(bs []BlockInfo) []byte {
+	h := sha256.New()
+	for _, b := range bs {
+		_, _ = h.Write(b.Hash)
+		_ = binary.Write(h, binary.BigEndian, b.WeakHash)
+	}
+	return h.Sum(nil)
+}
+
+func VectorHash(v Vector) []byte {
+	h := sha256.New()
+	for _, c := range v.Counters {
+		if err := binary.Write(h, binary.BigEndian, c.ID); err != nil {
+			panic("impossible: failed to write c.ID to hash function: " + err.Error())
+		}
+		if err := binary.Write(h, binary.BigEndian, c.Value); err != nil {
+			panic("impossible: failed to write c.Value to hash function: " + err.Error())
+		}
+	}
+	return h.Sum(nil)
+}
+
 // Xattrs is a convenience method to return the extended attributes of the
 // file for the current platform.
-func (f *PlatformData) Xattrs() []Xattr {
+func (p *PlatformData) Xattrs() []Xattr {
 	switch {
-	case build.IsLinux && f.Linux != nil:
-		return f.Linux.Xattrs
-	case build.IsDarwin && f.Darwin != nil:
-		return f.Darwin.Xattrs
-	case build.IsFreeBSD && f.FreeBSD != nil:
-		return f.FreeBSD.Xattrs
-	case build.IsNetBSD && f.NetBSD != nil:
-		return f.NetBSD.Xattrs
+	case build.IsLinux && p.Linux != nil:
+		return p.Linux.Xattrs
+	case build.IsDarwin && p.Darwin != nil:
+		return p.Darwin.Xattrs
+	case build.IsFreeBSD && p.FreeBSD != nil:
+		return p.FreeBSD.Xattrs
+	case build.IsNetBSD && p.NetBSD != nil:
+		return p.NetBSD.Xattrs
 	default:
 		return nil
 	}
@@ -422,119 +700,122 @@ func blocksEqual(a, b []BlockInfo) bool {
 	return true
 }

-func (f *FileInfo) SetMustRescan() {
-	f.setLocalFlags(FlagLocalMustRescan)
+type PlatformData struct {
+	Unix    *UnixData
+	Windows *WindowsData
+	Linux   *XattrData
+	Darwin  *XattrData
+	FreeBSD *XattrData
+	NetBSD  *XattrData
 }

-func (f *FileInfo) SetIgnored() {
-	f.setLocalFlags(FlagLocalIgnored)
-}
-
-func (f *FileInfo) SetUnsupported() {
-	f.setLocalFlags(FlagLocalUnsupported)
-}
-
-func (f *FileInfo) SetDeleted(by ShortID) {
-	f.ModifiedBy = by
-	f.Deleted = true
-	f.Version = f.Version.Update(by)
-	f.ModifiedS = time.Now().Unix()
-	f.setNoContent()
-}
-
-func (f *FileInfo) setLocalFlags(flags uint32) {
-	f.RawInvalid = false
-	f.LocalFlags = flags
-	f.setNoContent()
-}
-
-func (f *FileInfo) setNoContent() {
-	f.Blocks = nil
-	f.BlocksHash = nil
-	f.Size = 0
-}
-
-func (b BlockInfo) String() string {
-	return fmt.Sprintf("Block{%d/%d/%d/%x}", b.Offset, b.Size, b.WeakHash, b.Hash)
-}
-
-// IsEmpty returns true if the block is a full block of zeroes.
-func (b BlockInfo) IsEmpty() bool {
-	if v, ok := sha256OfEmptyBlock[int(b.Size)]; ok {
-		return bytes.Equal(b.Hash, v[:])
+func (p *PlatformData) toWire() *bep.PlatformData {
+	return &bep.PlatformData{
+		Unix:    p.Unix.toWire(),
+		Windows: p.Windows,
+		Linux:   p.Linux.toWire(),
+		Darwin:  p.Darwin.toWire(),
+		Freebsd: p.FreeBSD.toWire(),
+		Netbsd:  p.NetBSD.toWire(),
 	}
-	return false
 }

-type IndexID uint64
-
-func (i IndexID) String() string {
-	return fmt.Sprintf("0x%016X", uint64(i))
-}
-
-func (i IndexID) Marshal() ([]byte, error) {
-	bs := make([]byte, 8)
-	binary.BigEndian.PutUint64(bs, uint64(i))
-	return bs, nil
-}
-
-func (i *IndexID) Unmarshal(bs []byte) error {
-	if len(bs) != 8 {
-		return errors.New("incorrect IndexID length")
+func platformDataFromWire(w *bep.PlatformData) PlatformData {
+	if w == nil {
+		return PlatformData{}
 	}
-	*i = IndexID(binary.BigEndian.Uint64(bs))
-	return nil
-}
-
-func NewIndexID() IndexID {
-	return IndexID(rand.Uint64())
-}
-
-func (f Folder) Description() string {
-	// used by logging stuff
-	if f.Label == "" {
-		return f.ID
+	return PlatformData{
+		Unix:    unixDataFromWire(w.Unix),
+		Windows: w.Windows,
+		Linux:   xattrDataFromWire(w.Linux),
+		Darwin:  xattrDataFromWire(w.Darwin),
+		FreeBSD: xattrDataFromWire(w.Freebsd),
+		NetBSD:  xattrDataFromWire(w.Netbsd),
 	}
-	return fmt.Sprintf("%q (%s)", f.Label, f.ID)
 }

-func BlocksHash(bs []BlockInfo) []byte {
-	h := sha256.New()
-	for _, b := range bs {
-		_, _ = h.Write(b.Hash)
-		_ = binary.Write(h, binary.BigEndian, b.WeakHash)
-	}
-	return h.Sum(nil)
+type UnixData struct {
+	// The owner name and group name are set when known (i.e., could be
+	// resolved on the source device), while the UID and GID are always set
+	// as they come directly from the stat() call.
+	OwnerName string
+	GroupName string
+	UID       int
+	GID       int
 }

-func VectorHash(v Vector) []byte {
-	h := sha256.New()
-	for _, c := range v.Counters {
-		if err := binary.Write(h, binary.BigEndian, c.ID); err != nil {
-			panic("impossible: failed to write c.ID to hash function: " + err.Error())
-		}
-		if err := binary.Write(h, binary.BigEndian, c.Value); err != nil {
-			panic("impossible: failed to write c.Value to hash function: " + err.Error())
-		}
+func (u *UnixData) toWire() *bep.UnixData {
+	if u == nil {
+		return nil
+	}
+	return &bep.UnixData{
+		OwnerName: u.OwnerName,
+		GroupName: u.GroupName,
+		Uid:       int32(u.UID),
+		Gid:       int32(u.GID),
 	}
-	return h.Sum(nil)
 }

-func (x *FileInfoType) MarshalJSON() ([]byte, error) {
-	return json.Marshal(x.String())
+func unixDataFromWire(w *bep.UnixData) *UnixData {
+	if w == nil {
+		return nil
+	}
+	return &UnixData{
+		OwnerName: w.OwnerName,
+		GroupName: w.GroupName,
+		UID:       int(w.Uid),
+		GID:       int(w.Gid),
+	}
 }

-func (x *FileInfoType) UnmarshalJSON(data []byte) error {
-	var s string
-	if err := json.Unmarshal(data, &s); err != nil {
-		return err
+type WindowsData = bep.WindowsData
+
+type XattrData struct {
+	Xattrs []Xattr
+}
+
+func (x *XattrData) toWire() *bep.XattrData {
+	if x == nil {
+		return nil
 	}
-	n, ok := FileInfoType_value[s]
-	if !ok {
-		return errors.New("invalid value: " + s)
+	xattrs := make([]*bep.Xattr, len(x.Xattrs))
+	for i, a := range x.Xattrs {
+		xattrs[i] = a.toWire()
+	}
+	return &bep.XattrData{
+		Xattrs: xattrs,
+	}
+}
+
+func xattrDataFromWire(w *bep.XattrData) *XattrData {
+	if w == nil {
+		return nil
+	}
+	x := &XattrData{}
+	x.Xattrs = make([]Xattr, len(w.Xattrs))
+	for i, a := range w.Xattrs {
+		x.Xattrs[i] = xattrFromWire(a)
+	}
+	return x
+}
+
+type Xattr struct {
+	Name  string
+	Value []byte
+}
+
+func (a Xattr) toWire() *bep.Xattr {
+	return &bep.Xattr{
+		Name:  a.Name,
+		Value: a.Value,
+	}
+}
+
+func xattrFromWire(w *bep.Xattr) Xattr {
+	return Xattr{
+		Name:  w.Name,
+		Value: w.Value,
 	}
-	*x = FileInfoType(n)
-	return nil
 }

 func xattrsEqual(a, b *XattrData) bool {
@@ -0,0 +1,293 @@
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+package protocol
+
+import (
+	"crypto/sha256"
+	"testing"
+
+	"github.com/syncthing/syncthing/lib/build"
+)
+
+func TestLocalFlagBits(t *testing.T) {
+	var f FileInfo
+	if f.IsIgnored() || f.MustRescan() || f.IsInvalid() {
+		t.Error("file should have no weird bits set by default")
+	}
+
+	f.SetIgnored()
+	if !f.IsIgnored() || f.MustRescan() || !f.IsInvalid() {
+		t.Error("file should be ignored and invalid")
+	}
+
+	f.SetMustRescan()
+	if f.IsIgnored() || !f.MustRescan() || !f.IsInvalid() {
+		t.Error("file should be must-rescan and invalid")
+	}
+
+	f.SetUnsupported()
+	if f.IsIgnored() || f.MustRescan() || !f.IsInvalid() {
+		t.Error("file should be invalid")
+	}
+}
+
+func TestIsEquivalent(t *testing.T) {
+	b := func(v bool) *bool {
+		return &v
+	}
+
+	type testCase struct {
+		a         FileInfo
+		b         FileInfo
+		ignPerms  *bool // nil means should not matter, we'll test both variants
+		ignBlocks *bool
+		ignFlags  uint32
+		eq        bool
+	}
+	cases := []testCase{
+		// Empty FileInfos are equivalent
+		{eq: true},
+
+		// Various basic attributes, all of which cause inequality when
+		// they differ
+		{
+			a:  FileInfo{Name: "foo"},
+			b:  FileInfo{Name: "bar"},
+			eq: false,
+		},
+		{
+			a:  FileInfo{Type: FileInfoTypeFile},
+			b:  FileInfo{Type: FileInfoTypeDirectory},
+			eq: false,
+		},
+		{
+			a:  FileInfo{Size: 1234},
+			b:  FileInfo{Size: 2345},
+			eq: false,
+		},
+		{
+			a:  FileInfo{Deleted: false},
+			b:  FileInfo{Deleted: true},
+			eq: false,
+		},
+		{
+			a:  FileInfo{RawInvalid: false},
+			b:  FileInfo{RawInvalid: true},
+			eq: false,
+		},
+		{
+			a:  FileInfo{ModifiedS: 1234},
+			b:  FileInfo{ModifiedS: 2345},
+			eq: false,
+		},
+		{
+			a:  FileInfo{ModifiedNs: 1234},
+			b:  FileInfo{ModifiedNs: 2345},
+			eq: false,
+		},
+
+		// Special handling of local flags and invalidity. "MustRescan"
+		// files are never equivalent to each other. Otherwise, equivalence
+		// is based just on whether the file becomes IsInvalid() or not, not
+		// the specific reason or flag bits.
+		{
+			a:  FileInfo{LocalFlags: FlagLocalMustRescan},
+			b:  FileInfo{LocalFlags: FlagLocalMustRescan},
+			eq: false,
+		},
+		{
+			a:  FileInfo{RawInvalid: true},
+			b:  FileInfo{RawInvalid: true},
+			eq: true,
+		},
+		{
+			a:  FileInfo{LocalFlags: FlagLocalUnsupported},
+			b:  FileInfo{LocalFlags: FlagLocalUnsupported},
+			eq: true,
+		},
+		{
+			a:  FileInfo{RawInvalid: true},
+			b:  FileInfo{LocalFlags: FlagLocalUnsupported},
+			eq: true,
+		},
+		{
+			a:  FileInfo{LocalFlags: 0},
+			b:  FileInfo{LocalFlags: FlagLocalReceiveOnly},
+			eq: false,
+		},
+		{
+			a:        FileInfo{LocalFlags: 0},
+			b:        FileInfo{LocalFlags: FlagLocalReceiveOnly},
+			ignFlags: FlagLocalReceiveOnly,
+			eq:       true,
+		},
+
+		// Difference in blocks is not OK
+		{
+			a:         FileInfo{Blocks: []BlockInfo{{Hash: []byte{1, 2, 3, 4}}}},
+			b:         FileInfo{Blocks: []BlockInfo{{Hash: []byte{2, 3, 4, 5}}}},
+			ignBlocks: b(false),
+			eq:        false,
+		},
+
+		// ... unless we say it is
+		{
+			a:         FileInfo{Blocks: []BlockInfo{{Hash: []byte{1, 2, 3, 4}}}},
+			b:         FileInfo{Blocks: []BlockInfo{{Hash: []byte{2, 3, 4, 5}}}},
+			ignBlocks: b(true),
+			eq:        true,
+		},
+
+		// Difference in permissions is not OK.
+		{
+			a:        FileInfo{Permissions: 0o444},
+			b:        FileInfo{Permissions: 0o666},
+			ignPerms: b(false),
+			eq:       false,
+		},
+
+		// ... unless we say it is
+		{
+			a:        FileInfo{Permissions: 0o666},
+			b:        FileInfo{Permissions: 0o444},
+			ignPerms: b(true),
+			eq:       true,
+		},
+
+		// These attributes are not checked at all
+		{
+			a:  FileInfo{NoPermissions: false},
+			b:  FileInfo{NoPermissions: true},
+			eq: true,
+		},
+		{
+			a:  FileInfo{Version: Vector{Counters: []Counter{{ID: 1, Value: 42}}}},
+			b:  FileInfo{Version: Vector{Counters: []Counter{{ID: 42, Value: 1}}}},
+			eq: true,
+		},
+		{
+			a:  FileInfo{Sequence: 1},
+			b:  FileInfo{Sequence: 2},
+			eq: true,
+		},
+
+		// The block size is not checked (but this would fail the blocks
+		// check in real world)
+		{
+			a:  FileInfo{RawBlockSize: 1},
+			b:  FileInfo{RawBlockSize: 2},
+			eq: true,
+		},
+
+		// The symlink target is checked for symlinks
+		{
+			a:  FileInfo{Type: FileInfoTypeSymlink, SymlinkTarget: "a"},
+			b:  FileInfo{Type: FileInfoTypeSymlink, SymlinkTarget: "b"},
+			eq: false,
+		},
+
+		// ... but not for non-symlinks
+		{
+			a:  FileInfo{Type: FileInfoTypeFile, SymlinkTarget: "a"},
+			b:  FileInfo{Type: FileInfoTypeFile, SymlinkTarget: "b"},
+			eq: true,
+		},
+	}
+
+	if build.IsWindows {
+		// On windows we only check the user writable bit of the permission
+		// set, so these are equivalent.
+		cases = append(cases, testCase{
+			a:        FileInfo{Permissions: 0o777},
+			b:        FileInfo{Permissions: 0o600},
+			ignPerms: b(false),
+			eq:       true,
+		})
+	}
+
+	for i, tc := range cases {
+		// Check the standard attributes with all permutations of the
+		// special ignore flags, unless the value of those flags are given
+		// in the tests.
+		for _, ignPerms := range []bool{true, false} {
+			for _, ignBlocks := range []bool{true, false} {
+				if tc.ignPerms != nil && *tc.ignPerms != ignPerms {
+					continue
+				}
+				if tc.ignBlocks != nil && *tc.ignBlocks != ignBlocks {
+					continue
+				}
+
+				if res := tc.a.isEquivalent(tc.b, FileInfoComparison{IgnorePerms: ignPerms, IgnoreBlocks: ignBlocks, IgnoreFlags: tc.ignFlags}); res != tc.eq {
+					t.Errorf("Case %d:\na: %v\nb: %v\na.IsEquivalent(b, %v, %v) => %v, expected %v", i, tc.a, tc.b, ignPerms, ignBlocks, res, tc.eq)
+				}
+				if res := tc.b.isEquivalent(tc.a, FileInfoComparison{IgnorePerms: ignPerms, IgnoreBlocks: ignBlocks, IgnoreFlags: tc.ignFlags}); res != tc.eq {
+					t.Errorf("Case %d:\na: %v\nb: %v\nb.IsEquivalent(a, %v, %v) => %v, expected %v", i, tc.a, tc.b, ignPerms, ignBlocks, res, tc.eq)
+				}
+			}
+		}
+	}
+}
+
+func TestSha256OfEmptyBlock(t *testing.T) {
+	// every block size should have a correct entry in sha256OfEmptyBlock
+	for blockSize := MinBlockSize; blockSize <= MaxBlockSize; blockSize *= 2 {
+		expected := sha256.Sum256(make([]byte, blockSize))
+		if sha256OfEmptyBlock[blockSize] != expected {
+			t.Error("missing or wrong hash for block of size", blockSize)
+		}
+	}
+}
+
+func TestBlocksEqual(t *testing.T) {
+	blocksOne := []BlockInfo{{Hash: []byte{1, 2, 3, 4}}}
+	blocksTwo := []BlockInfo{{Hash: []byte{5, 6, 7, 8}}}
+	hashOne := []byte{42, 42, 42, 42}
+	hashTwo := []byte{29, 29, 29, 29}
+
+	cases := []struct {
+		b1 []BlockInfo
+		h1 []byte
+		b2 []BlockInfo
+		h2 []byte
+		eq bool
+	}{
+		{blocksOne, hashOne, blocksOne, hashOne, true},  // everything equal
+		{blocksOne, hashOne, blocksTwo, hashTwo, false}, // nothing equal
+		{blocksOne, hashOne, blocksOne, nil, true},      // blocks compared
+		{blocksOne, nil, blocksOne, nil, true},          // blocks compared
+		{blocksOne, nil, blocksTwo, nil, false},         // blocks compared
+		{blocksOne, hashOne, blocksTwo, hashOne, true},  // hashes equal, blocks not looked at
+		{blocksOne, hashOne, blocksOne, hashTwo, true},  // hashes different, blocks compared
+		{blocksOne, hashOne, blocksTwo, hashTwo, false}, // hashes different, blocks compared
+		{blocksOne, hashOne, nil, nil, false},           // blocks is different from no blocks
+		{blocksOne, nil, nil, nil, false},               // blocks is different from no blocks
+		{nil, hashOne, nil, nil, true},                  // nil blocks are equal, even of one side has a hash
+	}
+
+	for _, tc := range cases {
+		f1 := FileInfo{Blocks: tc.b1, BlocksHash: tc.h1}
+		f2 := FileInfo{Blocks: tc.b2, BlocksHash: tc.h2}
+
+		if !f1.BlocksEqual(f1) {
+			t.Error("f1 is always equal to itself", f1)
+		}
+		if !f2.BlocksEqual(f2) {
+			t.Error("f2 is always equal to itself", f2)
+		}
+		if res := f1.BlocksEqual(f2); res != tc.eq {
+			t.Log("f1", f1.BlocksHash, f1.Blocks)
+			t.Log("f2", f2.BlocksHash, f2.Blocks)
+			t.Errorf("f1.BlocksEqual(f2) == %v but should be %v", res, tc.eq)
+		}
+		if res := f2.BlocksEqual(f1); res != tc.eq {
+			t.Log("f1", f1.BlocksHash, f1.Blocks)
+			t.Log("f2", f2.BlocksHash, f2.Blocks)
+			t.Errorf("f2.BlocksEqual(f1) == %v but should be %v", res, tc.eq)
+		}
+	}
+}
@@ -1,4 +1,8 @@
-// Copyright (C) 2016 The Protocol Authors.
+// Copyright (C) 2016 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -6,6 +10,15 @@ import (
 	"encoding/binary"
 	"errors"
 	"io"
+
+	"google.golang.org/protobuf/proto"
+
+	"github.com/syncthing/syncthing/internal/gen/bep"
+)
+
+const (
+	HelloMessageMagic   uint32 = 0x2EA7D90B
+	Version13HelloMagic uint32 = 0x9F79BC40 // old
 )

 var (
@@ -17,6 +30,38 @@ var (
 	ErrUnknownMagic = errors.New("the remote device speaks an unknown (newer?) version of the protocol")
 )

+type Hello struct {
+	DeviceName     string
+	ClientName     string
+	ClientVersion  string
+	NumConnections int
+	Timestamp      int64
+}
+
+func (h *Hello) toWire() *bep.Hello {
+	return &bep.Hello{
+		DeviceName:     h.DeviceName,
+		ClientName:     h.ClientName,
+		ClientVersion:  h.ClientVersion,
+		NumConnections: int32(h.NumConnections),
+		Timestamp:      h.Timestamp,
+	}
+}
+
+func helloFromWire(w *bep.Hello) Hello {
+	return Hello{
+		DeviceName:     w.DeviceName,
+		ClientName:     w.ClientName,
+		ClientVersion:  w.ClientVersion,
+		NumConnections: int(w.NumConnections),
+		Timestamp:      w.Timestamp,
+	}
+}
+
+func (Hello) Magic() uint32 {
+	return HelloMessageMagic
+}
+
 func ExchangeHello(c io.ReadWriter, h Hello) (Hello, error) {
 	if h.Timestamp == 0 {
 		panic("bug: missing timestamp in outgoing hello")
@@ -30,12 +75,7 @@ func ExchangeHello(c io.ReadWriter, h Hello) (Hello, error) {
 // IsVersionMismatch returns true if the error is a reliable indication of a
 // version mismatch that we might want to alert the user about.
 func IsVersionMismatch(err error) bool {
-	switch err {
-	case ErrTooOldVersion, ErrUnknownMagic:
-		return true
-	default:
-		return false
-	}
+	return errors.Is(err, ErrTooOldVersion) || errors.Is(err, ErrUnknownMagic)
 }

 func readHello(c io.Reader) (Hello, error) {
@@ -59,11 +99,12 @@ func readHello(c io.Reader) (Hello, error) {
 			return Hello{}, err
 		}

-		var hello Hello
-		if err := hello.Unmarshal(buf); err != nil {
+		var wh bep.Hello
+		if err := proto.Unmarshal(buf, &wh); err != nil {
 			return Hello{}, err
 		}
-		return Hello(hello), nil
+
+		return helloFromWire(&wh), nil

 	case 0x00010001, 0x00010000, Version13HelloMagic:
 		// This is the first word of an older cluster config message or an
@@ -76,7 +117,7 @@ func readHello(c io.Reader) (Hello, error) {
 }

 func writeHello(c io.Writer, h Hello) error {
-	msg, err := h.Marshal()
+	msg, err := proto.Marshal(h.toWire())
 	if err != nil {
 		return err
 	}
@@ -1,4 +1,8 @@
-// Copyright (C) 2016 The Protocol Authors.
+// Copyright (C) 2016 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -8,6 +12,8 @@ import (
 	"encoding/hex"
 	"io"
 	"testing"
+
+	"google.golang.org/protobuf/proto"
 )

 func TestVersion14Hello(t *testing.T) {
@@ -18,7 +24,7 @@ func TestVersion14Hello(t *testing.T) {
 		ClientName:    "syncthing",
 		ClientVersion: "v0.14.5",
 	}
-	msgBuf, err := expected.Marshal()
+	msgBuf, err := proto.Marshal(expected.toWire())
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -0,0 +1,78 @@
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+package protocol
+
+import "github.com/syncthing/syncthing/internal/gen/bep"
+
+type Index struct {
+	Folder       string
+	Files        []FileInfo
+	LastSequence int64
+}
+
+func (i *Index) toWire() *bep.Index {
+	files := make([]*bep.FileInfo, len(i.Files))
+	for j, f := range i.Files {
+		files[j] = f.ToWire(false)
+	}
+	return &bep.Index{
+		Folder:       i.Folder,
+		Files:        files,
+		LastSequence: i.LastSequence,
+	}
+}
+
+func indexFromWire(w *bep.Index) *Index {
+	if w == nil {
+		return nil
+	}
+	i := &Index{
+		Folder:       w.Folder,
+		LastSequence: w.LastSequence,
+	}
+	i.Files = make([]FileInfo, len(w.Files))
+	for j, f := range w.Files {
+		i.Files[j] = FileInfoFromWire(f)
+	}
+	return i
+}
+
+type IndexUpdate struct {
+	Folder       string
+	Files        []FileInfo
+	LastSequence int64
+	PrevSequence int64
+}
+
+func (i *IndexUpdate) toWire() *bep.IndexUpdate {
+	files := make([]*bep.FileInfo, len(i.Files))
+	for j, f := range i.Files {
+		files[j] = f.ToWire(false)
+	}
+	return &bep.IndexUpdate{
+		Folder:       i.Folder,
+		Files:        files,
+		LastSequence: i.LastSequence,
+		PrevSequence: i.PrevSequence,
+	}
+}
+
+func indexUpdateFromWire(w *bep.IndexUpdate) *IndexUpdate {
+	if w == nil {
+		return nil
+	}
+	i := &IndexUpdate{
+		Folder:       w.Folder,
+		LastSequence: w.LastSequence,
+		PrevSequence: w.PrevSequence,
+	}
+	i.Files = make([]FileInfo, len(w.Files))
+	for j, f := range w.Files {
+		i.Files[j] = FileInfoFromWire(f)
+	}
+	return i
+}
@@ -0,0 +1,80 @@
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+package protocol
+
+import "github.com/syncthing/syncthing/internal/gen/bep"
+
+type ErrorCode = bep.ErrorCode
+
+const (
+	ErrorCodeNoError     = bep.ErrorCode_ERROR_CODE_NO_ERROR
+	ErrorCodeGeneric     = bep.ErrorCode_ERROR_CODE_GENERIC
+	ErrorCodeNoSuchFile  = bep.ErrorCode_ERROR_CODE_NO_SUCH_FILE
+	ErrorCodeInvalidFile = bep.ErrorCode_ERROR_CODE_INVALID_FILE
+)
+
+type Request struct {
+	ID            int
+	Folder        string
+	Name          string
+	Offset        int64
+	Size          int
+	Hash          []byte
+	FromTemporary bool
+	WeakHash      uint32
+	BlockNo       int
+}
+
+func (r *Request) toWire() *bep.Request {
+	return &bep.Request{
+		Id:            int32(r.ID),
+		Folder:        r.Folder,
+		Name:          r.Name,
+		Offset:        r.Offset,
+		Size:          int32(r.Size),
+		Hash:          r.Hash,
+		FromTemporary: r.FromTemporary,
+		WeakHash:      r.WeakHash,
+		BlockNo:       int32(r.BlockNo),
+	}
+}
+
+func requestFromWire(w *bep.Request) *Request {
+	return &Request{
+		ID:            int(w.Id),
+		Folder:        w.Folder,
+		Name:          w.Name,
+		Offset:        w.Offset,
+		Size:          int(w.Size),
+		Hash:          w.Hash,
+		FromTemporary: w.FromTemporary,
+		WeakHash:      w.WeakHash,
+		BlockNo:       int(w.BlockNo),
+	}
+}
+
+type Response struct {
+	ID   int
+	Data []byte
+	Code ErrorCode
+}
+
+func (r *Response) toWire() *bep.Response {
+	return &bep.Response{
+		Id:   int32(r.ID),
+		Data: r.Data,
+		Code: r.Code,
+	}
+}
+
+func responseFromWire(w *bep.Response) *Response {
+	return &Response{
+		ID:   int(w.Id),
+		Data: w.Data,
+		Code: w.Code,
+	}
+}
@@ -1,4 +1,8 @@
-// Copyright (C) 2016 The Protocol Authors.
+// Copyright (C) 2016 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -8,9 +12,8 @@ import (
 	"sync/atomic"
 )

-// Global pool to get buffers from. Requires Blocksizes to be initialised,
-// therefore it is initialized in the same init() as BlockSizes
-var BufferPool bufferPool
+// Global pool to get buffers from. Initialized in init().
+var BufferPool *bufferPool

 type bufferPool struct {
 	puts   atomic.Int64
@@ -20,8 +23,8 @@ type bufferPool struct {
 	hits   []atomic.Int64
 }

-func newBufferPool() bufferPool {
-	return bufferPool{
+func newBufferPool() *bufferPool {
+	return &bufferPool{
 		pools: make([]sync.Pool, len(BlockSizes)),
 		hits:  make([]atomic.Int64, len(BlockSizes)),
 	}
@@ -1,4 +1,8 @@
-// Copyright (C) 2019 The Protocol Authors.
+// Copyright (C) 2019 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -1,8 +1,14 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

-import "time"
+import (
+	"time"
+)

 type TestModel struct {
 	data          []byte
@@ -1,39 +0,0 @@
-// Copyright (C) 2015 The Protocol Authors.
-
-package protocol
-
-import "fmt"
-
-const (
-	compressionThreshold = 128 // don't bother compressing messages smaller than this many bytes
-)
-
-var compressionMarshal = map[Compression]string{
-	CompressionNever:    "never",
-	CompressionMetadata: "metadata",
-	CompressionAlways:   "always",
-}
-
-var compressionUnmarshal = map[string]Compression{
-	// Legacy
-	"false": CompressionNever,
-	"true":  CompressionMetadata,
-
-	// Current
-	"never":    CompressionNever,
-	"metadata": CompressionMetadata,
-	"always":   CompressionAlways,
-}
-
-func (c Compression) GoString() string {
-	return fmt.Sprintf("%q", c.String())
-}
-
-func (c Compression) MarshalText() ([]byte, error) {
-	return []byte(compressionMarshal[c]), nil
-}
-
-func (c *Compression) UnmarshalText(bs []byte) error {
-	*c = compressionUnmarshal[string(bs)]
-	return nil
-}
@@ -1,49 +0,0 @@
-// Copyright (C) 2015 The Protocol Authors.
-
-package protocol
-
-import "testing"
-
-func TestCompressionMarshal(t *testing.T) {
-	uTestcases := []struct {
-		s string
-		c Compression
-	}{
-		{"true", CompressionMetadata},
-		{"false", CompressionNever},
-		{"never", CompressionNever},
-		{"metadata", CompressionMetadata},
-		{"always", CompressionAlways},
-		{"whatever", CompressionMetadata},
-	}
-
-	mTestcases := []struct {
-		s string
-		c Compression
-	}{
-		{"never", CompressionNever},
-		{"metadata", CompressionMetadata},
-		{"always", CompressionAlways},
-	}
-
-	var c Compression
-	for _, tc := range uTestcases {
-		err := c.UnmarshalText([]byte(tc.s))
-		if err != nil {
-			t.Error(err)
-		}
-		if c != tc.c {
-			t.Errorf("%s unmarshalled to %d, not %d", tc.s, c, tc.c)
-		}
-	}
-
-	for _, tc := range mTestcases {
-		bs, err := tc.c.MarshalText()
-		if err != nil {
-			t.Error(err)
-		}
-		if s := string(bs); s != tc.s {
-			t.Errorf("%d marshalled to %q, not %q", tc.c, s, tc.s)
-		}
-	}
-}
@@ -1,8 +1,14 @@
-// Copyright (C) 2015 The Protocol Authors.
+// Copyright (C) 2015 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

-import "testing"
+import (
+	"testing"
+)

 func TestWinsConflict(t *testing.T) {
 	testcases := [][2]FileInfo{
@@ -14,10 +20,10 @@ func TestWinsConflict(t *testing.T) {
 	}

 	for _, tc := range testcases {
-		if !WinsConflict(tc[0], tc[1]) {
+		if !tc[0].WinsConflict(tc[1]) {
 			t.Errorf("%v should win over %v", tc[0], tc[1])
 		}
-		if WinsConflict(tc[1], tc[0]) {
+		if tc[1].WinsConflict(tc[0]) {
 			t.Errorf("%v should not win over %v", tc[1], tc[0])
 		}
 	}
@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -36,17 +40,22 @@ func repeatedDeviceID(v byte) (d DeviceID) {
 	return
 }

-// NewDeviceID generates a new device ID from the raw bytes of a certificate
+// NewDeviceID generates a new device ID from SHA256 hash of the given piece
+// of data (usually raw certificate bytes).
 func NewDeviceID(rawCert []byte) DeviceID {
 	return DeviceID(sha256.Sum256(rawCert))
 }

+// DeviceIDFromString parses a device ID from a string. The string is expected
+// to be in the canonical format, with check digits.
 func DeviceIDFromString(s string) (DeviceID, error) {
 	var n DeviceID
 	err := n.UnmarshalText([]byte(s))
 	return n, err
 }

+// DeviceIDFromBytes converts a 32 byte slice to a DeviceID. A slice of the
+// wrong length results in an error.
 func DeviceIDFromBytes(bs []byte) (DeviceID, error) {
 	var n DeviceID
 	if len(bs) != len(n) {
@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -107,59 +111,6 @@ func TestDeviceIDFromBytes(t *testing.T) {
 	}
 }

-func TestNewDeviceIDMarshalling(t *testing.T) {
-	// The new DeviceID.Unmarshal / DeviceID.MarshalTo serialization should
-	// be message compatible with how we used to serialize DeviceIDs.
-
-	// Create a message with a device ID in old style bytes format
-
-	id0, _ := DeviceIDFromString(formatted)
-	msg0 := TestOldDeviceID{Test: id0[:]}
-
-	//  Marshal it
-
-	bs, err := msg0.Marshal()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// Unmarshal using the new DeviceID.Unmarshal
-
-	var msg1 TestNewDeviceID
-	if err := msg1.Unmarshal(bs); err != nil {
-		t.Fatal(err)
-	}
-
-	// Verify it's the same
-
-	if msg1.Test != id0 {
-		t.Error("Mismatch in old -> new direction")
-	}
-
-	// Marshal using the new DeviceID.MarshalTo
-
-	bs, err = msg1.Marshal()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// Create an old style message and attempt unmarshal
-
-	var msg2 TestOldDeviceID
-	if err := msg2.Unmarshal(bs); err != nil {
-		t.Fatal(err)
-	}
-
-	// Verify it's the same
-
-	id1, err := DeviceIDFromBytes(msg2.Test)
-	if err != nil {
-		t.Fatal(err)
-	} else if id1 != id0 {
-		t.Error("Mismatch in old -> new direction")
-	}
-}
-
 var resStr string

 func BenchmarkLuhnify(b *testing.B) {
@@ -1,481 +0,0 @@
-// Code generated by protoc-gen-gogo. DO NOT EDIT.
-// source: lib/protocol/deviceid_test.proto
-
-package protocol
-
-import (
-	fmt "fmt"
-	_ "github.com/gogo/protobuf/gogoproto"
-	proto "github.com/gogo/protobuf/proto"
-	_ "github.com/syncthing/syncthing/proto/ext"
-	io "io"
-	math "math"
-	math_bits "math/bits"
-)
-
-// Reference imports to suppress errors if they are not otherwise used.
-var _ = proto.Marshal
-var _ = fmt.Errorf
-var _ = math.Inf
-
-// This is a compile-time assertion to ensure that this generated file
-// is compatible with the proto package it is being compiled against.
-// A compilation error at this line likely means your copy of the
-// proto package needs to be updated.
-const _ = proto.GoGoProtoPackageIsVersion3 // please upgrade the proto package
-
-type TestOldDeviceID struct {
-	Test []byte `protobuf:"bytes,1,opt,name=test,proto3" json:"test" xml:"test"`
-}
-
-func (m *TestOldDeviceID) Reset()         { *m = TestOldDeviceID{} }
-func (m *TestOldDeviceID) String() string { return proto.CompactTextString(m) }
-func (*TestOldDeviceID) ProtoMessage()    {}
-func (*TestOldDeviceID) Descriptor() ([]byte, []int) {
-	return fileDescriptor_f4a75253a19e48a2, []int{0}
-}
-func (m *TestOldDeviceID) XXX_Unmarshal(b []byte) error {
-	return m.Unmarshal(b)
-}
-func (m *TestOldDeviceID) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
-	if deterministic {
-		return xxx_messageInfo_TestOldDeviceID.Marshal(b, m, deterministic)
-	} else {
-		b = b[:cap(b)]
-		n, err := m.MarshalToSizedBuffer(b)
-		if err != nil {
-			return nil, err
-		}
-		return b[:n], nil
-	}
-}
-func (m *TestOldDeviceID) XXX_Merge(src proto.Message) {
-	xxx_messageInfo_TestOldDeviceID.Merge(m, src)
-}
-func (m *TestOldDeviceID) XXX_Size() int {
-	return m.ProtoSize()
-}
-func (m *TestOldDeviceID) XXX_DiscardUnknown() {
-	xxx_messageInfo_TestOldDeviceID.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_TestOldDeviceID proto.InternalMessageInfo
-
-type TestNewDeviceID struct {
-	Test DeviceID `protobuf:"bytes,1,opt,name=test,proto3,customtype=DeviceID" json:"test" xml:"test"`
-}
-
-func (m *TestNewDeviceID) Reset()         { *m = TestNewDeviceID{} }
-func (m *TestNewDeviceID) String() string { return proto.CompactTextString(m) }
-func (*TestNewDeviceID) ProtoMessage()    {}
-func (*TestNewDeviceID) Descriptor() ([]byte, []int) {
-	return fileDescriptor_f4a75253a19e48a2, []int{1}
-}
-func (m *TestNewDeviceID) XXX_Unmarshal(b []byte) error {
-	return m.Unmarshal(b)
-}
-func (m *TestNewDeviceID) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) {
-	if deterministic {
-		return xxx_messageInfo_TestNewDeviceID.Marshal(b, m, deterministic)
-	} else {
-		b = b[:cap(b)]
-		n, err := m.MarshalToSizedBuffer(b)
-		if err != nil {
-			return nil, err
-		}
-		return b[:n], nil
-	}
-}
-func (m *TestNewDeviceID) XXX_Merge(src proto.Message) {
-	xxx_messageInfo_TestNewDeviceID.Merge(m, src)
-}
-func (m *TestNewDeviceID) XXX_Size() int {
-	return m.ProtoSize()
-}
-func (m *TestNewDeviceID) XXX_DiscardUnknown() {
-	xxx_messageInfo_TestNewDeviceID.DiscardUnknown(m)
-}
-
-var xxx_messageInfo_TestNewDeviceID proto.InternalMessageInfo
-
-func init() {
-	proto.RegisterType((*TestOldDeviceID)(nil), "protocol.TestOldDeviceID")
-	proto.RegisterType((*TestNewDeviceID)(nil), "protocol.TestNewDeviceID")
-}
-
-func init() { proto.RegisterFile("lib/protocol/deviceid_test.proto", fileDescriptor_f4a75253a19e48a2) }
-
-var fileDescriptor_f4a75253a19e48a2 = []byte{
-	// 237 bytes of a gzipped FileDescriptorProto
-	0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0x52, 0xc8, 0xc9, 0x4c, 0xd2,
-	0x2f, 0x28, 0xca, 0x2f, 0xc9, 0x4f, 0xce, 0xcf, 0xd1, 0x4f, 0x49, 0x2d, 0xcb, 0x4c, 0x4e, 0xcd,
-	0x4c, 0x89, 0x2f, 0x49, 0x2d, 0x2e, 0xd1, 0x03, 0x0b, 0x0b, 0x71, 0xc0, 0x64, 0xa5, 0x38, 0x53,
-	0x2b, 0xa0, 0x82, 0x52, 0xca, 0x45, 0xa9, 0x05, 0xf9, 0xc5, 0x10, 0x8d, 0x49, 0xa5, 0x69, 0xfa,
-	0xe9, 0xf9, 0xe9, 0xf9, 0x60, 0x0e, 0x98, 0x05, 0x51, 0xa4, 0x64, 0xcb, 0xc5, 0x1f, 0x92, 0x5a,
-	0x5c, 0xe2, 0x9f, 0x93, 0xe2, 0x02, 0x36, 0xd7, 0xd3, 0x45, 0x48, 0x8b, 0x8b, 0x05, 0x64, 0xb4,
-	0x04, 0xa3, 0x02, 0xa3, 0x06, 0x8f, 0x93, 0xd8, 0xab, 0x7b, 0xf2, 0x60, 0xfe, 0xa7, 0x7b, 0xf2,
-	0x5c, 0x15, 0xb9, 0x39, 0x56, 0x4a, 0x20, 0x8e, 0x52, 0x10, 0x58, 0x4c, 0x29, 0x10, 0xa2, 0xdd,
-	0x2f, 0xb5, 0x1c, 0xae, 0xdd, 0x0e, 0x45, 0xbb, 0xd6, 0x89, 0x7b, 0xf2, 0x0c, 0xb7, 0xee, 0xc9,
-	0x73, 0xc0, 0xe4, 0xb1, 0x1b, 0xd7, 0x71, 0x41, 0x85, 0x11, 0x62, 0xa4, 0x93, 0xef, 0x89, 0x87,
-	0x72, 0x0c, 0x17, 0x1e, 0xca, 0x31, 0x9c, 0x78, 0x24, 0xc7, 0x78, 0xe1, 0x91, 0x1c, 0xe3, 0x84,
-	0xc7, 0x72, 0x0c, 0x0b, 0x1e, 0xcb, 0x31, 0x5e, 0x78, 0x2c, 0xc7, 0x70, 0xe3, 0xb1, 0x1c, 0x43,
-	0x94, 0x76, 0x7a, 0x66, 0x49, 0x46, 0x69, 0x92, 0x5e, 0x72, 0x7e, 0xae, 0x7e, 0x71, 0x65, 0x5e,
-	0x72, 0x49, 0x46, 0x66, 0x5e, 0x3a, 0x12, 0x0b, 0x39, 0xb8, 0x92, 0xd8, 0xc0, 0x2c, 0x63, 0x40,
-	0x00, 0x00, 0x00, 0xff, 0xff, 0x9a, 0x0a, 0x77, 0x43, 0x45, 0x01, 0x00, 0x00,
-}
-
-func (m *TestOldDeviceID) Marshal() (dAtA []byte, err error) {
-	size := m.ProtoSize()
-	dAtA = make([]byte, size)
-	n, err := m.MarshalToSizedBuffer(dAtA[:size])
-	if err != nil {
-		return nil, err
-	}
-	return dAtA[:n], nil
-}
-
-func (m *TestOldDeviceID) MarshalTo(dAtA []byte) (int, error) {
-	size := m.ProtoSize()
-	return m.MarshalToSizedBuffer(dAtA[:size])
-}
-
-func (m *TestOldDeviceID) MarshalToSizedBuffer(dAtA []byte) (int, error) {
-	i := len(dAtA)
-	_ = i
-	var l int
-	_ = l
-	if len(m.Test) > 0 {
-		i -= len(m.Test)
-		copy(dAtA[i:], m.Test)
-		i = encodeVarintDeviceidTest(dAtA, i, uint64(len(m.Test)))
-		i--
-		dAtA[i] = 0xa
-	}
-	return len(dAtA) - i, nil
-}
-
-func (m *TestNewDeviceID) Marshal() (dAtA []byte, err error) {
-	size := m.ProtoSize()
-	dAtA = make([]byte, size)
-	n, err := m.MarshalToSizedBuffer(dAtA[:size])
-	if err != nil {
-		return nil, err
-	}
-	return dAtA[:n], nil
-}
-
-func (m *TestNewDeviceID) MarshalTo(dAtA []byte) (int, error) {
-	size := m.ProtoSize()
-	return m.MarshalToSizedBuffer(dAtA[:size])
-}
-
-func (m *TestNewDeviceID) MarshalToSizedBuffer(dAtA []byte) (int, error) {
-	i := len(dAtA)
-	_ = i
-	var l int
-	_ = l
-	{
-		size := m.Test.ProtoSize()
-		i -= size
-		if _, err := m.Test.MarshalTo(dAtA[i:]); err != nil {
-			return 0, err
-		}
-		i = encodeVarintDeviceidTest(dAtA, i, uint64(size))
-	}
-	i--
-	dAtA[i] = 0xa
-	return len(dAtA) - i, nil
-}
-
-func encodeVarintDeviceidTest(dAtA []byte, offset int, v uint64) int {
-	offset -= sovDeviceidTest(v)
-	base := offset
-	for v >= 1<<7 {
-		dAtA[offset] = uint8(v&0x7f | 0x80)
-		v >>= 7
-		offset++
-	}
-	dAtA[offset] = uint8(v)
-	return base
-}
-func (m *TestOldDeviceID) ProtoSize() (n int) {
-	if m == nil {
-		return 0
-	}
-	var l int
-	_ = l
-	l = len(m.Test)
-	if l > 0 {
-		n += 1 + l + sovDeviceidTest(uint64(l))
-	}
-	return n
-}
-
-func (m *TestNewDeviceID) ProtoSize() (n int) {
-	if m == nil {
-		return 0
-	}
-	var l int
-	_ = l
-	l = m.Test.ProtoSize()
-	n += 1 + l + sovDeviceidTest(uint64(l))
-	return n
-}
-
-func sovDeviceidTest(x uint64) (n int) {
-	return (math_bits.Len64(x|1) + 6) / 7
-}
-func sozDeviceidTest(x uint64) (n int) {
-	return sovDeviceidTest(uint64((x << 1) ^ uint64((int64(x) >> 63))))
-}
-func (m *TestOldDeviceID) Unmarshal(dAtA []byte) error {
-	l := len(dAtA)
-	iNdEx := 0
-	for iNdEx < l {
-		preIndex := iNdEx
-		var wire uint64
-		for shift := uint(0); ; shift += 7 {
-			if shift >= 64 {
-				return ErrIntOverflowDeviceidTest
-			}
-			if iNdEx >= l {
-				return io.ErrUnexpectedEOF
-			}
-			b := dAtA[iNdEx]
-			iNdEx++
-			wire |= uint64(b&0x7F) << shift
-			if b < 0x80 {
-				break
-			}
-		}
-		fieldNum := int32(wire >> 3)
-		wireType := int(wire & 0x7)
-		if wireType == 4 {
-			return fmt.Errorf("proto: TestOldDeviceID: wiretype end group for non-group")
-		}
-		if fieldNum <= 0 {
-			return fmt.Errorf("proto: TestOldDeviceID: illegal tag %d (wire type %d)", fieldNum, wire)
-		}
-		switch fieldNum {
-		case 1:
-			if wireType != 2 {
-				return fmt.Errorf("proto: wrong wireType = %d for field Test", wireType)
-			}
-			var byteLen int
-			for shift := uint(0); ; shift += 7 {
-				if shift >= 64 {
-					return ErrIntOverflowDeviceidTest
-				}
-				if iNdEx >= l {
-					return io.ErrUnexpectedEOF
-				}
-				b := dAtA[iNdEx]
-				iNdEx++
-				byteLen |= int(b&0x7F) << shift
-				if b < 0x80 {
-					break
-				}
-			}
-			if byteLen < 0 {
-				return ErrInvalidLengthDeviceidTest
-			}
-			postIndex := iNdEx + byteLen
-			if postIndex < 0 {
-				return ErrInvalidLengthDeviceidTest
-			}
-			if postIndex > l {
-				return io.ErrUnexpectedEOF
-			}
-			m.Test = append(m.Test[:0], dAtA[iNdEx:postIndex]...)
-			if m.Test == nil {
-				m.Test = []byte{}
-			}
-			iNdEx = postIndex
-		default:
-			iNdEx = preIndex
-			skippy, err := skipDeviceidTest(dAtA[iNdEx:])
-			if err != nil {
-				return err
-			}
-			if (skippy < 0) || (iNdEx+skippy) < 0 {
-				return ErrInvalidLengthDeviceidTest
-			}
-			if (iNdEx + skippy) > l {
-				return io.ErrUnexpectedEOF
-			}
-			iNdEx += skippy
-		}
-	}
-
-	if iNdEx > l {
-		return io.ErrUnexpectedEOF
-	}
-	return nil
-}
-func (m *TestNewDeviceID) Unmarshal(dAtA []byte) error {
-	l := len(dAtA)
-	iNdEx := 0
-	for iNdEx < l {
-		preIndex := iNdEx
-		var wire uint64
-		for shift := uint(0); ; shift += 7 {
-			if shift >= 64 {
-				return ErrIntOverflowDeviceidTest
-			}
-			if iNdEx >= l {
-				return io.ErrUnexpectedEOF
-			}
-			b := dAtA[iNdEx]
-			iNdEx++
-			wire |= uint64(b&0x7F) << shift
-			if b < 0x80 {
-				break
-			}
-		}
-		fieldNum := int32(wire >> 3)
-		wireType := int(wire & 0x7)
-		if wireType == 4 {
-			return fmt.Errorf("proto: TestNewDeviceID: wiretype end group for non-group")
-		}
-		if fieldNum <= 0 {
-			return fmt.Errorf("proto: TestNewDeviceID: illegal tag %d (wire type %d)", fieldNum, wire)
-		}
-		switch fieldNum {
-		case 1:
-			if wireType != 2 {
-				return fmt.Errorf("proto: wrong wireType = %d for field Test", wireType)
-			}
-			var byteLen int
-			for shift := uint(0); ; shift += 7 {
-				if shift >= 64 {
-					return ErrIntOverflowDeviceidTest
-				}
-				if iNdEx >= l {
-					return io.ErrUnexpectedEOF
-				}
-				b := dAtA[iNdEx]
-				iNdEx++
-				byteLen |= int(b&0x7F) << shift
-				if b < 0x80 {
-					break
-				}
-			}
-			if byteLen < 0 {
-				return ErrInvalidLengthDeviceidTest
-			}
-			postIndex := iNdEx + byteLen
-			if postIndex < 0 {
-				return ErrInvalidLengthDeviceidTest
-			}
-			if postIndex > l {
-				return io.ErrUnexpectedEOF
-			}
-			if err := m.Test.Unmarshal(dAtA[iNdEx:postIndex]); err != nil {
-				return err
-			}
-			iNdEx = postIndex
-		default:
-			iNdEx = preIndex
-			skippy, err := skipDeviceidTest(dAtA[iNdEx:])
-			if err != nil {
-				return err
-			}
-			if (skippy < 0) || (iNdEx+skippy) < 0 {
-				return ErrInvalidLengthDeviceidTest
-			}
-			if (iNdEx + skippy) > l {
-				return io.ErrUnexpectedEOF
-			}
-			iNdEx += skippy
-		}
-	}
-
-	if iNdEx > l {
-		return io.ErrUnexpectedEOF
-	}
-	return nil
-}
-func skipDeviceidTest(dAtA []byte) (n int, err error) {
-	l := len(dAtA)
-	iNdEx := 0
-	depth := 0
-	for iNdEx < l {
-		var wire uint64
-		for shift := uint(0); ; shift += 7 {
-			if shift >= 64 {
-				return 0, ErrIntOverflowDeviceidTest
-			}
-			if iNdEx >= l {
-				return 0, io.ErrUnexpectedEOF
-			}
-			b := dAtA[iNdEx]
-			iNdEx++
-			wire |= (uint64(b) & 0x7F) << shift
-			if b < 0x80 {
-				break
-			}
-		}
-		wireType := int(wire & 0x7)
-		switch wireType {
-		case 0:
-			for shift := uint(0); ; shift += 7 {
-				if shift >= 64 {
-					return 0, ErrIntOverflowDeviceidTest
-				}
-				if iNdEx >= l {
-					return 0, io.ErrUnexpectedEOF
-				}
-				iNdEx++
-				if dAtA[iNdEx-1] < 0x80 {
-					break
-				}
-			}
-		case 1:
-			iNdEx += 8
-		case 2:
-			var length int
-			for shift := uint(0); ; shift += 7 {
-				if shift >= 64 {
-					return 0, ErrIntOverflowDeviceidTest
-				}
-				if iNdEx >= l {
-					return 0, io.ErrUnexpectedEOF
-				}
-				b := dAtA[iNdEx]
-				iNdEx++
-				length |= (int(b) & 0x7F) << shift
-				if b < 0x80 {
-					break
-				}
-			}
-			if length < 0 {
-				return 0, ErrInvalidLengthDeviceidTest
-			}
-			iNdEx += length
-		case 3:
-			depth++
-		case 4:
-			if depth == 0 {
-				return 0, ErrUnexpectedEndOfGroupDeviceidTest
-			}
-			depth--
-		case 5:
-			iNdEx += 4
-		default:
-			return 0, fmt.Errorf("proto: illegal wireType %d", wireType)
-		}
-		if iNdEx < 0 {
-			return 0, ErrInvalidLengthDeviceidTest
-		}
-		if depth == 0 {
-			return iNdEx, nil
-		}
-	}
-	return 0, io.ErrUnexpectedEOF
-}
-
-var (
-	ErrInvalidLengthDeviceidTest        = fmt.Errorf("proto: negative length found during unmarshaling")
-	ErrIntOverflowDeviceidTest          = fmt.Errorf("proto: integer overflow")
-	ErrUnexpectedEndOfGroupDeviceidTest = fmt.Errorf("proto: unexpected end of group")
-)
@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 // Package protocol implements the Block Exchange Protocol.
 package protocol
@@ -17,13 +17,15 @@ import (
 	"strings"
 	"sync"

-	"github.com/gogo/protobuf/proto"
 	lru "github.com/hashicorp/golang-lru/v2"
 	"github.com/miscreant/miscreant.go"
-	"github.com/syncthing/syncthing/lib/rand"
 	"golang.org/x/crypto/chacha20poly1305"
 	"golang.org/x/crypto/hkdf"
 	"golang.org/x/crypto/scrypt"
+	"google.golang.org/protobuf/proto"
+
+	"github.com/syncthing/syncthing/internal/gen/bep"
+	"github.com/syncthing/syncthing/lib/rand"
 )

 const (
@@ -290,7 +292,7 @@ func encryptFileInfo(keyGen *KeyGenerator, fi FileInfo, folderKey *[keySize]byte
 	// The entire FileInfo is encrypted with a random nonce, and concatenated
 	// with that nonce.

-	bs, err := proto.Marshal(&fi)
+	bs, err := proto.Marshal(fi.ToWire(false))
 	if err != nil {
 		panic("impossible serialization mishap: " + err.Error())
 	}
@@ -366,7 +368,7 @@ func encryptFileInfo(keyGen *KeyGenerator, fi FileInfo, folderKey *[keySize]byte
 	if typ == FileInfoTypeFile {
 		enc.Size = offset // new total file size
 		enc.Blocks = blocks
-		enc.RawBlockSize = fi.BlockSize() + blockOverhead
+		enc.RawBlockSize = int32(fi.BlockSize() + blockOverhead)
 	}

 	return enc
@@ -407,7 +409,7 @@ func DecryptFileInfo(keyGen *KeyGenerator, fi FileInfo, folderKey *[keySize]byte
 		return FileInfo{}, err
 	}

-	var decFI FileInfo
+	var decFI bep.FileInfo
 	if err := proto.Unmarshal(dec, &decFI); err != nil {
 		return FileInfo{}, err
 	}
@@ -415,7 +417,7 @@ func DecryptFileInfo(keyGen *KeyGenerator, fi FileInfo, folderKey *[keySize]byte
 	// Preserve sequence, which is legitimately controlled by the untrusted device
 	decFI.Sequence = fi.Sequence

-	return decFI, nil
+	return FileInfoFromWire(&decFI), nil
 }

 var base32Hex = base32.HexEncoding.WithPadding(base32.NoPadding)
@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -24,12 +28,12 @@ func codeToError(code ErrorCode) error {
 }

 func errorToCode(err error) ErrorCode {
-	switch err {
-	case nil:
+	switch {
+	case err == nil:
 		return ErrorCodeNoError
-	case ErrNoSuchFile:
+	case errors.Is(err, ErrNoSuchFile):
 		return ErrorCodeNoSuchFile
-	case ErrInvalid:
+	case errors.Is(err, ErrInvalid):
 		return ErrorCodeInvalidFile
 	default:
 		return ErrorCodeGeneric
@@ -0,0 +1,39 @@
+// Copyright (C) 2019 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.
+
+package protocol
+
+import (
+	"encoding/binary"
+	"errors"
+	"fmt"
+
+	"github.com/syncthing/syncthing/lib/rand"
+)
+
+type IndexID uint64
+
+func (i IndexID) String() string {
+	return fmt.Sprintf("0x%016X", uint64(i))
+}
+
+func (i IndexID) Marshal() ([]byte, error) {
+	bs := make([]byte, 8)
+	binary.BigEndian.PutUint64(bs, uint64(i))
+	return bs, nil
+}
+
+func (i *IndexID) Unmarshal(bs []byte) error {
+	if len(bs) != 8 {
+		return errors.New("incorrect IndexID length")
+	}
+	*i = IndexID(binary.BigEndian.Uint64(bs))
+	return nil
+}
+
+func NewIndexID() IndexID {
+	return IndexID(rand.Uint64())
+}
@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 //go:build darwin
 // +build darwin
@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 //go:build !windows && !darwin
 // +build !windows,!darwin
@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 //go:build windows
 // +build windows
@@ -1,4 +1,8 @@
-// Copyright (C) 2016 The Protocol Authors.
+// Copyright (C) 2016 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 //go:generate -command counterfeiter go run github.com/maxbrunsfeld/counterfeiter/v6

@@ -13,7 +17,6 @@ package protocol

 import (
 	"context"
-	"crypto/sha256"
 	"encoding/binary"
 	"errors"
 	"fmt"
@@ -25,6 +28,10 @@ import (
 	"time"

 	lz4 "github.com/pierrec/lz4/v4"
+	"google.golang.org/protobuf/proto"
+
+	"github.com/syncthing/syncthing/internal/gen/bep"
+	"github.com/syncthing/syncthing/internal/protoutil"
 )

 const (
@@ -46,70 +53,20 @@ const (

 	// DesiredPerFileBlocks is the number of blocks we aim for per file
 	DesiredPerFileBlocks = 2000
+
+	SyntheticDirectorySize = 128
+
+	// don't bother compressing messages smaller than this many bytes
+	compressionThreshold = 128
 )

-// BlockSizes is the list of valid block sizes, from min to max
-var BlockSizes []int
-
-// For each block size, the hash of a block of all zeroes
-var sha256OfEmptyBlock = map[int][sha256.Size]byte{
-	128 << KiB: {0xfa, 0x43, 0x23, 0x9b, 0xce, 0xe7, 0xb9, 0x7c, 0xa6, 0x2f, 0x0, 0x7c, 0xc6, 0x84, 0x87, 0x56, 0xa, 0x39, 0xe1, 0x9f, 0x74, 0xf3, 0xdd, 0xe7, 0x48, 0x6d, 0xb3, 0xf9, 0x8d, 0xf8, 0xe4, 0x71},
-	256 << KiB: {0x8a, 0x39, 0xd2, 0xab, 0xd3, 0x99, 0x9a, 0xb7, 0x3c, 0x34, 0xdb, 0x24, 0x76, 0x84, 0x9c, 0xdd, 0xf3, 0x3, 0xce, 0x38, 0x9b, 0x35, 0x82, 0x68, 0x50, 0xf9, 0xa7, 0x0, 0x58, 0x9b, 0x4a, 0x90},
-	512 << KiB: {0x7, 0x85, 0x4d, 0x2f, 0xef, 0x29, 0x7a, 0x6, 0xba, 0x81, 0x68, 0x5e, 0x66, 0xc, 0x33, 0x2d, 0xe3, 0x6d, 0x5d, 0x18, 0xd5, 0x46, 0x92, 0x7d, 0x30, 0xda, 0xad, 0x6d, 0x7f, 0xda, 0x15, 0x41},
-	1 << MiB:   {0x30, 0xe1, 0x49, 0x55, 0xeb, 0xf1, 0x35, 0x22, 0x66, 0xdc, 0x2f, 0xf8, 0x6, 0x7e, 0x68, 0x10, 0x46, 0x7, 0xe7, 0x50, 0xab, 0xb9, 0xd3, 0xb3, 0x65, 0x82, 0xb8, 0xaf, 0x90, 0x9f, 0xcb, 0x58},
-	2 << MiB:   {0x56, 0x47, 0xf0, 0x5e, 0xc1, 0x89, 0x58, 0x94, 0x7d, 0x32, 0x87, 0x4e, 0xeb, 0x78, 0x8f, 0xa3, 0x96, 0xa0, 0x5d, 0xb, 0xab, 0x7c, 0x1b, 0x71, 0xf1, 0x12, 0xce, 0xb7, 0xe9, 0xb3, 0x1e, 0xee},
-	4 << MiB:   {0xbb, 0x9f, 0x8d, 0xf6, 0x14, 0x74, 0xd2, 0x5e, 0x71, 0xfa, 0x0, 0x72, 0x23, 0x18, 0xcd, 0x38, 0x73, 0x96, 0xca, 0x17, 0x36, 0x60, 0x5e, 0x12, 0x48, 0x82, 0x1c, 0xc0, 0xde, 0x3d, 0x3a, 0xf8},
-	8 << MiB:   {0x2d, 0xae, 0xb1, 0xf3, 0x60, 0x95, 0xb4, 0x4b, 0x31, 0x84, 0x10, 0xb3, 0xf4, 0xe8, 0xb5, 0xd9, 0x89, 0xdc, 0xc7, 0xbb, 0x2, 0x3d, 0x14, 0x26, 0xc4, 0x92, 0xda, 0xb0, 0xa3, 0x5, 0x3e, 0x74},
-	16 << MiB:  {0x8, 0xa, 0xcf, 0x35, 0xa5, 0x7, 0xac, 0x98, 0x49, 0xcf, 0xcb, 0xa4, 0x7d, 0xc2, 0xad, 0x83, 0xe0, 0x1b, 0x75, 0x66, 0x3a, 0x51, 0x62, 0x79, 0xc8, 0xb9, 0xd2, 0x43, 0xb7, 0x19, 0x64, 0x3e},
-}
-
 var errNotCompressible = errors.New("not compressible")

-func init() {
-	for blockSize := MinBlockSize; blockSize <= MaxBlockSize; blockSize *= 2 {
-		BlockSizes = append(BlockSizes, blockSize)
-		if _, ok := sha256OfEmptyBlock[blockSize]; !ok {
-			panic("missing hard coded value for sha256 of empty block")
-		}
-	}
-	BufferPool = newBufferPool()
-}
-
-// BlockSize returns the block size to use for the given file size
-func BlockSize(fileSize int64) int {
-	var blockSize int
-	for _, blockSize = range BlockSizes {
-		if fileSize < DesiredPerFileBlocks*int64(blockSize) {
-			break
-		}
-	}
-
-	return blockSize
-}
-
 const (
 	stateInitial = iota
 	stateReady
 )

-// FileInfo.LocalFlags flags
-const (
-	FlagLocalUnsupported = 1 << 0 // The kind is unsupported, e.g. symlinks on Windows
-	FlagLocalIgnored     = 1 << 1 // Matches local ignore patterns
-	FlagLocalMustRescan  = 1 << 2 // Doesn't match content on disk, must be rechecked fully
-	FlagLocalReceiveOnly = 1 << 3 // Change detected on receive only folder
-
-	// Flags that should result in the Invalid bit on outgoing updates
-	LocalInvalidFlags = FlagLocalUnsupported | FlagLocalIgnored | FlagLocalMustRescan | FlagLocalReceiveOnly
-
-	// Flags that should result in a file being in conflict with its
-	// successor, due to us not having an up to date picture of its state on
-	// disk.
-	LocalConflictFlags = FlagLocalUnsupported | FlagLocalIgnored | FlagLocalReceiveOnly
-
-	LocalAllFlags = FlagLocalUnsupported | FlagLocalIgnored | FlagLocalMustRescan | FlagLocalReceiveOnly
-)
-
 var (
 	ErrClosed             = errors.New("connection closed")
 	ErrTimeout            = errors.New("read timeout")
@@ -220,7 +177,7 @@ type rawConnection struct {

 	idxMut sync.Mutex // ensures serialization of Index calls

-	inbox                 chan message
+	inbox                 chan proto.Message
 	outbox                chan asyncMessage
 	closeBox              chan asyncMessage
 	clusterConfigBox      chan *ClusterConfig
@@ -239,15 +196,8 @@ type asyncResult struct {
 	err error
 }

-type message interface {
-	ProtoSize() int
-	Marshal() ([]byte, error)
-	MarshalTo([]byte) (int, error)
-	Unmarshal([]byte) error
-}
-
 type asyncMessage struct {
-	msg  message
+	msg  proto.Message
 	done chan struct{} // done closes when we're done sending the message
 }

@@ -303,7 +253,7 @@ func newRawConnection(deviceID DeviceID, reader io.Reader, writer io.Writer, clo
 		cw:                    cw,
 		closer:                closer,
 		awaiting:              make(map[int]chan asyncResult),
-		inbox:                 make(chan message),
+		inbox:                 make(chan proto.Message),
 		outbox:                make(chan asyncMessage),
 		closeBox:              make(chan asyncMessage),
 		clusterConfigBox:      make(chan *ClusterConfig),
@@ -359,7 +309,7 @@ func (c *rawConnection) Index(ctx context.Context, idx *Index) error {
 	default:
 	}
 	c.idxMut.Lock()
-	c.send(ctx, idx, nil)
+	c.send(ctx, idx.toWire(), nil)
 	c.idxMut.Unlock()
 	return nil
 }
@@ -374,7 +324,7 @@ func (c *rawConnection) IndexUpdate(ctx context.Context, idxUp *IndexUpdate) err
 	default:
 	}
 	c.idxMut.Lock()
-	c.send(ctx, idxUp, nil)
+	c.send(ctx, idxUp.toWire(), nil)
 	c.idxMut.Unlock()
 	return nil
 }
@@ -402,7 +352,7 @@ func (c *rawConnection) Request(ctx context.Context, req *Request) ([]byte, erro
 	c.awaitingMut.Unlock()

 	req.ID = id
-	ok := c.send(ctx, req, nil)
+	ok := c.send(ctx, req.toWire(), nil)
 	if !ok {
 		return nil, ErrClosed
 	}
@@ -432,11 +382,11 @@ func (c *rawConnection) Closed() <-chan struct{} {

 // DownloadProgress sends the progress updates for the files that are currently being downloaded.
 func (c *rawConnection) DownloadProgress(ctx context.Context, dp *DownloadProgress) {
-	c.send(ctx, dp, nil)
+	c.send(ctx, dp.toWire(), nil)
 }

 func (c *rawConnection) ping() bool {
-	return c.send(context.Background(), &Ping{}, nil)
+	return c.send(context.Background(), &bep.Ping{}, nil)
 }

 func (c *rawConnection) readerLoop() {
@@ -456,13 +406,12 @@ func (c *rawConnection) readerLoop() {
 		case <-c.closed:
 			return
 		}
-
 	}
 }

 func (c *rawConnection) dispatcherLoop() (err error) {
 	defer close(c.dispatcherLoopStopped)
-	var msg message
+	var msg proto.Message
 	state := stateInitial
 	for {
 		select {
@@ -485,11 +434,11 @@ func (c *rawConnection) dispatcherLoop() (err error) {
 		l.Debugf("handle %v message", msgContext)

 		switch msg := msg.(type) {
-		case *ClusterConfig:
+		case *bep.ClusterConfig:
 			if state == stateInitial {
 				state = stateReady
 			}
-		case *Close:
+		case *bep.Close:
 			return fmt.Errorf("closed by remote: %v", msg.Reason)
 		default:
 			if state != stateReady {
@@ -498,13 +447,7 @@ func (c *rawConnection) dispatcherLoop() (err error) {
 		}

 		switch msg := msg.(type) {
-		case *Index:
-			err = checkIndexConsistency(msg.Files)
-
-		case *IndexUpdate:
-			err = checkIndexConsistency(msg.Files)
-
-		case *Request:
+		case *bep.Request:
 			err = checkFilename(msg.Name)
 		}
 		if err != nil {
@@ -512,23 +455,31 @@ func (c *rawConnection) dispatcherLoop() (err error) {
 		}

 		switch msg := msg.(type) {
-		case *ClusterConfig:
-			err = c.model.ClusterConfig(msg)
+		case *bep.ClusterConfig:
+			err = c.model.ClusterConfig(clusterConfigFromWire(msg))

-		case *Index:
-			err = c.handleIndex(msg)
+		case *bep.Index:
+			idx := indexFromWire(msg)
+			if err := checkIndexConsistency(idx.Files); err != nil {
+				return newProtocolError(err, msgContext)
+			}
+			err = c.handleIndex(idx)

-		case *IndexUpdate:
-			err = c.handleIndexUpdate(msg)
+		case *bep.IndexUpdate:
+			idxUp := indexUpdateFromWire(msg)
+			if err := checkIndexConsistency(idxUp.Files); err != nil {
+				return newProtocolError(err, msgContext)
+			}
+			err = c.handleIndexUpdate(idxUp)

-		case *Request:
-			go c.handleRequest(msg)
+		case *bep.Request:
+			go c.handleRequest(requestFromWire(msg))

-		case *Response:
-			c.handleResponse(msg)
+		case *bep.Response:
+			c.handleResponse(responseFromWire(msg))

-		case *DownloadProgress:
-			err = c.model.DownloadProgress(msg)
+		case *bep.DownloadProgress:
+			err = c.model.DownloadProgress(downloadProgressFromWire(msg))
 		}
 		if err != nil {
 			return newHandleError(err, msgContext)
@@ -536,7 +487,7 @@ func (c *rawConnection) dispatcherLoop() (err error) {
 	}
 }

-func (c *rawConnection) readMessage(fourByteBuf []byte) (message, error) {
+func (c *rawConnection) readMessage(fourByteBuf []byte) (proto.Message, error) {
 	hdr, err := c.readHeader(fourByteBuf)
 	if err != nil {
 		return nil, err
@@ -545,7 +496,7 @@ func (c *rawConnection) readMessage(fourByteBuf []byte) (message, error) {
 	return c.readMessageAfterHeader(hdr, fourByteBuf)
 }

-func (c *rawConnection) readMessageAfterHeader(hdr Header, fourByteBuf []byte) (message, error) {
+func (c *rawConnection) readMessageAfterHeader(hdr *bep.Header, fourByteBuf []byte) (proto.Message, error) {
 	// First comes a 4 byte message length

 	if _, err := io.ReadFull(c.cr, fourByteBuf[:4]); err != nil {
@@ -569,10 +520,10 @@ func (c *rawConnection) readMessageAfterHeader(hdr Header, fourByteBuf []byte) (
 	// ... which might be compressed

 	switch hdr.Compression {
-	case MessageCompressionNone:
+	case bep.MessageCompression_MESSAGE_COMPRESSION_NONE:
 		// Nothing

-	case MessageCompressionLZ4:
+	case bep.MessageCompression_MESSAGE_COMPRESSION_LZ4:
 		decomp, err := lz4Decompress(buf)
 		BufferPool.Put(buf)
 		if err != nil {
@@ -593,7 +544,7 @@ func (c *rawConnection) readMessageAfterHeader(hdr Header, fourByteBuf []byte) (
 		BufferPool.Put(buf)
 		return nil, err
 	}
-	if err := msg.Unmarshal(buf); err != nil {
+	if err := proto.Unmarshal(buf, msg); err != nil {
 		BufferPool.Put(buf)
 		return nil, fmt.Errorf("unmarshalling message: %w", err)
 	}
@@ -602,15 +553,15 @@ func (c *rawConnection) readMessageAfterHeader(hdr Header, fourByteBuf []byte) (
 	return msg, nil
 }

-func (c *rawConnection) readHeader(fourByteBuf []byte) (Header, error) {
+func (c *rawConnection) readHeader(fourByteBuf []byte) (*bep.Header, error) {
 	// First comes a 2 byte header length

 	if _, err := io.ReadFull(c.cr, fourByteBuf[:2]); err != nil {
-		return Header{}, fmt.Errorf("reading length: %w", err)
+		return nil, fmt.Errorf("reading length: %w", err)
 	}
 	hdrLen := int16(binary.BigEndian.Uint16(fourByteBuf))
 	if hdrLen < 0 {
-		return Header{}, fmt.Errorf("negative header length %d", hdrLen)
+		return nil, fmt.Errorf("negative header length %d", hdrLen)
 	}

 	// Then comes the header
@@ -618,19 +569,19 @@ func (c *rawConnection) readHeader(fourByteBuf []byte) (Header, error) {
 	buf := BufferPool.Get(int(hdrLen))
 	if _, err := io.ReadFull(c.cr, buf); err != nil {
 		BufferPool.Put(buf)
-		return Header{}, fmt.Errorf("reading header: %w", err)
+		return nil, fmt.Errorf("reading header: %w", err)
 	}

-	var hdr Header
-	err := hdr.Unmarshal(buf)
+	var hdr bep.Header
+	err := proto.Unmarshal(buf, &hdr)
 	BufferPool.Put(buf)
 	if err != nil {
-		return Header{}, fmt.Errorf("unmarshalling header: %w", err)
+		return nil, fmt.Errorf("unmarshalling header: %w %x", err, buf)
 	}

 	metricDeviceRecvDecompressedBytes.WithLabelValues(c.idString).Add(float64(2 + len(buf)))

-	return hdr, nil
+	return &hdr, nil
 }

 func (c *rawConnection) handleIndex(im *Index) error {
@@ -708,18 +659,20 @@ func checkFilename(name string) error {
 func (c *rawConnection) handleRequest(req *Request) {
 	res, err := c.model.Request(req)
 	if err != nil {
-		c.send(context.Background(), &Response{
+		resp := &Response{
 			ID:   req.ID,
 			Code: errorToCode(err),
-		}, nil)
+		}
+		c.send(context.Background(), resp.toWire(), nil)
 		return
 	}
 	done := make(chan struct{})
-	c.send(context.Background(), &Response{
+	resp := &Response{
 		ID:   req.ID,
 		Data: res.Data(),
 		Code: errorToCode(nil),
-	}, done)
+	}
+	c.send(context.Background(), resp.toWire(), done)
 	<-done
 	res.Close()
 }
@@ -734,7 +687,7 @@ func (c *rawConnection) handleResponse(resp *Response) {
 	c.awaitingMut.Unlock()
 }

-func (c *rawConnection) send(ctx context.Context, msg message, done chan struct{}) bool {
+func (c *rawConnection) send(ctx context.Context, msg proto.Message, done chan struct{}) bool {
 	select {
 	case c.outbox <- asyncMessage{msg, done}:
 		return true
@@ -750,7 +703,7 @@ func (c *rawConnection) send(ctx context.Context, msg message, done chan struct{
 func (c *rawConnection) writerLoop() {
 	select {
 	case cc := <-c.clusterConfigBox:
-		err := c.writeMessage(cc)
+		err := c.writeMessage(cc.toWire())
 		if err != nil {
 			c.internalClose(err)
 			return
@@ -776,7 +729,7 @@ func (c *rawConnection) writerLoop() {
 		}
 		select {
 		case cc := <-c.clusterConfigBox:
-			err := c.writeMessage(cc)
+			err := c.writeMessage(cc.toWire())
 			if err != nil {
 				c.internalClose(err)
 				return
@@ -802,7 +755,7 @@ func (c *rawConnection) writerLoop() {
 	}
 }

-func (c *rawConnection) writeMessage(msg message) error {
+func (c *rawConnection) writeMessage(msg proto.Message) error {
 	msgContext, _ := messageContext(msg)
 	l.Debugf("Writing %v", msgContext)

@@ -810,11 +763,11 @@ func (c *rawConnection) writeMessage(msg message) error {
 		metricDeviceSentMessages.WithLabelValues(c.idString).Inc()
 	}()

-	size := msg.ProtoSize()
-	hdr := Header{
+	size := proto.Size(msg)
+	hdr := &bep.Header{
 		Type: typeOf(msg),
 	}
-	hdrSize := hdr.ProtoSize()
+	hdrSize := proto.Size(hdr)
 	if hdrSize > 1<<16-1 {
 		panic("impossibly large header")
 	}
@@ -825,7 +778,7 @@ func (c *rawConnection) writeMessage(msg message) error {
 	defer BufferPool.Put(buf)

 	// Message
-	if _, err := msg.MarshalTo(buf[2+hdrSize+4:]); err != nil {
+	if _, err := protoutil.MarshalTo(buf[overhead:], msg); err != nil {
 		return fmt.Errorf("marshalling message: %w", err)
 	}

@@ -841,7 +794,7 @@ func (c *rawConnection) writeMessage(msg message) error {
 	// Header length
 	binary.BigEndian.PutUint16(buf, uint16(hdrSize))
 	// Header
-	if _, err := hdr.MarshalTo(buf[2:]); err != nil {
+	if _, err := protoutil.MarshalTo(buf[2:], hdr); err != nil {
 		return fmt.Errorf("marshalling header: %w", err)
 	}
 	// Message length
@@ -860,12 +813,12 @@ func (c *rawConnection) writeMessage(msg message) error {
 //
 // The first return value indicates whether compression succeeded.
 // If not, the caller should retry without compression.
-func (c *rawConnection) writeCompressedMessage(msg message, marshaled []byte) (ok bool, err error) {
-	hdr := Header{
+func (c *rawConnection) writeCompressedMessage(msg proto.Message, marshaled []byte) (ok bool, err error) {
+	hdr := &bep.Header{
 		Type:        typeOf(msg),
-		Compression: MessageCompressionLZ4,
+		Compression: bep.MessageCompression_MESSAGE_COMPRESSION_LZ4,
 	}
-	hdrSize := hdr.ProtoSize()
+	hdrSize := proto.Size(hdr)
 	if hdrSize > 1<<16-1 {
 		panic("impossibly large header")
 	}
@@ -890,7 +843,7 @@ func (c *rawConnection) writeCompressedMessage(msg message, marshaled []byte) (o
 	// Header length
 	binary.BigEndian.PutUint16(buf, uint16(hdrSize))
 	// Header
-	if _, err := hdr.MarshalTo(buf[2:]); err != nil {
+	if _, err := protoutil.MarshalTo(buf[2:], hdr); err != nil {
 		return true, fmt.Errorf("marshalling header: %w", err)
 	}
 	// Message length
@@ -904,65 +857,65 @@ func (c *rawConnection) writeCompressedMessage(msg message, marshaled []byte) (o
 	return true, nil
 }

-func typeOf(msg message) MessageType {
+func typeOf(msg proto.Message) bep.MessageType {
 	switch msg.(type) {
-	case *ClusterConfig:
-		return MessageTypeClusterConfig
-	case *Index:
-		return MessageTypeIndex
-	case *IndexUpdate:
-		return MessageTypeIndexUpdate
-	case *Request:
-		return MessageTypeRequest
-	case *Response:
-		return MessageTypeResponse
-	case *DownloadProgress:
-		return MessageTypeDownloadProgress
-	case *Ping:
-		return MessageTypePing
-	case *Close:
-		return MessageTypeClose
+	case *bep.ClusterConfig:
+		return bep.MessageType_MESSAGE_TYPE_CLUSTER_CONFIG
+	case *bep.Index:
+		return bep.MessageType_MESSAGE_TYPE_INDEX
+	case *bep.IndexUpdate:
+		return bep.MessageType_MESSAGE_TYPE_INDEX_UPDATE
+	case *bep.Request:
+		return bep.MessageType_MESSAGE_TYPE_REQUEST
+	case *bep.Response:
+		return bep.MessageType_MESSAGE_TYPE_RESPONSE
+	case *bep.DownloadProgress:
+		return bep.MessageType_MESSAGE_TYPE_DOWNLOAD_PROGRESS
+	case *bep.Ping:
+		return bep.MessageType_MESSAGE_TYPE_PING
+	case *bep.Close:
+		return bep.MessageType_MESSAGE_TYPE_CLOSE
 	default:
 		panic("bug: unknown message type")
 	}
 }

-func newMessage(t MessageType) (message, error) {
+func newMessage(t bep.MessageType) (proto.Message, error) {
 	switch t {
-	case MessageTypeClusterConfig:
-		return new(ClusterConfig), nil
-	case MessageTypeIndex:
-		return new(Index), nil
-	case MessageTypeIndexUpdate:
-		return new(IndexUpdate), nil
-	case MessageTypeRequest:
-		return new(Request), nil
-	case MessageTypeResponse:
-		return new(Response), nil
-	case MessageTypeDownloadProgress:
-		return new(DownloadProgress), nil
-	case MessageTypePing:
-		return new(Ping), nil
-	case MessageTypeClose:
-		return new(Close), nil
+	case bep.MessageType_MESSAGE_TYPE_CLUSTER_CONFIG:
+		return new(bep.ClusterConfig), nil
+	case bep.MessageType_MESSAGE_TYPE_INDEX:
+		return new(bep.Index), nil
+	case bep.MessageType_MESSAGE_TYPE_INDEX_UPDATE:
+		return new(bep.IndexUpdate), nil
+	case bep.MessageType_MESSAGE_TYPE_REQUEST:
+		return new(bep.Request), nil
+	case bep.MessageType_MESSAGE_TYPE_RESPONSE:
+		return new(bep.Response), nil
+	case bep.MessageType_MESSAGE_TYPE_DOWNLOAD_PROGRESS:
+		return new(bep.DownloadProgress), nil
+	case bep.MessageType_MESSAGE_TYPE_PING:
+		return new(bep.Ping), nil
+	case bep.MessageType_MESSAGE_TYPE_CLOSE:
+		return new(bep.Close), nil
 	default:
 		return nil, errUnknownMessage
 	}
 }

-func (c *rawConnection) shouldCompressMessage(msg message) bool {
+func (c *rawConnection) shouldCompressMessage(msg proto.Message) bool {
 	switch c.compression {
 	case CompressionNever:
 		return false

 	case CompressionAlways:
 		// Use compression for large enough messages
-		return msg.ProtoSize() >= compressionThreshold
+		return proto.Size(msg) >= compressionThreshold

 	case CompressionMetadata:
-		_, isResponse := msg.(*Response)
+		_, isResponse := msg.(*bep.Response)
 		// Compress if it's large enough and not a response message
-		return !isResponse && msg.ProtoSize() >= compressionThreshold
+		return !isResponse && proto.Size(msg) >= compressionThreshold

 	default:
 		panic("unknown compression setting")
@@ -977,7 +930,7 @@ func (c *rawConnection) Close(err error) {
 		done := make(chan struct{})
 		timeout := time.NewTimer(CloseTimeout)
 		select {
-		case c.closeBox <- asyncMessage{&Close{err.Error()}, done}:
+		case c.closeBox <- asyncMessage{&bep.Close{Reason: err.Error()}, done}:
 			select {
 			case <-done:
 			case <-timeout.C:
@@ -1127,23 +1080,23 @@ func newHandleError(err error, msgContext string) error {
 	return fmt.Errorf("handling %v: %w", msgContext, err)
 }

-func messageContext(msg message) (string, error) {
+func messageContext(msg proto.Message) (string, error) {
 	switch msg := msg.(type) {
-	case *ClusterConfig:
+	case *bep.ClusterConfig:
 		return "cluster-config", nil
-	case *Index:
+	case *bep.Index:
 		return fmt.Sprintf("index for %v", msg.Folder), nil
-	case *IndexUpdate:
+	case *bep.IndexUpdate:
 		return fmt.Sprintf("index-update for %v", msg.Folder), nil
-	case *Request:
+	case *bep.Request:
 		return fmt.Sprintf(`request for "%v" in %v`, msg.Name, msg.Folder), nil
-	case *Response:
+	case *bep.Response:
 		return "response", nil
-	case *DownloadProgress:
+	case *bep.DownloadProgress:
 		return fmt.Sprintf("download-progress for %v", msg.Folder), nil
-	case *Ping:
+	case *bep.Ping:
 		return "ping", nil
-	case *Close:
+	case *bep.Close:
 		return "close", nil
 	default:
 		return "", errors.New("unknown or empty message")
@@ -1,11 +1,14 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

 import (
 	"bytes"
 	"context"
-	"crypto/sha256"
 	"encoding/hex"
 	"encoding/json"
 	"errors"
@@ -13,19 +16,19 @@ import (
 	"os"
 	"sync"
 	"testing"
-	"testing/quick"
 	"time"

 	lz4 "github.com/pierrec/lz4/v4"
-	"github.com/syncthing/syncthing/lib/build"
+	"google.golang.org/protobuf/proto"
+
+	"github.com/syncthing/syncthing/internal/gen/bep"
 	"github.com/syncthing/syncthing/lib/rand"
 	"github.com/syncthing/syncthing/lib/testutil"
 )

 var (
-	c0ID     = NewDeviceID([]byte{1})
-	c1ID     = NewDeviceID([]byte{2})
-	quickCfg = &quick.Config{}
+	c0ID = NewDeviceID([]byte{1})
+	c1ID = NewDeviceID([]byte{2})
 )

 func TestPing(t *testing.T) {
@@ -199,7 +202,7 @@ func TestClusterConfigFirst(t *testing.T) {
 	defer closeAndWait(c, rw)

 	select {
-	case c.outbox <- asyncMessage{&Ping{}, nil}:
+	case c.outbox <- asyncMessage{&bep.Ping{}, nil}:
 		t.Fatal("able to send ping before cluster config")
 	case <-time.After(100 * time.Millisecond):
 		// Allow some time for c.writerLoop to setup after c.Start
@@ -208,7 +211,7 @@ func TestClusterConfigFirst(t *testing.T) {
 	c.ClusterConfig(&ClusterConfig{})

 	done := make(chan struct{})
-	if ok := c.send(context.Background(), &Ping{}, done); !ok {
+	if ok := c.send(context.Background(), &bep.Ping{}, done); !ok {
 		t.Fatal("send ping after cluster config returned false")
 	}
 	select {
@@ -263,168 +266,27 @@ func TestCloseTimeout(t *testing.T) {
 	}
 }

-func TestMarshalIndexMessage(t *testing.T) {
-	if testing.Short() {
-		quickCfg.MaxCount = 10
-	}
-
-	f := func(m1 Index) bool {
-		if len(m1.Files) == 0 {
-			m1.Files = nil
-		}
-		for i, f := range m1.Files {
-			if len(f.BlocksHash) == 0 {
-				m1.Files[i].BlocksHash = nil
-			}
-			if len(f.VersionHash) == 0 {
-				m1.Files[i].VersionHash = nil
-			}
-			if len(f.Blocks) == 0 {
-				m1.Files[i].Blocks = nil
-			} else {
-				for j := range f.Blocks {
-					f.Blocks[j].Offset = 0
-					if len(f.Blocks[j].Hash) == 0 {
-						f.Blocks[j].Hash = nil
-					}
-				}
-			}
-			if len(f.Version.Counters) == 0 {
-				m1.Files[i].Version.Counters = nil
-			}
-			if len(f.Encrypted) == 0 {
-				m1.Files[i].Encrypted = nil
-			}
-		}
-
-		return testMarshal(t, "index", &m1, &Index{})
-	}
-
-	if err := quick.Check(f, quickCfg); err != nil {
-		t.Error(err)
-	}
-}
-
-func TestMarshalRequestMessage(t *testing.T) {
-	if testing.Short() {
-		quickCfg.MaxCount = 10
-	}
-
-	f := func(m1 Request) bool {
-		if len(m1.Hash) == 0 {
-			m1.Hash = nil
-		}
-		return testMarshal(t, "request", &m1, &Request{})
-	}
-
-	if err := quick.Check(f, quickCfg); err != nil {
-		t.Error(err)
-	}
-}
-
-func TestMarshalResponseMessage(t *testing.T) {
-	if testing.Short() {
-		quickCfg.MaxCount = 10
-	}
-
-	f := func(m1 Response) bool {
-		if len(m1.Data) == 0 {
-			m1.Data = nil
-		}
-		return testMarshal(t, "response", &m1, &Response{})
-	}
-
-	if err := quick.Check(f, quickCfg); err != nil {
-		t.Error(err)
-	}
-}
-
-func TestMarshalClusterConfigMessage(t *testing.T) {
-	if testing.Short() {
-		quickCfg.MaxCount = 10
-	}
-
-	f := func(m1 ClusterConfig) bool {
-		if len(m1.Folders) == 0 {
-			m1.Folders = nil
-		}
-		for i := range m1.Folders {
-			if len(m1.Folders[i].Devices) == 0 {
-				m1.Folders[i].Devices = nil
-			}
-			for j := range m1.Folders[i].Devices {
-				if len(m1.Folders[i].Devices[j].Addresses) == 0 {
-					m1.Folders[i].Devices[j].Addresses = nil
-				}
-				if len(m1.Folders[i].Devices[j].EncryptionPasswordToken) == 0 {
-					m1.Folders[i].Devices[j].EncryptionPasswordToken = nil
-				}
-			}
-		}
-
-		return testMarshal(t, "clusterconfig", &m1, &ClusterConfig{})
-	}
-
-	if err := quick.Check(f, quickCfg); err != nil {
-		t.Error(err)
-	}
-}
-
-func TestMarshalCloseMessage(t *testing.T) {
-	if testing.Short() {
-		quickCfg.MaxCount = 10
-	}
-
-	f := func(m1 Close) bool {
-		return testMarshal(t, "close", &m1, &Close{})
-	}
-
-	if err := quick.Check(f, quickCfg); err != nil {
-		t.Error(err)
-	}
-}
-
-func TestMarshalFDPU(t *testing.T) {
-	if testing.Short() {
-		quickCfg.MaxCount = 10
-	}
-
-	f := func(m1 FileDownloadProgressUpdate) bool {
-		if len(m1.Version.Counters) == 0 {
-			m1.Version.Counters = nil
-		}
-		if len(m1.BlockIndexes) == 0 {
-			m1.BlockIndexes = nil
-		}
-		return testMarshal(t, "fdpu", &m1, &FileDownloadProgressUpdate{})
-	}
-
-	if err := quick.Check(f, quickCfg); err != nil {
-		t.Error(err)
-	}
-}
-
 func TestUnmarshalFDPUv16v17(t *testing.T) {
-	var fdpu FileDownloadProgressUpdate
+	var fdpu bep.FileDownloadProgressUpdate

 	m0, _ := hex.DecodeString("08cda1e2e3011278f3918787f3b89b8af2958887f0aa9389f3a08588f3aa8f96f39aa8a5f48b9188f19286a0f3848da4f3aba799f3beb489f0a285b9f487b684f2a3bda2f48598b4f2938a89f2a28badf187a0a2f2aebdbdf4849494f4808fbbf2b3a2adf2bb95bff0a6ada4f198ab9af29a9c8bf1abb793f3baabb2f188a6ba1a0020bb9390f60220f6d9e42220b0c7e2b2fdffffffff0120fdb2dfcdfbffffffff0120cedab1d50120bd8784c0feffffffff0120ace99591fdffffffff0120eed7d09af9ffffffff01")
-	if err := fdpu.Unmarshal(m0); err != nil {
+	if err := proto.Unmarshal(m0, &fdpu); err != nil {
 		t.Fatal("Unmarshalling message from v0.14.16:", err)
 	}

 	m1, _ := hex.DecodeString("0880f1969905128401f099b192f0abb1b9f3b280aff19e9aa2f3b89e84f484b39df1a7a6b0f1aea4b1f0adac94f3b39caaf1939281f1928a8af0abb1b0f0a8b3b3f3a88e94f2bd85acf29c97a9f2969da6f0b7a188f1908ea2f09a9c9bf19d86a6f29aada8f389bb95f0bf9d88f1a09d89f1b1a4b5f29b9eabf298a59df1b2a589f2979ebdf0b69880f18986b21a440a1508c7d8fb8897ca93d90910e8c4d8e8f2f8f0ccee010a1508afa8ffd8c085b393c50110e5bdedc3bddefe9b0b0a1408a1bedddba4cac5da3c10b8e5d9958ca7e3ec19225ae2f88cb2f8ffffffff018ceda99cfbffffffff01b9c298a407e295e8e9fcffffffff01f3b9ade5fcffffffff01c08bfea9fdffffffff01a2c2e5e1ffffffffff0186dcc5dafdffffffff01e9ffc7e507c9d89db8fdffffffff01")
-	if err := fdpu.Unmarshal(m1); err != nil {
+	if err := proto.Unmarshal(m1, &fdpu); err != nil {
 		t.Fatal("Unmarshalling message from v0.14.16:", err)
 	}
 }

-func testMarshal(t *testing.T, prefix string, m1, m2 message) bool {
-	buf, err := m1.Marshal()
+func testMarshal(t *testing.T, prefix string, m1, m2 proto.Message) bool {
+	buf, err := proto.Marshal(m1)
 	if err != nil {
 		t.Fatal(err)
 	}

-	err = m2.Unmarshal(buf)
+	err = proto.Unmarshal(buf, m2)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -449,7 +311,7 @@ func TestWriteCompressed(t *testing.T) {
 			compression: CompressionAlways,
 		}

-		msg := &Response{Data: make([]byte, 10240)}
+		msg := (&Response{Data: make([]byte, 10240)}).toWire()
 		if random {
 			// This should make the message uncompressible.
 			rand.Read(msg.Data)
@@ -462,12 +324,12 @@ func TestWriteCompressed(t *testing.T) {
 		if err != nil {
 			t.Fatal(err)
 		}
-		if !bytes.Equal(got.(*Response).Data, msg.Data) {
+		if !bytes.Equal(got.(*bep.Response).Data, msg.Data) {
 			t.Error("received the wrong message")
 		}

-		hdr := Header{Type: typeOf(msg)}
-		size := int64(2 + hdr.ProtoSize() + 4 + msg.ProtoSize())
+		hdr := &bep.Header{Type: typeOf(msg)}
+		size := int64(2 + proto.Size(hdr) + 4 + proto.Size(msg))
 		if c.cr.Tot() > size {
 			t.Errorf("compression enlarged message from %d to %d",
 				size, c.cr.Tot())
@@ -663,236 +525,6 @@ func BenchmarkBlockSize(b *testing.B) {
 	}
 }

-func TestLocalFlagBits(t *testing.T) {
-	var f FileInfo
-	if f.IsIgnored() || f.MustRescan() || f.IsInvalid() {
-		t.Error("file should have no weird bits set by default")
-	}
-
-	f.SetIgnored()
-	if !f.IsIgnored() || f.MustRescan() || !f.IsInvalid() {
-		t.Error("file should be ignored and invalid")
-	}
-
-	f.SetMustRescan()
-	if f.IsIgnored() || !f.MustRescan() || !f.IsInvalid() {
-		t.Error("file should be must-rescan and invalid")
-	}
-
-	f.SetUnsupported()
-	if f.IsIgnored() || f.MustRescan() || !f.IsInvalid() {
-		t.Error("file should be invalid")
-	}
-}
-
-func TestIsEquivalent(t *testing.T) {
-	b := func(v bool) *bool {
-		return &v
-	}
-
-	type testCase struct {
-		a         FileInfo
-		b         FileInfo
-		ignPerms  *bool // nil means should not matter, we'll test both variants
-		ignBlocks *bool
-		ignFlags  uint32
-		eq        bool
-	}
-	cases := []testCase{
-		// Empty FileInfos are equivalent
-		{eq: true},
-
-		// Various basic attributes, all of which cause inequality when
-		// they differ
-		{
-			a:  FileInfo{Name: "foo"},
-			b:  FileInfo{Name: "bar"},
-			eq: false,
-		},
-		{
-			a:  FileInfo{Type: FileInfoTypeFile},
-			b:  FileInfo{Type: FileInfoTypeDirectory},
-			eq: false,
-		},
-		{
-			a:  FileInfo{Size: 1234},
-			b:  FileInfo{Size: 2345},
-			eq: false,
-		},
-		{
-			a:  FileInfo{Deleted: false},
-			b:  FileInfo{Deleted: true},
-			eq: false,
-		},
-		{
-			a:  FileInfo{RawInvalid: false},
-			b:  FileInfo{RawInvalid: true},
-			eq: false,
-		},
-		{
-			a:  FileInfo{ModifiedS: 1234},
-			b:  FileInfo{ModifiedS: 2345},
-			eq: false,
-		},
-		{
-			a:  FileInfo{ModifiedNs: 1234},
-			b:  FileInfo{ModifiedNs: 2345},
-			eq: false,
-		},
-
-		// Special handling of local flags and invalidity. "MustRescan"
-		// files are never equivalent to each other. Otherwise, equivalence
-		// is based just on whether the file becomes IsInvalid() or not, not
-		// the specific reason or flag bits.
-		{
-			a:  FileInfo{LocalFlags: FlagLocalMustRescan},
-			b:  FileInfo{LocalFlags: FlagLocalMustRescan},
-			eq: false,
-		},
-		{
-			a:  FileInfo{RawInvalid: true},
-			b:  FileInfo{RawInvalid: true},
-			eq: true,
-		},
-		{
-			a:  FileInfo{LocalFlags: FlagLocalUnsupported},
-			b:  FileInfo{LocalFlags: FlagLocalUnsupported},
-			eq: true,
-		},
-		{
-			a:  FileInfo{RawInvalid: true},
-			b:  FileInfo{LocalFlags: FlagLocalUnsupported},
-			eq: true,
-		},
-		{
-			a:  FileInfo{LocalFlags: 0},
-			b:  FileInfo{LocalFlags: FlagLocalReceiveOnly},
-			eq: false,
-		},
-		{
-			a:        FileInfo{LocalFlags: 0},
-			b:        FileInfo{LocalFlags: FlagLocalReceiveOnly},
-			ignFlags: FlagLocalReceiveOnly,
-			eq:       true,
-		},
-
-		// Difference in blocks is not OK
-		{
-			a:         FileInfo{Blocks: []BlockInfo{{Hash: []byte{1, 2, 3, 4}}}},
-			b:         FileInfo{Blocks: []BlockInfo{{Hash: []byte{2, 3, 4, 5}}}},
-			ignBlocks: b(false),
-			eq:        false,
-		},
-
-		// ... unless we say it is
-		{
-			a:         FileInfo{Blocks: []BlockInfo{{Hash: []byte{1, 2, 3, 4}}}},
-			b:         FileInfo{Blocks: []BlockInfo{{Hash: []byte{2, 3, 4, 5}}}},
-			ignBlocks: b(true),
-			eq:        true,
-		},
-
-		// Difference in permissions is not OK.
-		{
-			a:        FileInfo{Permissions: 0o444},
-			b:        FileInfo{Permissions: 0o666},
-			ignPerms: b(false),
-			eq:       false,
-		},
-
-		// ... unless we say it is
-		{
-			a:        FileInfo{Permissions: 0o666},
-			b:        FileInfo{Permissions: 0o444},
-			ignPerms: b(true),
-			eq:       true,
-		},
-
-		// These attributes are not checked at all
-		{
-			a:  FileInfo{NoPermissions: false},
-			b:  FileInfo{NoPermissions: true},
-			eq: true,
-		},
-		{
-			a:  FileInfo{Version: Vector{Counters: []Counter{{ID: 1, Value: 42}}}},
-			b:  FileInfo{Version: Vector{Counters: []Counter{{ID: 42, Value: 1}}}},
-			eq: true,
-		},
-		{
-			a:  FileInfo{Sequence: 1},
-			b:  FileInfo{Sequence: 2},
-			eq: true,
-		},
-
-		// The block size is not checked (but this would fail the blocks
-		// check in real world)
-		{
-			a:  FileInfo{RawBlockSize: 1},
-			b:  FileInfo{RawBlockSize: 2},
-			eq: true,
-		},
-
-		// The symlink target is checked for symlinks
-		{
-			a:  FileInfo{Type: FileInfoTypeSymlink, SymlinkTarget: "a"},
-			b:  FileInfo{Type: FileInfoTypeSymlink, SymlinkTarget: "b"},
-			eq: false,
-		},
-
-		// ... but not for non-symlinks
-		{
-			a:  FileInfo{Type: FileInfoTypeFile, SymlinkTarget: "a"},
-			b:  FileInfo{Type: FileInfoTypeFile, SymlinkTarget: "b"},
-			eq: true,
-		},
-	}
-
-	if build.IsWindows {
-		// On windows we only check the user writable bit of the permission
-		// set, so these are equivalent.
-		cases = append(cases, testCase{
-			a:        FileInfo{Permissions: 0o777},
-			b:        FileInfo{Permissions: 0o600},
-			ignPerms: b(false),
-			eq:       true,
-		})
-	}
-
-	for i, tc := range cases {
-		// Check the standard attributes with all permutations of the
-		// special ignore flags, unless the value of those flags are given
-		// in the tests.
-		for _, ignPerms := range []bool{true, false} {
-			for _, ignBlocks := range []bool{true, false} {
-				if tc.ignPerms != nil && *tc.ignPerms != ignPerms {
-					continue
-				}
-				if tc.ignBlocks != nil && *tc.ignBlocks != ignBlocks {
-					continue
-				}
-
-				if res := tc.a.isEquivalent(tc.b, FileInfoComparison{IgnorePerms: ignPerms, IgnoreBlocks: ignBlocks, IgnoreFlags: tc.ignFlags}); res != tc.eq {
-					t.Errorf("Case %d:\na: %v\nb: %v\na.IsEquivalent(b, %v, %v) => %v, expected %v", i, tc.a, tc.b, ignPerms, ignBlocks, res, tc.eq)
-				}
-				if res := tc.b.isEquivalent(tc.a, FileInfoComparison{IgnorePerms: ignPerms, IgnoreBlocks: ignBlocks, IgnoreFlags: tc.ignFlags}); res != tc.eq {
-					t.Errorf("Case %d:\na: %v\nb: %v\nb.IsEquivalent(a, %v, %v) => %v, expected %v", i, tc.a, tc.b, ignPerms, ignBlocks, res, tc.eq)
-				}
-			}
-		}
-	}
-}
-
-func TestSha256OfEmptyBlock(t *testing.T) {
-	// every block size should have a correct entry in sha256OfEmptyBlock
-	for blockSize := MinBlockSize; blockSize <= MaxBlockSize; blockSize *= 2 {
-		expected := sha256.Sum256(make([]byte, blockSize))
-		if sha256OfEmptyBlock[blockSize] != expected {
-			t.Error("missing or wrong hash for block of size", blockSize)
-		}
-	}
-}
-
 // TestClusterConfigAfterClose checks that ClusterConfig does not deadlock when
 // ClusterConfig is called on a closed connection.
 func TestClusterConfigAfterClose(t *testing.T) {
@@ -930,7 +562,7 @@ func TestDispatcherToCloseDeadlock(t *testing.T) {
 	c.Start()
 	defer closeAndWait(c, rw)

-	c.inbox <- &ClusterConfig{}
+	c.inbox <- &bep.ClusterConfig{}

 	select {
 	case <-c.dispatcherLoopStopped:
@@ -939,55 +571,6 @@ func TestDispatcherToCloseDeadlock(t *testing.T) {
 	}
 }

-func TestBlocksEqual(t *testing.T) {
-	blocksOne := []BlockInfo{{Hash: []byte{1, 2, 3, 4}}}
-	blocksTwo := []BlockInfo{{Hash: []byte{5, 6, 7, 8}}}
-	hashOne := []byte{42, 42, 42, 42}
-	hashTwo := []byte{29, 29, 29, 29}
-
-	cases := []struct {
-		b1 []BlockInfo
-		h1 []byte
-		b2 []BlockInfo
-		h2 []byte
-		eq bool
-	}{
-		{blocksOne, hashOne, blocksOne, hashOne, true},  // everything equal
-		{blocksOne, hashOne, blocksTwo, hashTwo, false}, // nothing equal
-		{blocksOne, hashOne, blocksOne, nil, true},      // blocks compared
-		{blocksOne, nil, blocksOne, nil, true},          // blocks compared
-		{blocksOne, nil, blocksTwo, nil, false},         // blocks compared
-		{blocksOne, hashOne, blocksTwo, hashOne, true},  // hashes equal, blocks not looked at
-		{blocksOne, hashOne, blocksOne, hashTwo, true},  // hashes different, blocks compared
-		{blocksOne, hashOne, blocksTwo, hashTwo, false}, // hashes different, blocks compared
-		{blocksOne, hashOne, nil, nil, false},           // blocks is different from no blocks
-		{blocksOne, nil, nil, nil, false},               // blocks is different from no blocks
-		{nil, hashOne, nil, nil, true},                  // nil blocks are equal, even of one side has a hash
-	}
-
-	for _, tc := range cases {
-		f1 := FileInfo{Blocks: tc.b1, BlocksHash: tc.h1}
-		f2 := FileInfo{Blocks: tc.b2, BlocksHash: tc.h2}
-
-		if !f1.BlocksEqual(f1) {
-			t.Error("f1 is always equal to itself", f1)
-		}
-		if !f2.BlocksEqual(f2) {
-			t.Error("f2 is always equal to itself", f2)
-		}
-		if res := f1.BlocksEqual(f2); res != tc.eq {
-			t.Log("f1", f1.BlocksHash, f1.Blocks)
-			t.Log("f2", f2.BlocksHash, f2.Blocks)
-			t.Errorf("f1.BlocksEqual(f2) == %v but should be %v", res, tc.eq)
-		}
-		if res := f2.BlocksEqual(f1); res != tc.eq {
-			t.Log("f1", f1.BlocksHash, f1.Blocks)
-			t.Log("f2", f2.BlocksHash, f2.Blocks)
-			t.Errorf("f2.BlocksEqual(f1) == %v but should be %v", res, tc.eq)
-		}
-	}
-}
-
 func TestIndexIDString(t *testing.T) {
 	// Index ID is a 64 bit, zero padded hex integer.
 	var i IndexID = 42
@@ -1,15 +1,66 @@
-// Copyright (C) 2015 The Protocol Authors.
+// Copyright (C) 2015 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

-import "time"
+import (
+	"time"
+
+	"github.com/syncthing/syncthing/internal/gen/bep"
+)

 // The Vector type represents a version vector. The zero value is a usable
 // version vector. The vector has slice semantics and some operations on it
 // are "append-like" in that they may return the same vector modified, or v
 // new allocated Vector with the modified contents.
+type Vector struct {
+	Counters []Counter
+}
+
+func (v *Vector) ToWire() *bep.Vector {
+	counters := make([]*bep.Counter, len(v.Counters))
+	for i, c := range v.Counters {
+		counters[i] = c.toWire()
+	}
+	return &bep.Vector{
+		Counters: counters,
+	}
+}
+
+func VectorFromWire(w *bep.Vector) Vector {
+	var v Vector
+	if w == nil || len(w.Counters) == 0 {
+		return v
+	}
+	v.Counters = make([]Counter, len(w.Counters))
+	for i, c := range w.Counters {
+		v.Counters[i] = counterFromWire(c)
+	}
+	return v
+}

 // Counter represents a single counter in the version vector.
+type Counter struct {
+	ID    ShortID
+	Value uint64
+}
+
+func (c *Counter) toWire() *bep.Counter {
+	return &bep.Counter{
+		Id:    uint64(c.ID),
+		Value: c.Value,
+	}
+}
+
+func counterFromWire(w *bep.Counter) Counter {
+	return Counter{
+		ID:    ShortID(w.Id),
+		Value: w.Value,
+	}
+}

 // Update returns a Vector with the index for the specific ID incremented by
 // one. If it is possible, the vector v is updated and returned. If it is not,
@@ -1,4 +1,8 @@
-// Copyright (C) 2015 The Protocol Authors.
+// Copyright (C) 2015 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol

@@ -1,4 +1,8 @@
-// Copyright (C) 2014 The Protocol Authors.
+// Copyright (C) 2014 The Syncthing Authors.
+//
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this file,
+// You can obtain one at https://mozilla.org/MPL/2.0/.

 package protocol