Compare commits

..
Author SHA1 Message Date
Jakob Borg 24e691925c Add Debian build stuff 2015-05-11 22:09:38 +02:00
426 changed files with 8973 additions and 26995 deletions
-1
View File
@@ -13,4 +13,3 @@ coverage.xml
syncthing.md5 syncthing.md5
syncthing.exe.md5 syncthing.exe.md5
RELEASE RELEASE
deb
+2 -11
View File
@@ -3,37 +3,28 @@
Aaron Bieber <qbit@deftly.net> Aaron Bieber <qbit@deftly.net>
Alexander Graf <register-github@alex-graf.de> Alexander Graf <register-github@alex-graf.de>
Andrew Dunham <andrew@du.nham.ca> Andrew Dunham <andrew@du.nham.ca>
Antony Male <antony.male@gmail.com>
Arthur Axel fREW Schmidt <frew@afoolishmanifesto.com> <frioux@gmail.com>
Audrius Butkevicius <audrius.butkevicius@gmail.com> Audrius Butkevicius <audrius.butkevicius@gmail.com>
Bart De Vries <devriesb@gmail.com> Arthur Axel fREW Schmidt <frew@afoolishmanifesto.com> <frioux@gmail.com>
Ben Curthoys <ben@bencurthoys.com> Ben Curthoys <ben@bencurthoys.com>
Ben Schulz <ueomkail@gmail.com> <uok@users.noreply.github.com> Ben Schulz <ueomkail@gmail.com> <uok@users.noreply.github.com>
Ben Sidhom <bsidhom@gmail.com> Ben Sidhom <bsidhom@gmail.com>
Brandon Philips <brandon@ifup.org> Brandon Philips <brandon@ifup.org>
Brendan Long <self@brendanlong.com> Brendan Long <self@brendanlong.com>
Brian R. Becker <brbecker@gmail.com>
Caleb Callaway <enlightened.despot@gmail.com> Caleb Callaway <enlightened.despot@gmail.com>
Carsten Hagemann <moter8@gmail.com> Carsten Hagemann <moter8@gmail.com>
Cathryne Linenweaver <cathryne.linenweaver@gmail.com> <Cathryne@users.noreply.github.com> Cathryne Linenweaver <cathryne.linenweaver@gmail.com> <Cathryne@users.noreply.github.com>
Chris Howie <me@chrishowie.com>
Chris Joel <chris@scriptolo.gy> Chris Joel <chris@scriptolo.gy>
Colin Kennedy <moshen.colin@gmail.com> Colin Kennedy <moshen.colin@gmail.com>
Daniel Bergmann <dan.arne.bergmann@gmail.com> <brgmnn@users.noreply.github.com>
Daniel Martí <mvdan@mvdan.cc> Daniel Martí <mvdan@mvdan.cc>
Denis A. <denisva@gmail.com>
Dennis Wilson <dw@risu.io> Dennis Wilson <dw@risu.io>
Dominik Heidler <dominik@heidler.eu> Dominik Heidler <dominik@heidler.eu>
Elias Jarlebring <jarlebring@gmail.com> Elias Jarlebring <jarlebring@gmail.com>
Emil Hessman <emil@hessman.se> Emil Hessman <emil@hessman.se>
Erik Meitner <e.meitner@willystreet.coop>
Federico Castagnini <federico.castagnini@gmail.com> Federico Castagnini <federico.castagnini@gmail.com>
Felix Ableitner <me@nutomic.com> Felix Ableitner <me@nutomic.com>
Felix Unterpaintner <bigbear2nd@gmail.com> Felix Unterpaintner <bigbear2nd@gmail.com>
Francois-Xavier Gsell <fxgsell@gmail.com> Francois-Xavier Gsell <fxgsell@gmail.com>
Frank Isemann <frank@isemann.name>
Gilli Sigurdsson <gilli@vx.is> Gilli Sigurdsson <gilli@vx.is>
Jacek Szafarkiewicz <szafar@linux.pl>
Jakob Borg <jakob@nym.se> Jakob Borg <jakob@nym.se>
James Patterson <jamespatterson@operamail.com> <jpjp@users.noreply.github.com> James Patterson <jamespatterson@operamail.com> <jpjp@users.noreply.github.com>
Jaroslav Malec <dzardacz@gmail.com> Jaroslav Malec <dzardacz@gmail.com>
@@ -44,9 +35,9 @@ Karol Różycki <rozycki.karol@gmail.com>
Ken'ichi Kamada <kamada@nanohz.org> Ken'ichi Kamada <kamada@nanohz.org>
Lode Hoste <zillode@zillode.be> Lode Hoste <zillode@zillode.be>
Lord Landon Agahnim <lordlandon@gmail.com> Lord Landon Agahnim <lordlandon@gmail.com>
Marcin Dziadus <dziadus.marcin@gmail.com>
Marc Laporte <marc@marclaporte.com> <marc@laporte.name> Marc Laporte <marc@marclaporte.com> <marc@laporte.name>
Marc Pujol <kilburn@la3.org> Marc Pujol <kilburn@la3.org>
Marcin Dziadus <dziadus.marcin@gmail.com>
Michael Jephcote <rewt0r@gmx.com> <Rewt0r@users.noreply.github.com> Michael Jephcote <rewt0r@gmx.com> <Rewt0r@users.noreply.github.com>
Michael Tilli <pyfisch@gmail.com> Michael Tilli <pyfisch@gmail.com>
Pascal Jungblut <github@pascalj.com> <mail@pascal-jungblut.com> Pascal Jungblut <github@pascalj.com> <mail@pascal-jungblut.com>
+95 -7
View File
@@ -32,15 +32,64 @@ latest info on Transifex.
## Contributing Code ## Contributing Code
Every contribution is welcome. If you want to contribute but are unsure Every contribution is welcome. If you want to contribute but are unsure
where to start, any open issues are fair game! See the [Contribution where to start, any open issues are fair game! Be prepared for a
Guidelines](http://docs.syncthing.net/dev/contributing.html) for the full [certain amount of review](https://github.com/syncthing/syncthing/wiki/FAQ#why-are-you-being-so-hard-on-my-pull-request);
story on committing code. it's all in the name of quality. :) Following the points below will make this
a smoother process.
## Contributing Documentation Individuals making significant and valuable contributions are given
commit-access to the project. If you make a significant contribution and
are not considered for commit-access, please contact any of the
Syncthing core team members.
Updates to the [documentation site](http://docs.syncthing.net/) can be All nontrivial contributions should go through the pull request
made as pull requests on the [documentation mechanism for internal review. Determining what is "nontrivial" is left
repository](https://github.com/syncthing/docs). at the discretion of the contributor.
### Authorship
All code authors are listed in the AUTHORS file. Commits must be made
with the same name and email as listed in the AUTHORS file. To
accomplish this, ensure that your git configuration is set correctly
prior to making your first commit;
$ git config --global user.name "Jane Doe"
$ git config --global user.email janedoe@example.com
You must be reachable on the given email address. If you do not wish to
use your real name for whatever reason, using a nickname or pseudonym is
perfectly acceptable.
### Core Team
The Syncthing core team currently consists of the following members;
- Jakob Borg (@calmh)
- Audrius Butkevicius (@AudriusButkevicius)
## Coding Style
- Follow the conventions laid out in [Effective Go](https://golang.org/doc/effective_go.html)
as much as makes sense.
- All text files use Unix line endings.
- Each commit should be `go fmt` clean.
- The commit message subject should be a single short sentence
describing the change, starting with a capital letter.
- Commits that resolve an existing issue must include the issue number
as `(fixes #123)` at the end of the commit message subject.
- Imports are grouped per `goimports` standard; that is, standard
library first, then third party libraries after a blank line.
- A contribution solving a single issue or introducing a single new
feature should probably be a single commit based on the current
`master` branch. You may be asked to "rebase" or "squash" your pull
request to make sure this is the case, especially if there have been
amendments during review.
## Licensing ## Licensing
@@ -50,3 +99,42 @@ strings which are licensed under the Creative Commons Attribution 4.0
International License. You retain the copyright to code you have International License. You retain the copyright to code you have
written. written.
When accepting your first contribution, the maintainer of the project
will ensure that you are added to the AUTHORS file. You are welcome to
add yourself as a separate commit in your first pull request.
## Building
[See the documentation](https://github.com/syncthing/syncthing/wiki/Building)
on how to get started with a build environment.
## Branches
- `master` is the main branch containing good code that will end up in
the next release. You should base your work on it. It won't ever be
rebased or force-pushed to.
- `vx.y` branches exist to make patch releases on otherwise obsolete
minor releases. Should only contain fixes cherry picked from master.
Don't base any work on them.
- Other branches are probably topic branches and may be subject to
rebasing. Don't base any work on them unless you specifically know
otherwise.
## Tags
All releases are tagged semver style as `vx.y.z`. Release tags are
signed by GPG key BCE524C7.
## Tests
Yes please!
## Documentation
[Over here!](https://github.com/syncthing/syncthing/wiki)
## License
MPLv2
+16 -20
View File
@@ -1,21 +1,17 @@
{ {
"ImportPath": "github.com/syncthing/syncthing", "ImportPath": "github.com/syncthing/syncthing",
"GoVersion": "devel", "GoVersion": "go1.4",
"Packages": [ "Packages": [
"./cmd/..." "./cmd/..."
], ],
"Deps": [ "Deps": [
{ {
"ImportPath": "github.com/bkaradzic/go-lz4", "ImportPath": "github.com/bkaradzic/go-lz4",
"Rev": "4f7c2045dbd17b802370e2e6022200468abf02ba" "Rev": "93a831dcee242be64a9cc9803dda84af25932de7"
},
{
"ImportPath": "github.com/calmh/du",
"Rev": "3c0690cca16228b97741327b1b6781397afbdb24"
}, },
{ {
"ImportPath": "github.com/calmh/logger", "ImportPath": "github.com/calmh/logger",
"Rev": "c96f6a1a8c7b6bf2f4860c667867d90174799eb2" "Rev": "4d4e2801954c5581e4c2a80a3d3beb3b3645fd04"
}, },
{ {
"ImportPath": "github.com/calmh/luhn", "ImportPath": "github.com/calmh/luhn",
@@ -25,29 +21,29 @@
"ImportPath": "github.com/calmh/xdr", "ImportPath": "github.com/calmh/xdr",
"Rev": "5f7208e86762911861c94f1849eddbfc0a60cbf0" "Rev": "5f7208e86762911861c94f1849eddbfc0a60cbf0"
}, },
{
"ImportPath": "github.com/golang/snappy",
"Rev": "0c7f8a7704bfec561913f4df52c832f094ef56f0"
},
{ {
"ImportPath": "github.com/juju/ratelimit", "ImportPath": "github.com/juju/ratelimit",
"Rev": "772f5c38e468398c4511514f4f6aa9a4185bc0a0" "Rev": "c5abe513796336ee2869745bff0638508450e9c5"
}, },
{ {
"ImportPath": "github.com/kardianos/osext", "ImportPath": "github.com/kardianos/osext",
"Rev": "6e7f843663477789fac7c02def0d0909e969b4e5" "Rev": "efacde03154693404c65e7aa7d461ac9014acd0c"
}, },
{ {
"ImportPath": "github.com/syncthing/protocol", "ImportPath": "github.com/syncthing/protocol",
"Rev": "ebcdea63c07327a342f65415bbadc497462b8f1f" "Rev": "e7db2648034fb71b051902a02bc25d4468ed492e"
}, },
{ {
"ImportPath": "github.com/syndtr/goleveldb/leveldb", "ImportPath": "github.com/syndtr/goleveldb/leveldb",
"Rev": "183614d6b32571e867df4cf086f5480ceefbdfac" "Rev": "87e4e645d80ae9c537e8f2dee52b28036a5dd75e"
},
{
"ImportPath": "github.com/syndtr/gosnappy/snappy",
"Rev": "156a073208e131d7d2e212cb749feae7c339e846"
}, },
{ {
"ImportPath": "github.com/thejerf/suture", "ImportPath": "github.com/thejerf/suture",
"Rev": "fc7aaeabdc43fe41c5328efa1479ffea0b820978" "Rev": "ff19fb384c3fe30f42717967eaa69da91e5f317c"
}, },
{ {
"ImportPath": "github.com/vitrun/qart/coding", "ImportPath": "github.com/vitrun/qart/coding",
@@ -63,19 +59,19 @@
}, },
{ {
"ImportPath": "golang.org/x/crypto/bcrypt", "ImportPath": "golang.org/x/crypto/bcrypt",
"Rev": "7d5b0be716b9d6d4269afdaae10032bb296d3cdf" "Rev": "c57d4a71915a248dbad846d60825145062b4c18e"
}, },
{ {
"ImportPath": "golang.org/x/crypto/blowfish", "ImportPath": "golang.org/x/crypto/blowfish",
"Rev": "7d5b0be716b9d6d4269afdaae10032bb296d3cdf" "Rev": "c57d4a71915a248dbad846d60825145062b4c18e"
}, },
{ {
"ImportPath": "golang.org/x/text/transform", "ImportPath": "golang.org/x/text/transform",
"Rev": "3eb7007b740b66a77f3c85f2660a0240b284115a" "Rev": "2076e9cab4147459c82bc81169e46c139d358547"
}, },
{ {
"ImportPath": "golang.org/x/text/unicode/norm", "ImportPath": "golang.org/x/text/unicode/norm",
"Rev": "3eb7007b740b66a77f3c85f2660a0240b284115a" "Rev": "2076e9cab4147459c82bc81169e46c139d358547"
} }
] ]
} }
-23
View File
@@ -1,23 +0,0 @@
// +build gofuzz
package lz4
import "encoding/binary"
func Fuzz(data []byte) int {
if len(data) < 4 {
return 0
}
ln := binary.LittleEndian.Uint32(data)
if ln > (1 << 21) {
return 0
}
if _, err := Decode(nil, data); err != nil {
return 0
}
return 1
}
+1 -6
View File
@@ -141,7 +141,7 @@ func Decode(dst, src []byte) ([]byte, error) {
length += ln length += ln
} }
if int(d.spos+length) > len(d.src) || int(d.dpos+length) > len(d.dst) { if int(d.spos+length) > len(d.src) {
return nil, ErrCorrupt return nil, ErrCorrupt
} }
@@ -179,12 +179,7 @@ func Decode(dst, src []byte) ([]byte, error) {
} }
literal := d.dpos - d.ref literal := d.dpos - d.ref
if literal < 4 { if literal < 4 {
if int(d.dpos+4) > len(d.dst) {
return nil, ErrCorrupt
}
d.cp(4, decr[literal]) d.cp(4, decr[literal])
} else { } else {
length += 4 length += 4
+2 -4
View File
@@ -25,10 +25,8 @@
package lz4 package lz4
import ( import "encoding/binary"
"encoding/binary" import "errors"
"errors"
)
const ( const (
minMatch = 4 minMatch = 4
-24
View File
@@ -1,24 +0,0 @@
This is free and unencumbered software released into the public domain.
Anyone is free to copy, modify, publish, use, compile, sell, or
distribute this software, either in source code form or as a compiled
binary, for any purpose, commercial or non-commercial, and by any
means.
In jurisdictions that recognize copyright laws, the author or authors
of this software dedicate any and all copyright interest in the
software to the public domain. We make this dedication for the benefit
of the public at large and to the detriment of our heirs and
successors. We intend this dedication to be an overt act of
relinquishment in perpetuity of all present and future rights to this
software under copyright law.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR
OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
For more information, please refer to <http://unlicense.org>
-14
View File
@@ -1,14 +0,0 @@
du
==
Get total and available disk space on a given volume.
Documentation
-------------
http://godoc.org/github.com/calmh/du
License
-------
Public Domain
-21
View File
@@ -1,21 +0,0 @@
package main
import (
"fmt"
"log"
"os"
"github.com/calmh/du"
)
var KB = int64(1024)
func main() {
usage, err := du.Get(os.Args[1])
if err != nil {
log.Fatal(err)
}
fmt.Println("Free:", usage.FreeBytes/(KB*KB), "MiB")
fmt.Println("Available:", usage.AvailBytes/(KB*KB), "MiB")
fmt.Println("Size:", usage.TotalBytes/(KB*KB), "MiB")
}
-8
View File
@@ -1,8 +0,0 @@
package du
// Usage holds information about total and available storage on a volume.
type Usage struct {
TotalBytes int64 // Size of volume
FreeBytes int64 // Unused size
AvailBytes int64 // Available to a non-privileged user
}
-24
View File
@@ -1,24 +0,0 @@
// +build !windows,!netbsd,!openbsd,!solaris
package du
import (
"path/filepath"
"syscall"
)
// Get returns the Usage of a given path, or an error if usage data is
// unavailable.
func Get(path string) (Usage, error) {
var stat syscall.Statfs_t
err := syscall.Statfs(filepath.Clean(path), &stat)
if err != nil {
return Usage{}, err
}
u := Usage{
FreeBytes: int64(stat.Bfree) * int64(stat.Bsize),
TotalBytes: int64(stat.Blocks) * int64(stat.Bsize),
AvailBytes: int64(stat.Bavail) * int64(stat.Bsize),
}
return u, nil
}
-13
View File
@@ -1,13 +0,0 @@
// +build netbsd openbsd solaris
package du
import "errors"
var ErrUnsupported = errors.New("unsupported platform")
// Get returns the Usage of a given path, or an error if usage data is
// unavailable.
func Get(path string) (Usage, error) {
return Usage{}, ErrUnsupported
}
-27
View File
@@ -1,27 +0,0 @@
package du
import (
"syscall"
"unsafe"
)
// Get returns the Usage of a given path, or an error if usage data is
// unavailable.
func Get(path string) (Usage, error) {
h := syscall.MustLoadDLL("kernel32.dll")
c := h.MustFindProc("GetDiskFreeSpaceExW")
var u Usage
ret, _, err := c.Call(
uintptr(unsafe.Pointer(syscall.StringToUTF16Ptr(path))),
uintptr(unsafe.Pointer(&u.FreeBytes)),
uintptr(unsafe.Pointer(&u.TotalBytes)),
uintptr(unsafe.Pointer(&u.AvailBytes)))
if ret == 0 {
return u, err
}
return u, nil
}
-8
View File
@@ -6,7 +6,6 @@ package logger
import ( import (
"fmt" "fmt"
"io/ioutil"
"log" "log"
"os" "os"
"strings" "strings"
@@ -38,13 +37,6 @@ type Logger struct {
var DefaultLogger = New() var DefaultLogger = New()
func New() *Logger { func New() *Logger {
if os.Getenv("LOGGER_DISCARD") != "" {
// Hack to completely disable logging, for example when running benchmarks.
return &Logger{
logger: log.New(ioutil.Discard, "", 0),
}
}
return &Logger{ return &Logger{
logger: log.New(os.Stdout, "", log.Ltime), logger: log.New(os.Stdout, "", log.Ltime),
} }
-14
View File
@@ -1,14 +0,0 @@
# This is the official list of Snappy-Go authors for copyright purposes.
# This file is distinct from the CONTRIBUTORS files.
# See the latter for an explanation.
# Names should be added to this file as
# Name or Organization <email address>
# The email address is not required for organizations.
# Please keep the list sorted.
Damian Gryski <dgryski@gmail.com>
Google Inc.
Jan Mercl <0xjnml@gmail.com>
Sebastien Binet <seb.binet@gmail.com>
-36
View File
@@ -1,36 +0,0 @@
# This is the official list of people who can contribute
# (and typically have contributed) code to the Snappy-Go repository.
# The AUTHORS file lists the copyright holders; this file
# lists people. For example, Google employees are listed here
# but not in AUTHORS, because Google holds the copyright.
#
# The submission process automatically checks to make sure
# that people submitting code are listed in this file (by email address).
#
# Names should be added to this file only after verifying that
# the individual or the individual's organization has agreed to
# the appropriate Contributor License Agreement, found here:
#
# http://code.google.com/legal/individual-cla-v1.0.html
# http://code.google.com/legal/corporate-cla-v1.0.html
#
# The agreement for individuals can be filled out on the web.
#
# When adding J Random Contributor's name to this file,
# either J's name or J's organization's name should be
# added to the AUTHORS file, depending on whether the
# individual or corporate CLA was used.
# Names should be added to this file like so:
# Name <email address>
# Please keep the list sorted.
Damian Gryski <dgryski@gmail.com>
Jan Mercl <0xjnml@gmail.com>
Kai Backman <kaib@golang.org>
Marc-Antoine Ruel <maruel@chromium.org>
Nigel Tao <nigeltao@golang.org>
Rob Pike <r@golang.org>
Russ Cox <rsc@golang.org>
Sebastien Binet <seb.binet@gmail.com>
-27
View File
@@ -1,27 +0,0 @@
Copyright (c) 2011 The Snappy-Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-7
View File
@@ -1,7 +0,0 @@
The Snappy compression format in the Go programming language.
To download and install from source:
$ go get github.com/golang/snappy
Unless otherwise noted, the Snappy-Go source files are distributed
under the BSD-style license found in the LICENSE file.
+2 -5
View File
@@ -1,8 +1,5 @@
All files in this repository are licensed as follows. If you contribute This package contains an efficient token-bucket-based rate limiter.
to this repository, it is assumed that you license your contribution Copyright (C) 2015 Canonical Ltd.
under the same license unless you state otherwise.
All files Copyright (C) 2015 Canonical Ltd. unless otherwise specified in the file.
This software is licensed under the LGPLv3, included below. This software is licensed under the LGPLv3, included below.
+1 -1
View File
@@ -20,7 +20,7 @@ token in the bucket represents one byte.
```go ```go
func Writer(w io.Writer, bucket *Bucket) io.Writer func Writer(w io.Writer, bucket *Bucket) io.Writer
``` ```
Writer returns a writer that is rate limited by the given token bucket. Each Writer returns a reader that is rate limited by the given token bucket. Each
token in the bucket represents one byte. token in the bucket represents one byte.
#### type Bucket #### type Bucket
+9 -4
View File
@@ -2,8 +2,7 @@
// Licensed under the LGPLv3 with static-linking exception. // Licensed under the LGPLv3 with static-linking exception.
// See LICENCE file for details. // See LICENCE file for details.
// The ratelimit package provides an efficient token bucket implementation // The ratelimit package provides an efficient token bucket implementation.
// that can be used to limit the rate of arbitrary things.
// See http://en.wikipedia.org/wiki/Token_bucket. // See http://en.wikipedia.org/wiki/Token_bucket.
package ratelimit package ratelimit
@@ -11,7 +10,6 @@ import (
"strconv" "strconv"
"sync" "sync"
"time" "time"
"math"
) )
// Bucket represents a token bucket that fills at a predetermined rate. // Bucket represents a token bucket that fills at a predetermined rate.
@@ -57,7 +55,7 @@ func NewBucketWithRate(rate float64, capacity int64) *Bucket {
continue continue
} }
tb := NewBucketWithQuantum(fillInterval, capacity, quantum) tb := NewBucketWithQuantum(fillInterval, capacity, quantum)
if diff := math.Abs(tb.Rate() - rate); diff/rate <= rateMargin { if diff := abs(tb.Rate() - rate); diff/rate <= rateMargin {
return tb return tb
} }
} }
@@ -219,3 +217,10 @@ func (tb *Bucket) adjust(now time.Time) (currentTick int64) {
tb.availTick = currentTick tb.availTick = currentTick
return return
} }
func abs(f float64) float64 {
if f < 0 {
return -f
}
return f
}
+1 -3
View File
@@ -4,9 +4,7 @@
There is sometimes utility in finding the current executable file There is sometimes utility in finding the current executable file
that is running. This can be used for upgrading the current executable that is running. This can be used for upgrading the current executable
or finding resources located relative to the executable file. Both or finding resources located relative to the executable file.
working directory and the os.Args[0] value are arbitrary and cannot
be relied on; os.Args[0] can be "faked".
Multi-platform and supports: Multi-platform and supports:
* Linux * Linux
+3 -3
View File
@@ -16,12 +16,12 @@ func Executable() (string, error) {
} }
// Returns same path as Executable, returns just the folder // Returns same path as Executable, returns just the folder
// path. Excludes the executable name and any trailing slash. // path. Excludes the executable name.
func ExecutableFolder() (string, error) { func ExecutableFolder() (string, error) {
p, err := Executable() p, err := Executable()
if err != nil { if err != nil {
return "", err return "", err
} }
folder, _ := filepath.Split(p)
return filepath.Dir(p), nil return folder, nil
} }
+2 -4
View File
@@ -17,14 +17,12 @@ import (
func executable() (string, error) { func executable() (string, error) {
switch runtime.GOOS { switch runtime.GOOS {
case "linux": case "linux":
const deletedTag = " (deleted)" const deletedSuffix = " (deleted)"
execpath, err := os.Readlink("/proc/self/exe") execpath, err := os.Readlink("/proc/self/exe")
if err != nil { if err != nil {
return execpath, err return execpath, err
} }
execpath = strings.TrimSuffix(execpath, deletedTag) return strings.TrimSuffix(execpath, deletedSuffix), nil
execpath = strings.TrimPrefix(execpath, deletedTag)
return execpath, nil
case "netbsd": case "netbsd":
return os.Readlink("/proc/curproc/exe") return os.Readlink("/proc/curproc/exe")
case "openbsd", "dragonfly": case "openbsd", "dragonfly":
-23
View File
@@ -24,29 +24,6 @@ const (
executableEnvValueDelete = "delete" executableEnvValueDelete = "delete"
) )
func TestPrintExecutable(t *testing.T) {
ef, err := Executable()
if err != nil {
t.Fatalf("Executable failed: %v", err)
}
t.Log("Executable:", ef)
}
func TestPrintExecutableFolder(t *testing.T) {
ef, err := ExecutableFolder()
if err != nil {
t.Fatalf("ExecutableFolder failed: %v", err)
}
t.Log("Executable Folder:", ef)
}
func TestExecutableFolder(t *testing.T) {
ef, err := ExecutableFolder()
if err != nil {
t.Fatalf("ExecutableFolder failed: %v", err)
}
if ef[len(ef)-1] == filepath.Separator {
t.Fatal("ExecutableFolder ends with a trailing slash.")
}
}
func TestExecutableMatch(t *testing.T) { func TestExecutableMatch(t *testing.T) {
ep, err := Executable() ep, err := Executable()
if err != nil { if err != nil {
+2 -3
View File
@@ -31,7 +31,7 @@ func (t *TestModel) Index(deviceID DeviceID, folder string, files []FileInfo, fl
func (t *TestModel) IndexUpdate(deviceID DeviceID, folder string, files []FileInfo, flags uint32, options []Option) { func (t *TestModel) IndexUpdate(deviceID DeviceID, folder string, files []FileInfo, flags uint32, options []Option) {
} }
func (t *TestModel) Request(deviceID DeviceID, folder, name string, offset int64, size int, hash []byte, flags uint32, options []Option, buf []byte) error { func (t *TestModel) Request(deviceID DeviceID, folder, name string, offset int64, size int, hash []byte, flags uint32, options []Option) ([]byte, error) {
t.folder = folder t.folder = folder
t.name = name t.name = name
t.offset = offset t.offset = offset
@@ -39,8 +39,7 @@ func (t *TestModel) Request(deviceID DeviceID, folder, name string, offset int64
t.hash = hash t.hash = hash
t.flags = flags t.flags = flags
t.options = options t.options = options
copy(buf, t.data) return t.data, nil
return nil
} }
func (t *TestModel) Close(deviceID DeviceID, err error) { func (t *TestModel) Close(deviceID DeviceID, err error) {
-23
View File
@@ -1,23 +0,0 @@
// Copyright (C) 2015 The Protocol Authors.
package protocol
import "testing"
func TestWinsConflict(t *testing.T) {
testcases := [][2]FileInfo{
// The first should always win over the second
{{Modified: 42}, {Modified: 41}},
{{Modified: 41}, {Modified: 42, Flags: FlagDeleted}},
{{Modified: 41, Version: Vector{{42, 2}, {43, 1}}}, {Modified: 41, Version: Vector{{42, 1}, {43, 2}}}},
}
for _, tc := range testcases {
if !tc[0].WinsConflict(tc[1]) {
t.Errorf("%v should win over %v", tc[0], tc[1])
}
if tc[1].WinsConflict(tc[0]) {
t.Errorf("%v should not win over %v", tc[1], tc[0])
}
}
}
-25
View File
@@ -58,31 +58,6 @@ func (f FileInfo) HasPermissionBits() bool {
return f.Flags&FlagNoPermBits == 0 return f.Flags&FlagNoPermBits == 0
} }
// WinsConflict returns true if "f" is the one to choose when it is in
// conflict with "other".
func (f FileInfo) WinsConflict(other FileInfo) bool {
// If a modification is in conflict with a delete, we pick the
// modification.
if !f.IsDeleted() && other.IsDeleted() {
return true
}
if f.IsDeleted() && !other.IsDeleted() {
return false
}
// The one with the newer modification time wins.
if f.Modified > other.Modified {
return true
}
if f.Modified < other.Modified {
return false
}
// The modification times were equal. Use the device ID in the version
// vector as tie breaker.
return f.Version.Compare(other.Version) == ConcurrentGreater
}
type BlockInfo struct { type BlockInfo struct {
Offset int64 // noencode (cache only) Offset int64 // noencode (cache only)
Size int32 Size int32
+2 -2
View File
@@ -26,9 +26,9 @@ func (m nativeModel) IndexUpdate(deviceID DeviceID, folder string, files []FileI
m.next.IndexUpdate(deviceID, folder, files, flags, options) m.next.IndexUpdate(deviceID, folder, files, flags, options)
} }
func (m nativeModel) Request(deviceID DeviceID, folder string, name string, offset int64, hash []byte, flags uint32, options []Option, buf []byte) error { func (m nativeModel) Request(deviceID DeviceID, folder string, name string, offset int64, size int, hash []byte, flags uint32, options []Option) ([]byte, error) {
name = norm.NFD.String(name) name = norm.NFD.String(name)
return m.next.Request(deviceID, folder, name, offset, hash, flags, options, buf) return m.next.Request(deviceID, folder, name, offset, size, hash, flags, options)
} }
func (m nativeModel) ClusterConfig(deviceID DeviceID, config ClusterConfigMessage) { func (m nativeModel) ClusterConfig(deviceID DeviceID, config ClusterConfigMessage) {
+2 -2
View File
@@ -18,8 +18,8 @@ func (m nativeModel) IndexUpdate(deviceID DeviceID, folder string, files []FileI
m.next.IndexUpdate(deviceID, folder, files, flags, options) m.next.IndexUpdate(deviceID, folder, files, flags, options)
} }
func (m nativeModel) Request(deviceID DeviceID, folder string, name string, offset int64, hash []byte, flags uint32, options []Option, buf []byte) error { func (m nativeModel) Request(deviceID DeviceID, folder string, name string, offset int64, size int, hash []byte, flags uint32, options []Option) ([]byte, error) {
return m.next.Request(deviceID, folder, name, offset, hash, flags, options, buf) return m.next.Request(deviceID, folder, name, offset, size, hash, flags, options)
} }
func (m nativeModel) ClusterConfig(deviceID DeviceID, config ClusterConfigMessage) { func (m nativeModel) ClusterConfig(deviceID DeviceID, config ClusterConfigMessage) {
@@ -25,18 +25,18 @@ type nativeModel struct {
} }
func (m nativeModel) Index(deviceID DeviceID, folder string, files []FileInfo, flags uint32, options []Option) { func (m nativeModel) Index(deviceID DeviceID, folder string, files []FileInfo, flags uint32, options []Option) {
fixupFiles(folder, files) fixupFiles(files)
m.next.Index(deviceID, folder, files, flags, options) m.next.Index(deviceID, folder, files, flags, options)
} }
func (m nativeModel) IndexUpdate(deviceID DeviceID, folder string, files []FileInfo, flags uint32, options []Option) { func (m nativeModel) IndexUpdate(deviceID DeviceID, folder string, files []FileInfo, flags uint32, options []Option) {
fixupFiles(folder, files) fixupFiles(files)
m.next.IndexUpdate(deviceID, folder, files, flags, options) m.next.IndexUpdate(deviceID, folder, files, flags, options)
} }
func (m nativeModel) Request(deviceID DeviceID, folder string, name string, offset int64, hash []byte, flags uint32, options []Option, buf []byte) error { func (m nativeModel) Request(deviceID DeviceID, folder string, name string, offset int64, size int, hash []byte, flags uint32, options []Option) ([]byte, error) {
name = filepath.FromSlash(name) name = filepath.FromSlash(name)
return m.next.Request(deviceID, folder, name, offset, hash, flags, options, buf) return m.next.Request(deviceID, folder, name, offset, size, hash, flags, options)
} }
func (m nativeModel) ClusterConfig(deviceID DeviceID, config ClusterConfigMessage) { func (m nativeModel) ClusterConfig(deviceID DeviceID, config ClusterConfigMessage) {
@@ -47,7 +47,7 @@ func (m nativeModel) Close(deviceID DeviceID, err error) {
m.next.Close(deviceID, err) m.next.Close(deviceID, err)
} }
func fixupFiles(folder string, files []FileInfo) { func fixupFiles(files []FileInfo) {
for i, f := range files { for i, f := range files {
if strings.ContainsAny(f.Name, disallowedCharacters) { if strings.ContainsAny(f.Name, disallowedCharacters) {
if f.IsDeleted() { if f.IsDeleted() {
@@ -56,7 +56,7 @@ func fixupFiles(folder string, files []FileInfo) {
continue continue
} }
files[i].Flags |= FlagInvalid files[i].Flags |= FlagInvalid
l.Warnf("File name %q (folder %q) contains invalid characters; marked as invalid.", f.Name, folder) l.Warnf("File name %q contains invalid characters; marked as invalid.", f.Name)
} }
files[i].Name = filepath.FromSlash(files[i].Name) files[i].Name = filepath.FromSlash(files[i].Name)
} }
+58 -101
View File
@@ -31,7 +31,8 @@ const (
const ( const (
stateInitial = iota stateInitial = iota
stateReady stateCCRcvd
stateIdxRcvd
) )
// FileInfo flags // FileInfo flags
@@ -81,7 +82,7 @@ type Model interface {
// An index update was received from the peer device // An index update was received from the peer device
IndexUpdate(deviceID DeviceID, folder string, files []FileInfo, flags uint32, options []Option) IndexUpdate(deviceID DeviceID, folder string, files []FileInfo, flags uint32, options []Option)
// A request was made by the peer device // A request was made by the peer device
Request(deviceID DeviceID, folder string, name string, offset int64, hash []byte, flags uint32, options []Option, buf []byte) error Request(deviceID DeviceID, folder string, name string, offset int64, size int, hash []byte, flags uint32, options []Option) ([]byte, error)
// A cluster configuration message was received // A cluster configuration message was received
ClusterConfig(deviceID DeviceID, config ClusterConfigMessage) ClusterConfig(deviceID DeviceID, config ClusterConfigMessage)
// The peer device closed the connection // The peer device closed the connection
@@ -89,7 +90,6 @@ type Model interface {
} }
type Connection interface { type Connection interface {
Start()
ID() DeviceID ID() DeviceID
Name() string Name() string
Index(folder string, files []FileInfo, flags uint32, options []Option) error Index(folder string, files []FileInfo, flags uint32, options []Option) error
@@ -103,6 +103,7 @@ type rawConnection struct {
id DeviceID id DeviceID
name string name string
receiver Model receiver Model
state int
cr *countingReader cr *countingReader
cw *countingWriter cw *countingWriter
@@ -112,11 +113,11 @@ type rawConnection struct {
idxMut sync.Mutex // ensures serialization of Index calls idxMut sync.Mutex // ensures serialization of Index calls
nextID chan int nextID chan int
outbox chan hdrMsg outbox chan hdrMsg
closed chan struct{} closed chan struct{}
once sync.Once once sync.Once
pool sync.Pool
compression Compression compression Compression
rdbuf0 []byte // used & reused by readMessage rdbuf0 []byte // used & reused by readMessage
@@ -129,9 +130,8 @@ type asyncResult struct {
} }
type hdrMsg struct { type hdrMsg struct {
hdr header hdr header
msg encodable msg encodable
done chan struct{}
} }
type encodable interface { type encodable interface {
@@ -142,9 +142,9 @@ type isEofer interface {
IsEOF() bool IsEOF() bool
} }
var ( const (
PingTimeout = 30 * time.Second pingTimeout = 30 * time.Second
PingIdleTime = 60 * time.Second pingIdleTime = 60 * time.Second
) )
func NewConnection(deviceID DeviceID, reader io.Reader, writer io.Writer, receiver Model, name string, compress Compression) Connection { func NewConnection(deviceID DeviceID, reader io.Reader, writer io.Writer, receiver Model, name string, compress Compression) Connection {
@@ -152,32 +152,24 @@ func NewConnection(deviceID DeviceID, reader io.Reader, writer io.Writer, receiv
cw := &countingWriter{Writer: writer} cw := &countingWriter{Writer: writer}
c := rawConnection{ c := rawConnection{
id: deviceID, id: deviceID,
name: name, name: name,
receiver: nativeModel{receiver}, receiver: nativeModel{receiver},
cr: cr, state: stateInitial,
cw: cw, cr: cr,
outbox: make(chan hdrMsg), cw: cw,
nextID: make(chan int), outbox: make(chan hdrMsg),
closed: make(chan struct{}), nextID: make(chan int),
pool: sync.Pool{ closed: make(chan struct{}),
New: func() interface{} {
return make([]byte, BlockSize)
},
},
compression: compress, compression: compress,
} }
return wireFormatConnection{&c}
}
// Start creates the goroutines for sending and receiving of messages. It must
// be called exactly once after creating a connection.
func (c *rawConnection) Start() {
go c.readerLoop() go c.readerLoop()
go c.writerLoop() go c.writerLoop()
go c.pingerLoop() go c.pingerLoop()
go c.idGenerator() go c.idGenerator()
return wireFormatConnection{&c}
} }
func (c *rawConnection) ID() DeviceID { func (c *rawConnection) ID() DeviceID {
@@ -201,7 +193,7 @@ func (c *rawConnection) Index(folder string, idx []FileInfo, flags uint32, optio
Files: idx, Files: idx,
Flags: flags, Flags: flags,
Options: options, Options: options,
}, nil) })
c.idxMut.Unlock() c.idxMut.Unlock()
return nil return nil
} }
@@ -219,7 +211,7 @@ func (c *rawConnection) IndexUpdate(folder string, idx []FileInfo, flags uint32,
Files: idx, Files: idx,
Flags: flags, Flags: flags,
Options: options, Options: options,
}, nil) })
c.idxMut.Unlock() c.idxMut.Unlock()
return nil return nil
} }
@@ -249,7 +241,7 @@ func (c *rawConnection) Request(folder string, name string, offset int64, size i
Hash: hash, Hash: hash,
Flags: flags, Flags: flags,
Options: options, Options: options,
}, nil) })
if !ok { if !ok {
return nil, ErrClosed return nil, ErrClosed
} }
@@ -263,7 +255,7 @@ func (c *rawConnection) Request(folder string, name string, offset int64, size i
// ClusterConfig send the cluster configuration message to the peer and returns any error // ClusterConfig send the cluster configuration message to the peer and returns any error
func (c *rawConnection) ClusterConfig(config ClusterConfigMessage) { func (c *rawConnection) ClusterConfig(config ClusterConfigMessage) {
c.send(-1, messageTypeClusterConfig, config, nil) c.send(-1, messageTypeClusterConfig, config)
} }
func (c *rawConnection) ping() bool { func (c *rawConnection) ping() bool {
@@ -279,7 +271,7 @@ func (c *rawConnection) ping() bool {
c.awaiting[id] = rc c.awaiting[id] = rc
c.awaitingMut.Unlock() c.awaitingMut.Unlock()
ok := c.send(id, messageTypePing, nil, nil) ok := c.send(id, messageTypePing, nil)
if !ok { if !ok {
return false return false
} }
@@ -293,7 +285,6 @@ func (c *rawConnection) readerLoop() (err error) {
c.close(err) c.close(err)
}() }()
state := stateInitial
for { for {
select { select {
case <-c.closed: case <-c.closed:
@@ -307,55 +298,48 @@ func (c *rawConnection) readerLoop() (err error) {
} }
switch msg := msg.(type) { switch msg := msg.(type) {
case ClusterConfigMessage:
if state != stateInitial {
return fmt.Errorf("protocol error: cluster config message in state %d", state)
}
go c.receiver.ClusterConfig(c.id, msg)
state = stateReady
case IndexMessage: case IndexMessage:
switch hdr.msgType { switch hdr.msgType {
case messageTypeIndex: case messageTypeIndex:
if state != stateReady { if c.state < stateCCRcvd {
return fmt.Errorf("protocol error: index message in state %d", state) return fmt.Errorf("protocol error: index message in state %d", c.state)
} }
c.handleIndex(msg) c.handleIndex(msg)
state = stateReady c.state = stateIdxRcvd
case messageTypeIndexUpdate: case messageTypeIndexUpdate:
if state != stateReady { if c.state < stateIdxRcvd {
return fmt.Errorf("protocol error: index update message in state %d", state) return fmt.Errorf("protocol error: index update message in state %d", c.state)
} }
c.handleIndexUpdate(msg) c.handleIndexUpdate(msg)
state = stateReady
} }
case RequestMessage: case RequestMessage:
if state != stateReady { if c.state < stateIdxRcvd {
return fmt.Errorf("protocol error: request message in state %d", state) return fmt.Errorf("protocol error: request message in state %d", c.state)
} }
// Requests are handled asynchronously // Requests are handled asynchronously
go c.handleRequest(hdr.msgID, msg) go c.handleRequest(hdr.msgID, msg)
case ResponseMessage: case ResponseMessage:
if state != stateReady { if c.state < stateIdxRcvd {
return fmt.Errorf("protocol error: response message in state %d", state) return fmt.Errorf("protocol error: response message in state %d", c.state)
} }
c.handleResponse(hdr.msgID, msg) c.handleResponse(hdr.msgID, msg)
case pingMessage: case pingMessage:
if state != stateReady { c.send(hdr.msgID, messageTypePong, pongMessage{})
return fmt.Errorf("protocol error: ping message in state %d", state)
}
c.send(hdr.msgID, messageTypePong, pongMessage{}, nil)
case pongMessage: case pongMessage:
if state != stateReady {
return fmt.Errorf("protocol error: pong message in state %d", state)
}
c.handlePong(hdr.msgID) c.handlePong(hdr.msgID)
case ClusterConfigMessage:
if c.state != stateInitial {
return fmt.Errorf("protocol error: cluster config message in state %d", c.state)
}
go c.receiver.ClusterConfig(c.id, msg)
c.state = stateCCRcvd
case CloseMessage: case CloseMessage:
return errors.New(msg.Reason) return errors.New(msg.Reason)
@@ -525,36 +509,12 @@ func filterIndexMessageFiles(fs []FileInfo) []FileInfo {
} }
func (c *rawConnection) handleRequest(msgID int, req RequestMessage) { func (c *rawConnection) handleRequest(msgID int, req RequestMessage) {
size := int(req.Size) data, err := c.receiver.Request(c.id, req.Folder, req.Name, int64(req.Offset), int(req.Size), req.Hash, req.Flags, req.Options)
usePool := size <= BlockSize
var buf []byte c.send(msgID, messageTypeResponse, ResponseMessage{
var done chan struct{} Data: data,
Code: errorToCode(err),
if usePool { })
buf = c.pool.Get().([]byte)[:size]
done = make(chan struct{})
} else {
buf = make([]byte, size)
}
err := c.receiver.Request(c.id, req.Folder, req.Name, int64(req.Offset), req.Hash, req.Flags, req.Options, buf)
if err != nil {
c.send(msgID, messageTypeResponse, ResponseMessage{
Data: nil,
Code: errorToCode(err),
}, done)
} else {
c.send(msgID, messageTypeResponse, ResponseMessage{
Data: buf,
Code: errorToCode(err),
}, done)
}
if usePool {
<-done
c.pool.Put(buf)
}
} }
func (c *rawConnection) handleResponse(msgID int, resp ResponseMessage) { func (c *rawConnection) handleResponse(msgID int, resp ResponseMessage) {
@@ -577,7 +537,7 @@ func (c *rawConnection) handlePong(msgID int) {
c.awaitingMut.Unlock() c.awaitingMut.Unlock()
} }
func (c *rawConnection) send(msgID int, msgType int, msg encodable, done chan struct{}) bool { func (c *rawConnection) send(msgID int, msgType int, msg encodable) bool {
if msgID < 0 { if msgID < 0 {
select { select {
case id := <-c.nextID: case id := <-c.nextID:
@@ -594,7 +554,7 @@ func (c *rawConnection) send(msgID int, msgType int, msg encodable, done chan st
} }
select { select {
case c.outbox <- hdrMsg{hdr, msg, done}: case c.outbox <- hdrMsg{hdr, msg}:
return true return true
case <-c.closed: case <-c.closed:
return false return false
@@ -613,9 +573,6 @@ func (c *rawConnection) writerLoop() {
if hm.msg != nil { if hm.msg != nil {
// Uncompressed message in uncBuf // Uncompressed message in uncBuf
uncBuf, err = hm.msg.AppendXDR(uncBuf[:0]) uncBuf, err = hm.msg.AppendXDR(uncBuf[:0])
if hm.done != nil {
close(hm.done)
}
if err != nil { if err != nil {
c.close(err) c.close(err)
return return
@@ -722,17 +679,17 @@ func (c *rawConnection) idGenerator() {
func (c *rawConnection) pingerLoop() { func (c *rawConnection) pingerLoop() {
var rc = make(chan bool, 1) var rc = make(chan bool, 1)
ticker := time.Tick(PingIdleTime / 2) ticker := time.Tick(pingIdleTime / 2)
for { for {
select { select {
case <-ticker: case <-ticker:
if d := time.Since(c.cr.Last()); d < PingIdleTime { if d := time.Since(c.cr.Last()); d < pingIdleTime {
if debug { if debug {
l.Debugln(c.id, "ping skipped after rd", d) l.Debugln(c.id, "ping skipped after rd", d)
} }
continue continue
} }
if d := time.Since(c.cw.Last()); d < PingIdleTime { if d := time.Since(c.cw.Last()); d < pingIdleTime {
if debug { if debug {
l.Debugln(c.id, "ping skipped after wr", d) l.Debugln(c.id, "ping skipped after wr", d)
} }
@@ -752,7 +709,7 @@ func (c *rawConnection) pingerLoop() {
if !ok { if !ok {
c.close(fmt.Errorf("ping failure")) c.close(fmt.Errorf("ping failure"))
} }
case <-time.After(PingTimeout): case <-time.After(pingTimeout):
c.close(fmt.Errorf("ping timeout")) c.close(fmt.Errorf("ping timeout"))
case <-c.closed: case <-c.closed:
return return
+9 -31
View File
@@ -67,12 +67,8 @@ func TestPing(t *testing.T) {
ar, aw := io.Pipe() ar, aw := io.Pipe()
br, bw := io.Pipe() br, bw := io.Pipe()
c0 := NewConnection(c0ID, ar, bw, newTestModel(), "name", CompressAlways).(wireFormatConnection).next.(*rawConnection) c0 := NewConnection(c0ID, ar, bw, nil, "name", CompressAlways).(wireFormatConnection).next.(*rawConnection)
c0.Start() c1 := NewConnection(c1ID, br, aw, nil, "name", CompressAlways).(wireFormatConnection).next.(*rawConnection)
c1 := NewConnection(c1ID, br, aw, newTestModel(), "name", CompressAlways).(wireFormatConnection).next.(*rawConnection)
c1.Start()
c0.ClusterConfig(ClusterConfigMessage{})
c1.ClusterConfig(ClusterConfigMessage{})
if ok := c0.ping(); !ok { if ok := c0.ping(); !ok {
t.Error("c0 ping failed") t.Error("c0 ping failed")
@@ -85,8 +81,8 @@ func TestPing(t *testing.T) {
func TestPingErr(t *testing.T) { func TestPingErr(t *testing.T) {
e := errors.New("something broke") e := errors.New("something broke")
for i := 0; i < 32; i++ { for i := 0; i < 16; i++ {
for j := 0; j < 32; j++ { for j := 0; j < 16; j++ {
m0 := newTestModel() m0 := newTestModel()
m1 := newTestModel() m1 := newTestModel()
@@ -96,18 +92,12 @@ func TestPingErr(t *testing.T) {
ebw := &ErrPipe{PipeWriter: *bw, max: j, err: e} ebw := &ErrPipe{PipeWriter: *bw, max: j, err: e}
c0 := NewConnection(c0ID, ar, ebw, m0, "name", CompressAlways).(wireFormatConnection).next.(*rawConnection) c0 := NewConnection(c0ID, ar, ebw, m0, "name", CompressAlways).(wireFormatConnection).next.(*rawConnection)
c0.Start() NewConnection(c1ID, br, eaw, m1, "name", CompressAlways)
c1 := NewConnection(c1ID, br, eaw, m1, "name", CompressAlways)
c1.Start()
c0.ClusterConfig(ClusterConfigMessage{})
c1.ClusterConfig(ClusterConfigMessage{})
res := c0.ping() res := c0.ping()
if (i < 8 || j < 8) && res { if (i < 8 || j < 8) && res {
// This should have resulted in failure, as there is no way an empty ClusterConfig plus a Ping message fits in eight bytes.
t.Errorf("Unexpected ping success; i=%d, j=%d", i, j) t.Errorf("Unexpected ping success; i=%d, j=%d", i, j)
} else if (i >= 28 && j >= 28) && !res { } else if (i >= 12 && j >= 12) && !res {
// This should have worked though, as 28 bytes is plenty for both.
t.Errorf("Unexpected ping fail; i=%d, j=%d", i, j) t.Errorf("Unexpected ping fail; i=%d, j=%d", i, j)
} }
} }
@@ -178,11 +168,7 @@ func TestVersionErr(t *testing.T) {
br, bw := io.Pipe() br, bw := io.Pipe()
c0 := NewConnection(c0ID, ar, bw, m0, "name", CompressAlways).(wireFormatConnection).next.(*rawConnection) c0 := NewConnection(c0ID, ar, bw, m0, "name", CompressAlways).(wireFormatConnection).next.(*rawConnection)
c0.Start() NewConnection(c1ID, br, aw, m1, "name", CompressAlways)
c1 := NewConnection(c1ID, br, aw, m1, "name", CompressAlways)
c1.Start()
c0.ClusterConfig(ClusterConfigMessage{})
c1.ClusterConfig(ClusterConfigMessage{})
w := xdr.NewWriter(c0.cw) w := xdr.NewWriter(c0.cw)
w.WriteUint32(encodeHeader(header{ w.WriteUint32(encodeHeader(header{
@@ -205,11 +191,7 @@ func TestTypeErr(t *testing.T) {
br, bw := io.Pipe() br, bw := io.Pipe()
c0 := NewConnection(c0ID, ar, bw, m0, "name", CompressAlways).(wireFormatConnection).next.(*rawConnection) c0 := NewConnection(c0ID, ar, bw, m0, "name", CompressAlways).(wireFormatConnection).next.(*rawConnection)
c0.Start() NewConnection(c1ID, br, aw, m1, "name", CompressAlways)
c1 := NewConnection(c1ID, br, aw, m1, "name", CompressAlways)
c1.Start()
c0.ClusterConfig(ClusterConfigMessage{})
c1.ClusterConfig(ClusterConfigMessage{})
w := xdr.NewWriter(c0.cw) w := xdr.NewWriter(c0.cw)
w.WriteUint32(encodeHeader(header{ w.WriteUint32(encodeHeader(header{
@@ -232,11 +214,7 @@ func TestClose(t *testing.T) {
br, bw := io.Pipe() br, bw := io.Pipe()
c0 := NewConnection(c0ID, ar, bw, m0, "name", CompressAlways).(wireFormatConnection).next.(*rawConnection) c0 := NewConnection(c0ID, ar, bw, m0, "name", CompressAlways).(wireFormatConnection).next.(*rawConnection)
c0.Start() NewConnection(c1ID, br, aw, m1, "name", CompressAlways)
c1 := NewConnection(c1ID, br, aw, m1, "name", CompressAlways)
c1.Start()
c0.ClusterConfig(ClusterConfigMessage{})
c1.ClusterConfig(ClusterConfigMessage{})
c0.close(nil) c0.close(nil)
-4
View File
@@ -12,10 +12,6 @@ type wireFormatConnection struct {
next Connection next Connection
} }
func (c wireFormatConnection) Start() {
c.next.Start()
}
func (c wireFormatConnection) ID() DeviceID { func (c wireFormatConnection) ID() DeviceID {
return c.next.ID() return c.next.ID()
} }
+145 -268
View File
@@ -63,14 +63,13 @@ type DB struct {
journalAckC chan error journalAckC chan error
// Compaction. // Compaction.
tcompCmdC chan cCmd tcompCmdC chan cCmd
tcompPauseC chan chan<- struct{} tcompPauseC chan chan<- struct{}
mcompCmdC chan cCmd mcompCmdC chan cCmd
compErrC chan error compErrC chan error
compPerErrC chan error compPerErrC chan error
compErrSetC chan error compErrSetC chan error
compWriteLocking bool compStats []cStats
compStats []cStats
// Close. // Close.
closeW sync.WaitGroup closeW sync.WaitGroup
@@ -109,44 +108,28 @@ func openDB(s *session) (*DB, error) {
closeC: make(chan struct{}), closeC: make(chan struct{}),
} }
// Read-only mode. if err := db.recoverJournal(); err != nil {
readOnly := s.o.GetReadOnly() return nil, err
}
if readOnly { // Remove any obsolete files.
// Recover journals (read-only mode). if err := db.checkAndCleanFiles(); err != nil {
if err := db.recoverJournalRO(); err != nil { // Close journal.
return nil, err if db.journal != nil {
db.journal.Close()
db.journalWriter.Close()
} }
} else { return nil, err
// Recover journals.
if err := db.recoverJournal(); err != nil {
return nil, err
}
// Remove any obsolete files.
if err := db.checkAndCleanFiles(); err != nil {
// Close journal.
if db.journal != nil {
db.journal.Close()
db.journalWriter.Close()
}
return nil, err
}
} }
// Doesn't need to be included in the wait group. // Doesn't need to be included in the wait group.
go db.compactionError() go db.compactionError()
go db.mpoolDrain() go db.mpoolDrain()
if readOnly { db.closeW.Add(3)
db.SetReadOnly() go db.tCompaction()
} else { go db.mCompaction()
db.closeW.Add(3) go db.jWriter()
go db.tCompaction()
go db.mCompaction()
go db.jWriter()
}
s.logf("db@open done T·%v", time.Since(start)) s.logf("db@open done T·%v", time.Since(start))
@@ -292,7 +275,7 @@ func recoverTable(s *session, o *opt.Options) error {
// We will drop corrupted table. // We will drop corrupted table.
strict = o.GetStrict(opt.StrictRecovery) strict = o.GetStrict(opt.StrictRecovery)
rec = &sessionRecord{} rec = &sessionRecord{numLevel: o.GetNumLevel()}
bpool = util.NewBufferPool(o.GetBlockSize() + 5) bpool = util.NewBufferPool(o.GetBlockSize() + 5)
) )
buildTable := func(iter iterator.Iterator) (tmp storage.File, size int64, err error) { buildTable := func(iter iterator.Iterator) (tmp storage.File, size int64, err error) {
@@ -467,136 +450,132 @@ func recoverTable(s *session, o *opt.Options) error {
} }
func (db *DB) recoverJournal() error { func (db *DB) recoverJournal() error {
// Get all journals and sort it by file number. // Get all tables and sort it by file number.
allJournalFiles, err := db.s.getFiles(storage.TypeJournal) journalFiles_, err := db.s.getFiles(storage.TypeJournal)
if err != nil { if err != nil {
return err return err
} }
files(allJournalFiles).sort() journalFiles := files(journalFiles_)
journalFiles.sort()
// Journals that will be recovered. // Discard older journal.
var recJournalFiles []storage.File prev := -1
for _, jf := range allJournalFiles { for i, file := range journalFiles {
if jf.Num() >= db.s.stJournalNum || jf.Num() == db.s.stPrevJournalNum { if file.Num() >= db.s.stJournalNum {
recJournalFiles = append(recJournalFiles, jf) if prev >= 0 {
i--
journalFiles[i] = journalFiles[prev]
}
journalFiles = journalFiles[i:]
break
} else if file.Num() == db.s.stPrevJournalNum {
prev = i
} }
} }
var ( var jr *journal.Reader
of storage.File // Obsolete file. var of storage.File
rec = &sessionRecord{} var mem *memdb.DB
) batch := new(Batch)
cm := newCMem(db.s)
buf := new(util.Buffer)
// Options.
strict := db.s.o.GetStrict(opt.StrictJournal)
checksum := db.s.o.GetStrict(opt.StrictJournalChecksum)
writeBuffer := db.s.o.GetWriteBuffer()
recoverJournal := func(file storage.File) error {
db.logf("journal@recovery recovering @%d", file.Num())
reader, err := file.Open()
if err != nil {
return err
}
defer reader.Close()
// Recover journals. // Create/reset journal reader instance.
if len(recJournalFiles) > 0 { if jr == nil {
db.logf("journal@recovery F·%d", len(recJournalFiles)) jr = journal.NewReader(reader, dropper{db.s, file}, strict, checksum)
} else {
// Mark file number as used. jr.Reset(reader, dropper{db.s, file}, strict, checksum)
db.s.markFileNum(recJournalFiles[len(recJournalFiles)-1].Num())
var (
// Options.
strict = db.s.o.GetStrict(opt.StrictJournal)
checksum = db.s.o.GetStrict(opt.StrictJournalChecksum)
writeBuffer = db.s.o.GetWriteBuffer()
jr *journal.Reader
mdb = memdb.New(db.s.icmp, writeBuffer)
buf = &util.Buffer{}
batch = &Batch{}
)
for _, jf := range recJournalFiles {
db.logf("journal@recovery recovering @%d", jf.Num())
fr, err := jf.Open()
if err != nil {
return err
}
// Create or reset journal reader instance.
if jr == nil {
jr = journal.NewReader(fr, dropper{db.s, jf}, strict, checksum)
} else {
jr.Reset(fr, dropper{db.s, jf}, strict, checksum)
}
// Flush memdb and remove obsolete journal file.
if of != nil {
if mdb.Len() > 0 {
if _, err := db.s.flushMemdb(rec, mdb, -1); err != nil {
fr.Close()
return err
}
}
rec.setJournalNum(jf.Num())
rec.setSeqNum(db.seq)
if err := db.s.commit(rec); err != nil {
fr.Close()
return err
}
rec.resetAddedTables()
of.Remove()
of = nil
}
// Replay journal to memdb.
mdb.Reset()
for {
r, err := jr.Next()
if err != nil {
if err == io.EOF {
break
}
fr.Close()
return errors.SetFile(err, jf)
}
buf.Reset()
if _, err := buf.ReadFrom(r); err != nil {
if err == io.ErrUnexpectedEOF {
// This is error returned due to corruption, with strict == false.
continue
}
fr.Close()
return errors.SetFile(err, jf)
}
if err := batch.memDecodeAndReplay(db.seq, buf.Bytes(), mdb); err != nil {
if !strict && errors.IsCorrupted(err) {
db.s.logf("journal error: %v (skipped)", err)
// We won't apply sequence number as it might be corrupted.
continue
}
fr.Close()
return errors.SetFile(err, jf)
}
// Save sequence number.
db.seq = batch.seq + uint64(batch.Len())
// Flush it if large enough.
if mdb.Size() >= writeBuffer {
if _, err := db.s.flushMemdb(rec, mdb, 0); err != nil {
fr.Close()
return err
}
mdb.Reset()
}
}
fr.Close()
of = jf
} }
// Flush the last memdb. // Flush memdb and remove obsolete journal file.
if mdb.Len() > 0 { if of != nil {
if _, err := db.s.flushMemdb(rec, mdb, 0); err != nil { if mem.Len() > 0 {
if err := cm.flush(mem, 0); err != nil {
return err
}
}
if err := cm.commit(file.Num(), db.seq); err != nil {
return err
}
cm.reset()
of.Remove()
of = nil
}
// Replay journal to memdb.
mem.Reset()
for {
r, err := jr.Next()
if err != nil {
if err == io.EOF {
break
}
return errors.SetFile(err, file)
}
buf.Reset()
if _, err := buf.ReadFrom(r); err != nil {
if err == io.ErrUnexpectedEOF {
// This is error returned due to corruption, with strict == false.
continue
} else {
return errors.SetFile(err, file)
}
}
if err := batch.memDecodeAndReplay(db.seq, buf.Bytes(), mem); err != nil {
if strict || !errors.IsCorrupted(err) {
return errors.SetFile(err, file)
} else {
db.s.logf("journal error: %v (skipped)", err)
// We won't apply sequence number as it might be corrupted.
continue
}
}
// Save sequence number.
db.seq = batch.seq + uint64(batch.Len())
// Flush it if large enough.
if mem.Size() >= writeBuffer {
if err := cm.flush(mem, 0); err != nil {
return err
}
mem.Reset()
}
}
of = file
return nil
}
// Recover all journals.
if len(journalFiles) > 0 {
db.logf("journal@recovery F·%d", len(journalFiles))
// Mark file number as used.
db.s.markFileNum(journalFiles[len(journalFiles)-1].Num())
mem = memdb.New(db.s.icmp, writeBuffer)
for _, file := range journalFiles {
if err := recoverJournal(file); err != nil {
return err
}
}
// Flush the last journal.
if mem.Len() > 0 {
if err := cm.flush(mem, 0); err != nil {
return err return err
} }
} }
@@ -608,10 +587,8 @@ func (db *DB) recoverJournal() error {
} }
// Commit. // Commit.
rec.setJournalNum(db.journalFile.Num()) if err := cm.commit(db.journalFile.Num(), db.seq); err != nil {
rec.setSeqNum(db.seq) // Close journal.
if err := db.s.commit(rec); err != nil {
// Close journal on error.
if db.journal != nil { if db.journal != nil {
db.journal.Close() db.journal.Close()
db.journalWriter.Close() db.journalWriter.Close()
@@ -627,103 +604,6 @@ func (db *DB) recoverJournal() error {
return nil return nil
} }
func (db *DB) recoverJournalRO() error {
// Get all journals and sort it by file number.
allJournalFiles, err := db.s.getFiles(storage.TypeJournal)
if err != nil {
return err
}
files(allJournalFiles).sort()
// Journals that will be recovered.
var recJournalFiles []storage.File
for _, jf := range allJournalFiles {
if jf.Num() >= db.s.stJournalNum || jf.Num() == db.s.stPrevJournalNum {
recJournalFiles = append(recJournalFiles, jf)
}
}
var (
// Options.
strict = db.s.o.GetStrict(opt.StrictJournal)
checksum = db.s.o.GetStrict(opt.StrictJournalChecksum)
writeBuffer = db.s.o.GetWriteBuffer()
mdb = memdb.New(db.s.icmp, writeBuffer)
)
// Recover journals.
if len(recJournalFiles) > 0 {
db.logf("journal@recovery RO·Mode F·%d", len(recJournalFiles))
var (
jr *journal.Reader
buf = &util.Buffer{}
batch = &Batch{}
)
for _, jf := range recJournalFiles {
db.logf("journal@recovery recovering @%d", jf.Num())
fr, err := jf.Open()
if err != nil {
return err
}
// Create or reset journal reader instance.
if jr == nil {
jr = journal.NewReader(fr, dropper{db.s, jf}, strict, checksum)
} else {
jr.Reset(fr, dropper{db.s, jf}, strict, checksum)
}
// Replay journal to memdb.
for {
r, err := jr.Next()
if err != nil {
if err == io.EOF {
break
}
fr.Close()
return errors.SetFile(err, jf)
}
buf.Reset()
if _, err := buf.ReadFrom(r); err != nil {
if err == io.ErrUnexpectedEOF {
// This is error returned due to corruption, with strict == false.
continue
}
fr.Close()
return errors.SetFile(err, jf)
}
if err := batch.memDecodeAndReplay(db.seq, buf.Bytes(), mdb); err != nil {
if !strict && errors.IsCorrupted(err) {
db.s.logf("journal error: %v (skipped)", err)
// We won't apply sequence number as it might be corrupted.
continue
}
fr.Close()
return errors.SetFile(err, jf)
}
// Save sequence number.
db.seq = batch.seq + uint64(batch.Len())
}
fr.Close()
}
}
// Set memDB.
db.mem = &memDB{db: db, DB: mdb, ref: 1}
return nil
}
func (db *DB) get(key []byte, seq uint64, ro *opt.ReadOptions) (value []byte, err error) { func (db *DB) get(key []byte, seq uint64, ro *opt.ReadOptions) (value []byte, err error) {
ikey := newIkey(key, seq, ktSeek) ikey := newIkey(key, seq, ktSeek)
@@ -734,7 +614,7 @@ func (db *DB) get(key []byte, seq uint64, ro *opt.ReadOptions) (value []byte, er
} }
defer m.decref() defer m.decref()
mk, mv, me := m.Find(ikey) mk, mv, me := m.mdb.Find(ikey)
if me == nil { if me == nil {
ukey, _, kt, kerr := parseIkey(mk) ukey, _, kt, kerr := parseIkey(mk)
if kerr != nil { if kerr != nil {
@@ -772,7 +652,7 @@ func (db *DB) has(key []byte, seq uint64, ro *opt.ReadOptions) (ret bool, err er
} }
defer m.decref() defer m.decref()
mk, _, me := m.Find(ikey) mk, _, me := m.mdb.Find(ikey)
if me == nil { if me == nil {
ukey, _, kt, kerr := parseIkey(mk) ukey, _, kt, kerr := parseIkey(mk)
if kerr != nil { if kerr != nil {
@@ -904,7 +784,7 @@ func (db *DB) GetProperty(name string) (value string, err error) {
const prefix = "leveldb." const prefix = "leveldb."
if !strings.HasPrefix(name, prefix) { if !strings.HasPrefix(name, prefix) {
return "", ErrNotFound return "", errors.New("leveldb: GetProperty: unknown property: " + name)
} }
p := name[len(prefix):] p := name[len(prefix):]
@@ -918,7 +798,7 @@ func (db *DB) GetProperty(name string) (value string, err error) {
var rest string var rest string
n, _ := fmt.Sscanf(p[len(numFilesPrefix):], "%d%s", &level, &rest) n, _ := fmt.Sscanf(p[len(numFilesPrefix):], "%d%s", &level, &rest)
if n != 1 || int(level) >= db.s.o.GetNumLevel() { if n != 1 || int(level) >= db.s.o.GetNumLevel() {
err = ErrNotFound err = errors.New("leveldb: GetProperty: invalid property: " + name)
} else { } else {
value = fmt.Sprint(v.tLen(int(level))) value = fmt.Sprint(v.tLen(int(level)))
} }
@@ -957,7 +837,7 @@ func (db *DB) GetProperty(name string) (value string, err error) {
case p == "aliveiters": case p == "aliveiters":
value = fmt.Sprintf("%d", atomic.LoadInt32(&db.aliveIters)) value = fmt.Sprintf("%d", atomic.LoadInt32(&db.aliveIters))
default: default:
err = ErrNotFound err = errors.New("leveldb: GetProperty: unknown property: " + name)
} }
return return
@@ -1020,9 +900,6 @@ func (db *DB) Close() error {
var err error var err error
select { select {
case err = <-db.compErrC: case err = <-db.compErrC:
if err == ErrReadOnly {
err = nil
}
default: default:
} }
+79 -35
View File
@@ -11,6 +11,7 @@ import (
"time" "time"
"github.com/syndtr/goleveldb/leveldb/errors" "github.com/syndtr/goleveldb/leveldb/errors"
"github.com/syndtr/goleveldb/leveldb/memdb"
"github.com/syndtr/goleveldb/leveldb/opt" "github.com/syndtr/goleveldb/leveldb/opt"
) )
@@ -61,8 +62,58 @@ func (p *cStatsStaging) stopTimer() {
} }
} }
type cMem struct {
s *session
level int
rec *sessionRecord
}
func newCMem(s *session) *cMem {
return &cMem{s: s, rec: &sessionRecord{numLevel: s.o.GetNumLevel()}}
}
func (c *cMem) flush(mem *memdb.DB, level int) error {
s := c.s
// Write memdb to table.
iter := mem.NewIterator(nil)
defer iter.Release()
t, n, err := s.tops.createFrom(iter)
if err != nil {
return err
}
// Pick level.
if level < 0 {
v := s.version()
level = v.pickLevel(t.imin.ukey(), t.imax.ukey())
v.release()
}
c.rec.addTableFile(level, t)
s.logf("mem@flush created L%d@%d N·%d S·%s %q:%q", level, t.file.Num(), n, shortenb(int(t.size)), t.imin, t.imax)
c.level = level
return nil
}
func (c *cMem) reset() {
c.rec = &sessionRecord{numLevel: c.s.o.GetNumLevel()}
}
func (c *cMem) commit(journal, seq uint64) error {
c.rec.setJournalNum(journal)
c.rec.setSeqNum(seq)
// Commit changes.
return c.s.commit(c.rec)
}
func (db *DB) compactionError() { func (db *DB) compactionError() {
var err error var (
err error
wlocked bool
)
noerr: noerr:
// No error. // No error.
for { for {
@@ -70,7 +121,7 @@ noerr:
case err = <-db.compErrSetC: case err = <-db.compErrSetC:
switch { switch {
case err == nil: case err == nil:
case err == ErrReadOnly, errors.IsCorrupted(err): case errors.IsCorrupted(err):
goto hasperr goto hasperr
default: default:
goto haserr goto haserr
@@ -88,7 +139,7 @@ haserr:
switch { switch {
case err == nil: case err == nil:
goto noerr goto noerr
case err == ErrReadOnly, errors.IsCorrupted(err): case errors.IsCorrupted(err):
goto hasperr goto hasperr
default: default:
} }
@@ -104,9 +155,9 @@ hasperr:
case db.compPerErrC <- err: case db.compPerErrC <- err:
case db.writeLockC <- struct{}{}: case db.writeLockC <- struct{}{}:
// Hold write lock, so that write won't pass-through. // Hold write lock, so that write won't pass-through.
db.compWriteLocking = true wlocked = true
case _, _ = <-db.closeC: case _, _ = <-db.closeC:
if db.compWriteLocking { if wlocked {
// We should release the lock or Close will hang. // We should release the lock or Close will hang.
<-db.writeLockC <-db.writeLockC
} }
@@ -236,18 +287,21 @@ func (db *DB) compactionExitTransact() {
} }
func (db *DB) memCompaction() { func (db *DB) memCompaction() {
mdb := db.getFrozenMem() mem := db.getFrozenMem()
if mdb == nil { if mem == nil {
return return
} }
defer mdb.decref() defer mem.decref()
db.logf("memdb@flush N·%d S·%s", mdb.Len(), shortenb(mdb.Size())) c := newCMem(db.s)
stats := new(cStatsStaging)
db.logf("mem@flush N·%d S·%s", mem.mdb.Len(), shortenb(mem.mdb.Size()))
// Don't compact empty memdb. // Don't compact empty memdb.
if mdb.Len() == 0 { if mem.mdb.Len() == 0 {
db.logf("memdb@flush skipping") db.logf("mem@flush skipping")
// drop frozen memdb // drop frozen mem
db.dropFrozenMem() db.dropFrozenMem()
return return
} }
@@ -263,20 +317,13 @@ func (db *DB) memCompaction() {
return return
} }
var ( db.compactionTransactFunc("mem@flush", func(cnt *compactionTransactCounter) (err error) {
rec = &sessionRecord{}
stats = &cStatsStaging{}
flushLevel int
)
db.compactionTransactFunc("memdb@flush", func(cnt *compactionTransactCounter) (err error) {
stats.startTimer() stats.startTimer()
flushLevel, err = db.s.flushMemdb(rec, mdb.DB, -1) defer stats.stopTimer()
stats.stopTimer() return c.flush(mem.mdb, -1)
return
}, func() error { }, func() error {
for _, r := range rec.addedTables { for _, r := range c.rec.addedTables {
db.logf("memdb@flush revert @%d", r.num) db.logf("mem@flush revert @%d", r.num)
f := db.s.getTableFile(r.num) f := db.s.getTableFile(r.num)
if err := f.Remove(); err != nil { if err := f.Remove(); err != nil {
return err return err
@@ -285,23 +332,20 @@ func (db *DB) memCompaction() {
return nil return nil
}) })
db.compactionTransactFunc("memdb@commit", func(cnt *compactionTransactCounter) (err error) { db.compactionTransactFunc("mem@commit", func(cnt *compactionTransactCounter) (err error) {
stats.startTimer() stats.startTimer()
rec.setJournalNum(db.journalFile.Num()) defer stats.stopTimer()
rec.setSeqNum(db.frozenSeq) return c.commit(db.journalFile.Num(), db.frozenSeq)
err = db.s.commit(rec)
stats.stopTimer()
return
}, nil) }, nil)
db.logf("memdb@flush committed F·%d T·%v", len(rec.addedTables), stats.duration) db.logf("mem@flush committed F·%d T·%v", len(c.rec.addedTables), stats.duration)
for _, r := range rec.addedTables { for _, r := range c.rec.addedTables {
stats.write += r.size stats.write += r.size
} }
db.compStats[flushLevel].add(stats) db.compStats[c.level].add(stats)
// Drop frozen memdb. // Drop frozen mem.
db.dropFrozenMem() db.dropFrozenMem()
// Resume table compaction. // Resume table compaction.
@@ -513,7 +557,7 @@ func (b *tableCompactionBuilder) revert() error {
func (db *DB) tableCompaction(c *compaction, noTrivial bool) { func (db *DB) tableCompaction(c *compaction, noTrivial bool) {
defer c.release() defer c.release()
rec := &sessionRecord{} rec := &sessionRecord{numLevel: db.s.o.GetNumLevel()}
rec.addCompPtr(c.level, c.imax) rec.addCompPtr(c.level, c.imax)
if !noTrivial && c.trivial() { if !noTrivial && c.trivial() {
+7 -25
View File
@@ -8,7 +8,6 @@ package leveldb
import ( import (
"errors" "errors"
"math/rand"
"runtime" "runtime"
"sync" "sync"
"sync/atomic" "sync/atomic"
@@ -40,11 +39,11 @@ func (db *DB) newRawIterator(slice *util.Range, ro *opt.ReadOptions) iterator.It
ti := v.getIterators(slice, ro) ti := v.getIterators(slice, ro)
n := len(ti) + 2 n := len(ti) + 2
i := make([]iterator.Iterator, 0, n) i := make([]iterator.Iterator, 0, n)
emi := em.NewIterator(slice) emi := em.mdb.NewIterator(slice)
emi.SetReleaser(&memdbReleaser{m: em}) emi.SetReleaser(&memdbReleaser{m: em})
i = append(i, emi) i = append(i, emi)
if fm != nil { if fm != nil {
fmi := fm.NewIterator(slice) fmi := fm.mdb.NewIterator(slice)
fmi.SetReleaser(&memdbReleaser{m: fm}) fmi.SetReleaser(&memdbReleaser{m: fm})
i = append(i, fmi) i = append(i, fmi)
} }
@@ -81,10 +80,6 @@ func (db *DB) newIterator(seq uint64, slice *util.Range, ro *opt.ReadOptions) *d
return iter return iter
} }
func (db *DB) iterSamplingRate() int {
return rand.Intn(2 * db.s.o.GetIteratorSamplingRate())
}
type dir int type dir int
const ( const (
@@ -103,21 +98,11 @@ type dbIter struct {
seq uint64 seq uint64
strict bool strict bool
smaplingGap int dir dir
dir dir key []byte
key []byte value []byte
value []byte err error
err error releaser util.Releaser
releaser util.Releaser
}
func (i *dbIter) sampleSeek() {
ikey := i.iter.Key()
i.smaplingGap -= len(ikey) + len(i.iter.Value())
for i.smaplingGap < 0 {
i.smaplingGap += i.db.iterSamplingRate()
i.db.sampleSeek(ikey)
}
} }
func (i *dbIter) setErr(err error) { func (i *dbIter) setErr(err error) {
@@ -190,7 +175,6 @@ func (i *dbIter) Seek(key []byte) bool {
func (i *dbIter) next() bool { func (i *dbIter) next() bool {
for { for {
if ukey, seq, kt, kerr := parseIkey(i.iter.Key()); kerr == nil { if ukey, seq, kt, kerr := parseIkey(i.iter.Key()); kerr == nil {
i.sampleSeek()
if seq <= i.seq { if seq <= i.seq {
switch kt { switch kt {
case ktDel: case ktDel:
@@ -241,7 +225,6 @@ func (i *dbIter) prev() bool {
if i.iter.Valid() { if i.iter.Valid() {
for { for {
if ukey, seq, kt, kerr := parseIkey(i.iter.Key()); kerr == nil { if ukey, seq, kt, kerr := parseIkey(i.iter.Key()); kerr == nil {
i.sampleSeek()
if seq <= i.seq { if seq <= i.seq {
if !del && i.icmp.uCompare(ukey, i.key) < 0 { if !del && i.icmp.uCompare(ukey, i.key) < 0 {
return true return true
@@ -283,7 +266,6 @@ func (i *dbIter) Prev() bool {
case dirForward: case dirForward:
for i.iter.Prev() { for i.iter.Prev() {
if ukey, _, _, kerr := parseIkey(i.iter.Key()); kerr == nil { if ukey, _, _, kerr := parseIkey(i.iter.Key()); kerr == nil {
i.sampleSeek()
if i.icmp.uCompare(ukey, i.key) < 0 { if i.icmp.uCompare(ukey, i.key) < 0 {
goto cont goto cont
} }
+7 -16
View File
@@ -15,8 +15,8 @@ import (
) )
type memDB struct { type memDB struct {
db *DB db *DB
*memdb.DB mdb *memdb.DB
ref int32 ref int32
} }
@@ -27,12 +27,12 @@ func (m *memDB) incref() {
func (m *memDB) decref() { func (m *memDB) decref() {
if ref := atomic.AddInt32(&m.ref, -1); ref == 0 { if ref := atomic.AddInt32(&m.ref, -1); ref == 0 {
// Only put back memdb with std capacity. // Only put back memdb with std capacity.
if m.Capacity() == m.db.s.o.GetWriteBuffer() { if m.mdb.Capacity() == m.db.s.o.GetWriteBuffer() {
m.Reset() m.mdb.Reset()
m.db.mpoolPut(m.DB) m.db.mpoolPut(m.mdb)
} }
m.db = nil m.db = nil
m.DB = nil m.mdb = nil
} else if ref < 0 { } else if ref < 0 {
panic("negative memdb ref") panic("negative memdb ref")
} }
@@ -48,15 +48,6 @@ func (db *DB) addSeq(delta uint64) {
atomic.AddUint64(&db.seq, delta) atomic.AddUint64(&db.seq, delta)
} }
func (db *DB) sampleSeek(ikey iKey) {
v := db.s.version()
if v.sampleSeek(ikey) {
// Trigger table compaction.
db.compSendTrigger(db.tcompCmdC)
}
v.release()
}
func (db *DB) mpoolPut(mem *memdb.DB) { func (db *DB) mpoolPut(mem *memdb.DB) {
defer func() { defer func() {
recover() recover()
@@ -126,7 +117,7 @@ func (db *DB) newMem(n int) (mem *memDB, err error) {
} }
mem = &memDB{ mem = &memDB{
db: db, db: db,
DB: mdb, mdb: mdb,
ref: 2, ref: 2,
} }
db.mem = mem db.mem = mem
+13 -135
View File
@@ -405,21 +405,19 @@ func (h *dbHarness) compactRange(min, max string) {
t.Log("DB range compaction done") t.Log("DB range compaction done")
} }
func (h *dbHarness) sizeOf(start, limit string) uint64 { func (h *dbHarness) sizeAssert(start, limit string, low, hi uint64) {
sz, err := h.db.SizeOf([]util.Range{ t := h.t
db := h.db
s, err := db.SizeOf([]util.Range{
{[]byte(start), []byte(limit)}, {[]byte(start), []byte(limit)},
}) })
if err != nil { if err != nil {
h.t.Error("SizeOf: got error: ", err) t.Error("SizeOf: got error: ", err)
} }
return sz.Sum() if s.Sum() < low || s.Sum() > hi {
} t.Errorf("sizeof %q to %q not in range, want %d - %d, got %d",
shorten(start), shorten(limit), low, hi, s.Sum())
func (h *dbHarness) sizeAssert(start, limit string, low, hi uint64) {
sz := h.sizeOf(start, limit)
if sz < low || sz > hi {
h.t.Errorf("sizeOf %q to %q not in range, want %d - %d, got %d",
shorten(start), shorten(limit), low, hi, sz)
} }
} }
@@ -2445,7 +2443,7 @@ func TestDB_TableCompactionBuilder(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
rec := &sessionRecord{} rec := &sessionRecord{numLevel: s.o.GetNumLevel()}
rec.addTableFile(i, tf) rec.addTableFile(i, tf)
if err := s.commit(rec); err != nil { if err := s.commit(rec); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -2455,7 +2453,7 @@ func TestDB_TableCompactionBuilder(t *testing.T) {
// Build grandparent. // Build grandparent.
v := s.version() v := s.version()
c := newCompaction(s, v, 1, append(tFiles{}, v.tables[1]...)) c := newCompaction(s, v, 1, append(tFiles{}, v.tables[1]...))
rec := &sessionRecord{} rec := &sessionRecord{numLevel: s.o.GetNumLevel()}
b := &tableCompactionBuilder{ b := &tableCompactionBuilder{
s: s, s: s,
c: c, c: c,
@@ -2479,7 +2477,7 @@ func TestDB_TableCompactionBuilder(t *testing.T) {
// Build level-1. // Build level-1.
v = s.version() v = s.version()
c = newCompaction(s, v, 0, append(tFiles{}, v.tables[0]...)) c = newCompaction(s, v, 0, append(tFiles{}, v.tables[0]...))
rec = &sessionRecord{} rec = &sessionRecord{numLevel: s.o.GetNumLevel()}
b = &tableCompactionBuilder{ b = &tableCompactionBuilder{
s: s, s: s,
c: c, c: c,
@@ -2523,7 +2521,7 @@ func TestDB_TableCompactionBuilder(t *testing.T) {
// Compaction with transient error. // Compaction with transient error.
v = s.version() v = s.version()
c = newCompaction(s, v, 1, append(tFiles{}, v.tables[1]...)) c = newCompaction(s, v, 1, append(tFiles{}, v.tables[1]...))
rec = &sessionRecord{} rec = &sessionRecord{numLevel: s.o.GetNumLevel()}
b = &tableCompactionBuilder{ b = &tableCompactionBuilder{
s: s, s: s,
c: c, c: c,
@@ -2579,123 +2577,3 @@ func TestDB_TableCompactionBuilder(t *testing.T) {
} }
v.release() v.release()
} }
func testDB_IterTriggeredCompaction(t *testing.T, limitDiv int) {
const (
vSize = 200 * opt.KiB
tSize = 100 * opt.MiB
mIter = 100
n = tSize / vSize
)
h := newDbHarnessWopt(t, &opt.Options{
Compression: opt.NoCompression,
DisableBlockCache: true,
})
defer h.close()
key := func(x int) string {
return fmt.Sprintf("v%06d", x)
}
// Fill.
value := strings.Repeat("x", vSize)
for i := 0; i < n; i++ {
h.put(key(i), value)
}
h.compactMem()
// Delete all.
for i := 0; i < n; i++ {
h.delete(key(i))
}
h.compactMem()
var (
limit = n / limitDiv
startKey = key(0)
limitKey = key(limit)
maxKey = key(n)
slice = &util.Range{Limit: []byte(limitKey)}
initialSize0 = h.sizeOf(startKey, limitKey)
initialSize1 = h.sizeOf(limitKey, maxKey)
)
t.Logf("inital size %s [rest %s]", shortenb(int(initialSize0)), shortenb(int(initialSize1)))
for r := 0; true; r++ {
if r >= mIter {
t.Fatal("taking too long to compact")
}
// Iterates.
iter := h.db.NewIterator(slice, h.ro)
for iter.Next() {
}
if err := iter.Error(); err != nil {
t.Fatalf("Iter err: %v", err)
}
iter.Release()
// Wait compaction.
h.waitCompaction()
// Check size.
size0 := h.sizeOf(startKey, limitKey)
size1 := h.sizeOf(limitKey, maxKey)
t.Logf("#%03d size %s [rest %s]", r, shortenb(int(size0)), shortenb(int(size1)))
if size0 < initialSize0/10 {
break
}
}
if initialSize1 > 0 {
h.sizeAssert(limitKey, maxKey, initialSize1/4-opt.MiB, initialSize1+opt.MiB)
}
}
func TestDB_IterTriggeredCompaction(t *testing.T) {
testDB_IterTriggeredCompaction(t, 1)
}
func TestDB_IterTriggeredCompactionHalf(t *testing.T) {
testDB_IterTriggeredCompaction(t, 2)
}
func TestDB_ReadOnly(t *testing.T) {
h := newDbHarness(t)
defer h.close()
h.put("foo", "v1")
h.put("bar", "v2")
h.compactMem()
h.put("xfoo", "v1")
h.put("xbar", "v2")
t.Log("Trigger read-only")
if err := h.db.SetReadOnly(); err != nil {
h.close()
t.Fatalf("SetReadOnly error: %v", err)
}
h.stor.SetEmuErr(storage.TypeAll, tsOpCreate, tsOpReplace, tsOpRemove, tsOpWrite, tsOpWrite, tsOpSync)
ro := func(key, value, wantValue string) {
if err := h.db.Put([]byte(key), []byte(value), h.wo); err != ErrReadOnly {
t.Fatalf("unexpected error: %v", err)
}
h.getVal(key, wantValue)
}
ro("foo", "vx", "v1")
h.o.ReadOnly = true
h.reopenDB()
ro("foo", "vx", "v1")
ro("bar", "vx", "v2")
h.assertNumKeys(4)
}
+24 -51
View File
@@ -63,24 +63,24 @@ func (db *DB) rotateMem(n int) (mem *memDB, err error) {
return return
} }
func (db *DB) flush(n int) (mdb *memDB, mdbFree int, err error) { func (db *DB) flush(n int) (mem *memDB, nn int, err error) {
delayed := false delayed := false
flush := func() (retry bool) { flush := func() (retry bool) {
v := db.s.version() v := db.s.version()
defer v.release() defer v.release()
mdb = db.getEffectiveMem() mem = db.getEffectiveMem()
defer func() { defer func() {
if retry { if retry {
mdb.decref() mem.decref()
mdb = nil mem = nil
} }
}() }()
mdbFree = mdb.Free() nn = mem.mdb.Free()
switch { switch {
case v.tLen(0) >= db.s.o.GetWriteL0SlowdownTrigger() && !delayed: case v.tLen(0) >= db.s.o.GetWriteL0SlowdownTrigger() && !delayed:
delayed = true delayed = true
time.Sleep(time.Millisecond) time.Sleep(time.Millisecond)
case mdbFree >= n: case nn >= n:
return false return false
case v.tLen(0) >= db.s.o.GetWriteL0PauseTrigger(): case v.tLen(0) >= db.s.o.GetWriteL0PauseTrigger():
delayed = true delayed = true
@@ -90,15 +90,15 @@ func (db *DB) flush(n int) (mdb *memDB, mdbFree int, err error) {
} }
default: default:
// Allow memdb to grow if it has no entry. // Allow memdb to grow if it has no entry.
if mdb.Len() == 0 { if mem.mdb.Len() == 0 {
mdbFree = n nn = n
} else { } else {
mdb.decref() mem.decref()
mdb, err = db.rotateMem(n) mem, err = db.rotateMem(n)
if err == nil { if err == nil {
mdbFree = mdb.Free() nn = mem.mdb.Free()
} else { } else {
mdbFree = 0 nn = 0
} }
} }
return false return false
@@ -157,18 +157,18 @@ func (db *DB) Write(b *Batch, wo *opt.WriteOptions) (err error) {
} }
}() }()
mdb, mdbFree, err := db.flush(b.size()) mem, memFree, err := db.flush(b.size())
if err != nil { if err != nil {
return return
} }
defer mdb.decref() defer mem.decref()
// Calculate maximum size of the batch. // Calculate maximum size of the batch.
m := 1 << 20 m := 1 << 20
if x := b.size(); x <= 128<<10 { if x := b.size(); x <= 128<<10 {
m = x + (128 << 10) m = x + (128 << 10)
} }
m = minInt(m, mdbFree) m = minInt(m, memFree)
// Merge with other batch. // Merge with other batch.
drain: drain:
@@ -197,7 +197,7 @@ drain:
select { select {
case db.journalC <- b: case db.journalC <- b:
// Write into memdb // Write into memdb
if berr := b.memReplay(mdb.DB); berr != nil { if berr := b.memReplay(mem.mdb); berr != nil {
panic(berr) panic(berr)
} }
case err = <-db.compPerErrC: case err = <-db.compPerErrC:
@@ -211,7 +211,7 @@ drain:
case err = <-db.journalAckC: case err = <-db.journalAckC:
if err != nil { if err != nil {
// Revert memdb if error detected // Revert memdb if error detected
if berr := b.revertMemReplay(mdb.DB); berr != nil { if berr := b.revertMemReplay(mem.mdb); berr != nil {
panic(berr) panic(berr)
} }
return return
@@ -225,7 +225,7 @@ drain:
if err != nil { if err != nil {
return return
} }
if berr := b.memReplay(mdb.DB); berr != nil { if berr := b.memReplay(mem.mdb); berr != nil {
panic(berr) panic(berr)
} }
} }
@@ -233,7 +233,7 @@ drain:
// Set last seq number. // Set last seq number.
db.addSeq(uint64(b.Len())) db.addSeq(uint64(b.Len()))
if b.size() >= mdbFree { if b.size() >= memFree {
db.rotateMem(0) db.rotateMem(0)
} }
return return
@@ -249,7 +249,8 @@ func (db *DB) Put(key, value []byte, wo *opt.WriteOptions) error {
return db.Write(b, wo) return db.Write(b, wo)
} }
// Delete deletes the value for the given key. // Delete deletes the value for the given key. It returns ErrNotFound if
// the DB does not contain the key.
// //
// It is safe to modify the contents of the arguments after Delete returns. // It is safe to modify the contents of the arguments after Delete returns.
func (db *DB) Delete(key []byte, wo *opt.WriteOptions) error { func (db *DB) Delete(key []byte, wo *opt.WriteOptions) error {
@@ -289,9 +290,9 @@ func (db *DB) CompactRange(r util.Range) error {
} }
// Check for overlaps in memdb. // Check for overlaps in memdb.
mdb := db.getEffectiveMem() mem := db.getEffectiveMem()
defer mdb.decref() defer mem.decref()
if isMemOverlaps(db.s.icmp, mdb.DB, r.Start, r.Limit) { if isMemOverlaps(db.s.icmp, mem.mdb, r.Start, r.Limit) {
// Memdb compaction. // Memdb compaction.
if _, err := db.rotateMem(0); err != nil { if _, err := db.rotateMem(0); err != nil {
<-db.writeLockC <-db.writeLockC
@@ -308,31 +309,3 @@ func (db *DB) CompactRange(r util.Range) error {
// Table compaction. // Table compaction.
return db.compSendRange(db.tcompCmdC, -1, r.Start, r.Limit) return db.compSendRange(db.tcompCmdC, -1, r.Start, r.Limit)
} }
// SetReadOnly makes DB read-only. It will stay read-only until reopened.
func (db *DB) SetReadOnly() error {
if err := db.ok(); err != nil {
return err
}
// Lock writer.
select {
case db.writeLockC <- struct{}{}:
db.compWriteLocking = true
case err := <-db.compPerErrC:
return err
case _, _ = <-db.closeC:
return ErrClosed
}
// Set compaction read-only.
select {
case db.compErrSetC <- ErrReadOnly:
case perr := <-db.compPerErrC:
return perr
case _, _ = <-db.closeC:
return ErrClosed
}
return nil
}
-1
View File
@@ -12,7 +12,6 @@ import (
var ( var (
ErrNotFound = errors.ErrNotFound ErrNotFound = errors.ErrNotFound
ErrReadOnly = errors.New("leveldb: read-only mode")
ErrSnapshotReleased = errors.New("leveldb: snapshot released") ErrSnapshotReleased = errors.New("leveldb: snapshot released")
ErrIterReleased = errors.New("leveldb: iterator released") ErrIterReleased = errors.New("leveldb: iterator released")
ErrClosed = errors.New("leveldb: closed") ErrClosed = errors.New("leveldb: closed")
+3 -6
View File
@@ -206,7 +206,6 @@ func (p *DB) randHeight() (h int) {
return return
} }
// Must hold RW-lock if prev == true, as it use shared prevNode slice.
func (p *DB) findGE(key []byte, prev bool) (int, bool) { func (p *DB) findGE(key []byte, prev bool) (int, bool) {
node := 0 node := 0
h := p.maxHeight - 1 h := p.maxHeight - 1
@@ -303,7 +302,7 @@ func (p *DB) Put(key []byte, value []byte) error {
node := len(p.nodeData) node := len(p.nodeData)
p.nodeData = append(p.nodeData, kvOffset, len(key), len(value), h) p.nodeData = append(p.nodeData, kvOffset, len(key), len(value), h)
for i, n := range p.prevNode[:h] { for i, n := range p.prevNode[:h] {
m := n + nNext + i m := n + 4 + i
p.nodeData = append(p.nodeData, p.nodeData[m]) p.nodeData = append(p.nodeData, p.nodeData[m])
p.nodeData[m] = node p.nodeData[m] = node
} }
@@ -435,22 +434,20 @@ func (p *DB) Len() int {
// Reset resets the DB to initial empty state. Allows reuse the buffer. // Reset resets the DB to initial empty state. Allows reuse the buffer.
func (p *DB) Reset() { func (p *DB) Reset() {
p.mu.Lock()
p.rnd = rand.New(rand.NewSource(0xdeadbeef)) p.rnd = rand.New(rand.NewSource(0xdeadbeef))
p.maxHeight = 1 p.maxHeight = 1
p.n = 0 p.n = 0
p.kvSize = 0 p.kvSize = 0
p.kvData = p.kvData[:0] p.kvData = p.kvData[:0]
p.nodeData = p.nodeData[:nNext+tMaxHeight] p.nodeData = p.nodeData[:4+tMaxHeight]
p.nodeData[nKV] = 0 p.nodeData[nKV] = 0
p.nodeData[nKey] = 0 p.nodeData[nKey] = 0
p.nodeData[nVal] = 0 p.nodeData[nVal] = 0
p.nodeData[nHeight] = tMaxHeight p.nodeData[nHeight] = tMaxHeight
for n := 0; n < tMaxHeight; n++ { for n := 0; n < tMaxHeight; n++ {
p.nodeData[nNext+n] = 0 p.nodeData[4+n] = 0
p.prevNode[n] = 0 p.prevNode[n] = 0
} }
p.mu.Unlock()
} }
// New creates a new initalized in-memory key/value DB. The capacity // New creates a new initalized in-memory key/value DB. The capacity
+2 -48
View File
@@ -34,11 +34,10 @@ var (
DefaultCompactionTotalSize = 10 * MiB DefaultCompactionTotalSize = 10 * MiB
DefaultCompactionTotalSizeMultiplier = 10.0 DefaultCompactionTotalSizeMultiplier = 10.0
DefaultCompressionType = SnappyCompression DefaultCompressionType = SnappyCompression
DefaultIteratorSamplingRate = 1 * MiB
DefaultMaxMemCompationLevel = 2
DefaultNumLevel = 7
DefaultOpenFilesCacher = LRUCacher DefaultOpenFilesCacher = LRUCacher
DefaultOpenFilesCacheCapacity = 500 DefaultOpenFilesCacheCapacity = 500
DefaultMaxMemCompationLevel = 2
DefaultNumLevel = 7
DefaultWriteBuffer = 4 * MiB DefaultWriteBuffer = 4 * MiB
DefaultWriteL0PauseTrigger = 12 DefaultWriteL0PauseTrigger = 12
DefaultWriteL0SlowdownTrigger = 8 DefaultWriteL0SlowdownTrigger = 8
@@ -250,11 +249,6 @@ type Options struct {
// The default value (DefaultCompression) uses snappy compression. // The default value (DefaultCompression) uses snappy compression.
Compression Compression Compression Compression
// DisableBufferPool allows disable use of util.BufferPool functionality.
//
// The default value is false.
DisableBufferPool bool
// DisableBlockCache allows disable use of cache.Cache functionality on // DisableBlockCache allows disable use of cache.Cache functionality on
// 'sorted table' block. // 'sorted table' block.
// //
@@ -294,13 +288,6 @@ type Options struct {
// The default value is nil. // The default value is nil.
Filter filter.Filter Filter filter.Filter
// IteratorSamplingRate defines approximate gap (in bytes) between read
// sampling of an iterator. The samples will be used to determine when
// compaction should be triggered.
//
// The default is 1MiB.
IteratorSamplingRate int
// MaxMemCompationLevel defines maximum level a newly compacted 'memdb' // MaxMemCompationLevel defines maximum level a newly compacted 'memdb'
// will be pushed into if doesn't creates overlap. This should less than // will be pushed into if doesn't creates overlap. This should less than
// NumLevel. Use -1 for level-0. // NumLevel. Use -1 for level-0.
@@ -326,11 +313,6 @@ type Options struct {
// The default value is 500. // The default value is 500.
OpenFilesCacheCapacity int OpenFilesCacheCapacity int
// If true then opens DB in read-only mode.
//
// The default value is false.
ReadOnly bool
// Strict defines the DB strict level. // Strict defines the DB strict level.
Strict Strict Strict Strict
@@ -482,20 +464,6 @@ func (o *Options) GetCompression() Compression {
return o.Compression return o.Compression
} }
func (o *Options) GetDisableBufferPool() bool {
if o == nil {
return false
}
return o.DisableBufferPool
}
func (o *Options) GetDisableBlockCache() bool {
if o == nil {
return false
}
return o.DisableBlockCache
}
func (o *Options) GetDisableCompactionBackoff() bool { func (o *Options) GetDisableCompactionBackoff() bool {
if o == nil { if o == nil {
return false return false
@@ -524,13 +492,6 @@ func (o *Options) GetFilter() filter.Filter {
return o.Filter return o.Filter
} }
func (o *Options) GetIteratorSamplingRate() int {
if o == nil || o.IteratorSamplingRate <= 0 {
return DefaultIteratorSamplingRate
}
return o.IteratorSamplingRate
}
func (o *Options) GetMaxMemCompationLevel() int { func (o *Options) GetMaxMemCompationLevel() int {
level := DefaultMaxMemCompationLevel level := DefaultMaxMemCompationLevel
if o != nil { if o != nil {
@@ -572,13 +533,6 @@ func (o *Options) GetOpenFilesCacheCapacity() int {
return o.OpenFilesCacheCapacity return o.OpenFilesCacheCapacity
} }
func (o *Options) GetReadOnly() bool {
if o == nil {
return false
}
return o.ReadOnly
}
func (o *Options) GetStrict(strict Strict) bool { func (o *Options) GetStrict(strict Strict) bool {
if o == nil || o.Strict == 0 { if o == nil || o.Strict == 0 {
return DefaultStrict&strict != 0 return DefaultStrict&strict != 0
+254 -10
View File
@@ -11,8 +11,10 @@ import (
"io" "io"
"os" "os"
"sync" "sync"
"sync/atomic"
"github.com/syndtr/goleveldb/leveldb/errors" "github.com/syndtr/goleveldb/leveldb/errors"
"github.com/syndtr/goleveldb/leveldb/iterator"
"github.com/syndtr/goleveldb/leveldb/journal" "github.com/syndtr/goleveldb/leveldb/journal"
"github.com/syndtr/goleveldb/leveldb/opt" "github.com/syndtr/goleveldb/leveldb/opt"
"github.com/syndtr/goleveldb/leveldb/storage" "github.com/syndtr/goleveldb/leveldb/storage"
@@ -125,16 +127,11 @@ func (s *session) recover() (err error) {
return return
} }
defer reader.Close() defer reader.Close()
strict := s.o.GetStrict(opt.StrictManifest)
jr := journal.NewReader(reader, dropper{s, m}, strict, true)
var ( staging := s.stVersion.newStaging()
// Options. rec := &sessionRecord{numLevel: s.o.GetNumLevel()}
numLevel = s.o.GetNumLevel()
strict = s.o.GetStrict(opt.StrictManifest)
jr = journal.NewReader(reader, dropper{s, m}, strict, true)
rec = &sessionRecord{}
staging = s.stVersion.newStaging()
)
for { for {
var r io.Reader var r io.Reader
r, err = jr.Next() r, err = jr.Next()
@@ -146,7 +143,7 @@ func (s *session) recover() (err error) {
return errors.SetFile(err, m) return errors.SetFile(err, m)
} }
err = rec.decode(r, numLevel) err = rec.decode(r)
if err == nil { if err == nil {
// save compact pointers // save compact pointers
for _, r := range rec.compPtrs { for _, r := range rec.compPtrs {
@@ -209,3 +206,250 @@ func (s *session) commit(r *sessionRecord) (err error) {
return return
} }
// Pick a compaction based on current state; need external synchronization.
func (s *session) pickCompaction() *compaction {
v := s.version()
var level int
var t0 tFiles
if v.cScore >= 1 {
level = v.cLevel
cptr := s.stCompPtrs[level]
tables := v.tables[level]
for _, t := range tables {
if cptr == nil || s.icmp.Compare(t.imax, cptr) > 0 {
t0 = append(t0, t)
break
}
}
if len(t0) == 0 {
t0 = append(t0, tables[0])
}
} else {
if p := atomic.LoadPointer(&v.cSeek); p != nil {
ts := (*tSet)(p)
level = ts.level
t0 = append(t0, ts.table)
} else {
v.release()
return nil
}
}
return newCompaction(s, v, level, t0)
}
// Create compaction from given level and range; need external synchronization.
func (s *session) getCompactionRange(level int, umin, umax []byte) *compaction {
v := s.version()
t0 := v.tables[level].getOverlaps(nil, s.icmp, umin, umax, level == 0)
if len(t0) == 0 {
v.release()
return nil
}
// Avoid compacting too much in one shot in case the range is large.
// But we cannot do this for level-0 since level-0 files can overlap
// and we must not pick one file and drop another older file if the
// two files overlap.
if level > 0 {
limit := uint64(v.s.o.GetCompactionSourceLimit(level))
total := uint64(0)
for i, t := range t0 {
total += t.size
if total >= limit {
s.logf("table@compaction limiting F·%d -> F·%d", len(t0), i+1)
t0 = t0[:i+1]
break
}
}
}
return newCompaction(s, v, level, t0)
}
func newCompaction(s *session, v *version, level int, t0 tFiles) *compaction {
c := &compaction{
s: s,
v: v,
level: level,
tables: [2]tFiles{t0, nil},
maxGPOverlaps: uint64(s.o.GetCompactionGPOverlaps(level)),
tPtrs: make([]int, s.o.GetNumLevel()),
}
c.expand()
c.save()
return c
}
// compaction represent a compaction state.
type compaction struct {
s *session
v *version
level int
tables [2]tFiles
maxGPOverlaps uint64
gp tFiles
gpi int
seenKey bool
gpOverlappedBytes uint64
imin, imax iKey
tPtrs []int
released bool
snapGPI int
snapSeenKey bool
snapGPOverlappedBytes uint64
snapTPtrs []int
}
func (c *compaction) save() {
c.snapGPI = c.gpi
c.snapSeenKey = c.seenKey
c.snapGPOverlappedBytes = c.gpOverlappedBytes
c.snapTPtrs = append(c.snapTPtrs[:0], c.tPtrs...)
}
func (c *compaction) restore() {
c.gpi = c.snapGPI
c.seenKey = c.snapSeenKey
c.gpOverlappedBytes = c.snapGPOverlappedBytes
c.tPtrs = append(c.tPtrs[:0], c.snapTPtrs...)
}
func (c *compaction) release() {
if !c.released {
c.released = true
c.v.release()
}
}
// Expand compacted tables; need external synchronization.
func (c *compaction) expand() {
limit := uint64(c.s.o.GetCompactionExpandLimit(c.level))
vt0, vt1 := c.v.tables[c.level], c.v.tables[c.level+1]
t0, t1 := c.tables[0], c.tables[1]
imin, imax := t0.getRange(c.s.icmp)
// We expand t0 here just incase ukey hop across tables.
t0 = vt0.getOverlaps(t0, c.s.icmp, imin.ukey(), imax.ukey(), c.level == 0)
if len(t0) != len(c.tables[0]) {
imin, imax = t0.getRange(c.s.icmp)
}
t1 = vt1.getOverlaps(t1, c.s.icmp, imin.ukey(), imax.ukey(), false)
// Get entire range covered by compaction.
amin, amax := append(t0, t1...).getRange(c.s.icmp)
// See if we can grow the number of inputs in "level" without
// changing the number of "level+1" files we pick up.
if len(t1) > 0 {
exp0 := vt0.getOverlaps(nil, c.s.icmp, amin.ukey(), amax.ukey(), c.level == 0)
if len(exp0) > len(t0) && t1.size()+exp0.size() < limit {
xmin, xmax := exp0.getRange(c.s.icmp)
exp1 := vt1.getOverlaps(nil, c.s.icmp, xmin.ukey(), xmax.ukey(), false)
if len(exp1) == len(t1) {
c.s.logf("table@compaction expanding L%d+L%d (F·%d S·%s)+(F·%d S·%s) -> (F·%d S·%s)+(F·%d S·%s)",
c.level, c.level+1, len(t0), shortenb(int(t0.size())), len(t1), shortenb(int(t1.size())),
len(exp0), shortenb(int(exp0.size())), len(exp1), shortenb(int(exp1.size())))
imin, imax = xmin, xmax
t0, t1 = exp0, exp1
amin, amax = append(t0, t1...).getRange(c.s.icmp)
}
}
}
// Compute the set of grandparent files that overlap this compaction
// (parent == level+1; grandparent == level+2)
if c.level+2 < c.s.o.GetNumLevel() {
c.gp = c.v.tables[c.level+2].getOverlaps(c.gp, c.s.icmp, amin.ukey(), amax.ukey(), false)
}
c.tables[0], c.tables[1] = t0, t1
c.imin, c.imax = imin, imax
}
// Check whether compaction is trivial.
func (c *compaction) trivial() bool {
return len(c.tables[0]) == 1 && len(c.tables[1]) == 0 && c.gp.size() <= c.maxGPOverlaps
}
func (c *compaction) baseLevelForKey(ukey []byte) bool {
for level, tables := range c.v.tables[c.level+2:] {
for c.tPtrs[level] < len(tables) {
t := tables[c.tPtrs[level]]
if c.s.icmp.uCompare(ukey, t.imax.ukey()) <= 0 {
// We've advanced far enough.
if c.s.icmp.uCompare(ukey, t.imin.ukey()) >= 0 {
// Key falls in this file's range, so definitely not base level.
return false
}
break
}
c.tPtrs[level]++
}
}
return true
}
func (c *compaction) shouldStopBefore(ikey iKey) bool {
for ; c.gpi < len(c.gp); c.gpi++ {
gp := c.gp[c.gpi]
if c.s.icmp.Compare(ikey, gp.imax) <= 0 {
break
}
if c.seenKey {
c.gpOverlappedBytes += gp.size
}
}
c.seenKey = true
if c.gpOverlappedBytes > c.maxGPOverlaps {
// Too much overlap for current output; start new output.
c.gpOverlappedBytes = 0
return true
}
return false
}
// Creates an iterator.
func (c *compaction) newIterator() iterator.Iterator {
// Creates iterator slice.
icap := len(c.tables)
if c.level == 0 {
// Special case for level-0
icap = len(c.tables[0]) + 1
}
its := make([]iterator.Iterator, 0, icap)
// Options.
ro := &opt.ReadOptions{
DontFillCache: true,
Strict: opt.StrictOverride,
}
strict := c.s.o.GetStrict(opt.StrictCompaction)
if strict {
ro.Strict |= opt.StrictReader
}
for i, tables := range c.tables {
if len(tables) == 0 {
continue
}
// Level-0 is not sorted and may overlaps each other.
if c.level+i == 0 {
for _, t := range tables {
its = append(its, c.s.tops.newIterator(t, nil, ro))
}
} else {
it := iterator.NewIndexedIterator(tables.newIndexIterator(c.s.tops, c.s.icmp, nil, ro), strict)
its = append(its, it)
}
}
return iterator.NewMergedIterator(its, c.s.icmp, strict)
}
@@ -1,287 +0,0 @@
// Copyright (c) 2012, Suryandaru Triandana <syndtr@gmail.com>
// All rights reserved.
//
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
package leveldb
import (
"sync/atomic"
"github.com/syndtr/goleveldb/leveldb/iterator"
"github.com/syndtr/goleveldb/leveldb/memdb"
"github.com/syndtr/goleveldb/leveldb/opt"
)
func (s *session) pickMemdbLevel(umin, umax []byte) int {
v := s.version()
defer v.release()
return v.pickMemdbLevel(umin, umax)
}
func (s *session) flushMemdb(rec *sessionRecord, mdb *memdb.DB, level int) (level_ int, err error) {
// Create sorted table.
iter := mdb.NewIterator(nil)
defer iter.Release()
t, n, err := s.tops.createFrom(iter)
if err != nil {
return level, err
}
// Pick level and add to record.
if level < 0 {
level = s.pickMemdbLevel(t.imin.ukey(), t.imax.ukey())
}
rec.addTableFile(level, t)
s.logf("memdb@flush created L%d@%d N·%d S·%s %q:%q", level, t.file.Num(), n, shortenb(int(t.size)), t.imin, t.imax)
return level, nil
}
// Pick a compaction based on current state; need external synchronization.
func (s *session) pickCompaction() *compaction {
v := s.version()
var level int
var t0 tFiles
if v.cScore >= 1 {
level = v.cLevel
cptr := s.stCompPtrs[level]
tables := v.tables[level]
for _, t := range tables {
if cptr == nil || s.icmp.Compare(t.imax, cptr) > 0 {
t0 = append(t0, t)
break
}
}
if len(t0) == 0 {
t0 = append(t0, tables[0])
}
} else {
if p := atomic.LoadPointer(&v.cSeek); p != nil {
ts := (*tSet)(p)
level = ts.level
t0 = append(t0, ts.table)
} else {
v.release()
return nil
}
}
return newCompaction(s, v, level, t0)
}
// Create compaction from given level and range; need external synchronization.
func (s *session) getCompactionRange(level int, umin, umax []byte) *compaction {
v := s.version()
t0 := v.tables[level].getOverlaps(nil, s.icmp, umin, umax, level == 0)
if len(t0) == 0 {
v.release()
return nil
}
// Avoid compacting too much in one shot in case the range is large.
// But we cannot do this for level-0 since level-0 files can overlap
// and we must not pick one file and drop another older file if the
// two files overlap.
if level > 0 {
limit := uint64(v.s.o.GetCompactionSourceLimit(level))
total := uint64(0)
for i, t := range t0 {
total += t.size
if total >= limit {
s.logf("table@compaction limiting F·%d -> F·%d", len(t0), i+1)
t0 = t0[:i+1]
break
}
}
}
return newCompaction(s, v, level, t0)
}
func newCompaction(s *session, v *version, level int, t0 tFiles) *compaction {
c := &compaction{
s: s,
v: v,
level: level,
tables: [2]tFiles{t0, nil},
maxGPOverlaps: uint64(s.o.GetCompactionGPOverlaps(level)),
tPtrs: make([]int, s.o.GetNumLevel()),
}
c.expand()
c.save()
return c
}
// compaction represent a compaction state.
type compaction struct {
s *session
v *version
level int
tables [2]tFiles
maxGPOverlaps uint64
gp tFiles
gpi int
seenKey bool
gpOverlappedBytes uint64
imin, imax iKey
tPtrs []int
released bool
snapGPI int
snapSeenKey bool
snapGPOverlappedBytes uint64
snapTPtrs []int
}
func (c *compaction) save() {
c.snapGPI = c.gpi
c.snapSeenKey = c.seenKey
c.snapGPOverlappedBytes = c.gpOverlappedBytes
c.snapTPtrs = append(c.snapTPtrs[:0], c.tPtrs...)
}
func (c *compaction) restore() {
c.gpi = c.snapGPI
c.seenKey = c.snapSeenKey
c.gpOverlappedBytes = c.snapGPOverlappedBytes
c.tPtrs = append(c.tPtrs[:0], c.snapTPtrs...)
}
func (c *compaction) release() {
if !c.released {
c.released = true
c.v.release()
}
}
// Expand compacted tables; need external synchronization.
func (c *compaction) expand() {
limit := uint64(c.s.o.GetCompactionExpandLimit(c.level))
vt0, vt1 := c.v.tables[c.level], c.v.tables[c.level+1]
t0, t1 := c.tables[0], c.tables[1]
imin, imax := t0.getRange(c.s.icmp)
// We expand t0 here just incase ukey hop across tables.
t0 = vt0.getOverlaps(t0, c.s.icmp, imin.ukey(), imax.ukey(), c.level == 0)
if len(t0) != len(c.tables[0]) {
imin, imax = t0.getRange(c.s.icmp)
}
t1 = vt1.getOverlaps(t1, c.s.icmp, imin.ukey(), imax.ukey(), false)
// Get entire range covered by compaction.
amin, amax := append(t0, t1...).getRange(c.s.icmp)
// See if we can grow the number of inputs in "level" without
// changing the number of "level+1" files we pick up.
if len(t1) > 0 {
exp0 := vt0.getOverlaps(nil, c.s.icmp, amin.ukey(), amax.ukey(), c.level == 0)
if len(exp0) > len(t0) && t1.size()+exp0.size() < limit {
xmin, xmax := exp0.getRange(c.s.icmp)
exp1 := vt1.getOverlaps(nil, c.s.icmp, xmin.ukey(), xmax.ukey(), false)
if len(exp1) == len(t1) {
c.s.logf("table@compaction expanding L%d+L%d (F·%d S·%s)+(F·%d S·%s) -> (F·%d S·%s)+(F·%d S·%s)",
c.level, c.level+1, len(t0), shortenb(int(t0.size())), len(t1), shortenb(int(t1.size())),
len(exp0), shortenb(int(exp0.size())), len(exp1), shortenb(int(exp1.size())))
imin, imax = xmin, xmax
t0, t1 = exp0, exp1
amin, amax = append(t0, t1...).getRange(c.s.icmp)
}
}
}
// Compute the set of grandparent files that overlap this compaction
// (parent == level+1; grandparent == level+2)
if c.level+2 < c.s.o.GetNumLevel() {
c.gp = c.v.tables[c.level+2].getOverlaps(c.gp, c.s.icmp, amin.ukey(), amax.ukey(), false)
}
c.tables[0], c.tables[1] = t0, t1
c.imin, c.imax = imin, imax
}
// Check whether compaction is trivial.
func (c *compaction) trivial() bool {
return len(c.tables[0]) == 1 && len(c.tables[1]) == 0 && c.gp.size() <= c.maxGPOverlaps
}
func (c *compaction) baseLevelForKey(ukey []byte) bool {
for level, tables := range c.v.tables[c.level+2:] {
for c.tPtrs[level] < len(tables) {
t := tables[c.tPtrs[level]]
if c.s.icmp.uCompare(ukey, t.imax.ukey()) <= 0 {
// We've advanced far enough.
if c.s.icmp.uCompare(ukey, t.imin.ukey()) >= 0 {
// Key falls in this file's range, so definitely not base level.
return false
}
break
}
c.tPtrs[level]++
}
}
return true
}
func (c *compaction) shouldStopBefore(ikey iKey) bool {
for ; c.gpi < len(c.gp); c.gpi++ {
gp := c.gp[c.gpi]
if c.s.icmp.Compare(ikey, gp.imax) <= 0 {
break
}
if c.seenKey {
c.gpOverlappedBytes += gp.size
}
}
c.seenKey = true
if c.gpOverlappedBytes > c.maxGPOverlaps {
// Too much overlap for current output; start new output.
c.gpOverlappedBytes = 0
return true
}
return false
}
// Creates an iterator.
func (c *compaction) newIterator() iterator.Iterator {
// Creates iterator slice.
icap := len(c.tables)
if c.level == 0 {
// Special case for level-0.
icap = len(c.tables[0]) + 1
}
its := make([]iterator.Iterator, 0, icap)
// Options.
ro := &opt.ReadOptions{
DontFillCache: true,
Strict: opt.StrictOverride,
}
strict := c.s.o.GetStrict(opt.StrictCompaction)
if strict {
ro.Strict |= opt.StrictReader
}
for i, tables := range c.tables {
if len(tables) == 0 {
continue
}
// Level-0 is not sorted and may overlaps each other.
if c.level+i == 0 {
for _, t := range tables {
its = append(its, c.s.tops.newIterator(t, nil, ro))
}
} else {
it := iterator.NewIndexedIterator(tables.newIndexIterator(c.s.tops, c.s.icmp, nil, ro), strict)
its = append(its, it)
}
}
return iterator.NewMergedIterator(its, c.s.icmp, strict)
}
@@ -52,6 +52,8 @@ type dtRecord struct {
} }
type sessionRecord struct { type sessionRecord struct {
numLevel int
hasRec int hasRec int
comparer string comparer string
journalNum uint64 journalNum uint64
@@ -228,7 +230,7 @@ func (p *sessionRecord) readBytes(field string, r byteReader) []byte {
return x return x
} }
func (p *sessionRecord) readLevel(field string, r io.ByteReader, numLevel int) int { func (p *sessionRecord) readLevel(field string, r io.ByteReader) int {
if p.err != nil { if p.err != nil {
return 0 return 0
} }
@@ -236,14 +238,14 @@ func (p *sessionRecord) readLevel(field string, r io.ByteReader, numLevel int) i
if p.err != nil { if p.err != nil {
return 0 return 0
} }
if x >= uint64(numLevel) { if x >= uint64(p.numLevel) {
p.err = errors.NewErrCorrupted(nil, &ErrManifestCorrupted{field, "invalid level number"}) p.err = errors.NewErrCorrupted(nil, &ErrManifestCorrupted{field, "invalid level number"})
return 0 return 0
} }
return int(x) return int(x)
} }
func (p *sessionRecord) decode(r io.Reader, numLevel int) error { func (p *sessionRecord) decode(r io.Reader) error {
br, ok := r.(byteReader) br, ok := r.(byteReader)
if !ok { if !ok {
br = bufio.NewReader(r) br = bufio.NewReader(r)
@@ -284,13 +286,13 @@ func (p *sessionRecord) decode(r io.Reader, numLevel int) error {
p.setSeqNum(x) p.setSeqNum(x)
} }
case recCompPtr: case recCompPtr:
level := p.readLevel("comp-ptr.level", br, numLevel) level := p.readLevel("comp-ptr.level", br)
ikey := p.readBytes("comp-ptr.ikey", br) ikey := p.readBytes("comp-ptr.ikey", br)
if p.err == nil { if p.err == nil {
p.addCompPtr(level, iKey(ikey)) p.addCompPtr(level, iKey(ikey))
} }
case recAddTable: case recAddTable:
level := p.readLevel("add-table.level", br, numLevel) level := p.readLevel("add-table.level", br)
num := p.readUvarint("add-table.num", br) num := p.readUvarint("add-table.num", br)
size := p.readUvarint("add-table.size", br) size := p.readUvarint("add-table.size", br)
imin := p.readBytes("add-table.imin", br) imin := p.readBytes("add-table.imin", br)
@@ -299,7 +301,7 @@ func (p *sessionRecord) decode(r io.Reader, numLevel int) error {
p.addTable(level, num, size, imin, imax) p.addTable(level, num, size, imin, imax)
} }
case recDelTable: case recDelTable:
level := p.readLevel("del-table.level", br, numLevel) level := p.readLevel("del-table.level", br)
num := p.readUvarint("del-table.num", br) num := p.readUvarint("del-table.num", br)
if p.err == nil { if p.err == nil {
p.delTable(level, num) p.delTable(level, num)
@@ -19,8 +19,8 @@ func decodeEncode(v *sessionRecord) (res bool, err error) {
if err != nil { if err != nil {
return return
} }
v2 := &sessionRecord{} v2 := &sessionRecord{numLevel: opt.DefaultNumLevel}
err = v.decode(b, opt.DefaultNumLevel) err = v.decode(b)
if err != nil { if err != nil {
return return
} }
@@ -34,7 +34,7 @@ func decodeEncode(v *sessionRecord) (res bool, err error) {
func TestSessionRecord_EncodeDecode(t *testing.T) { func TestSessionRecord_EncodeDecode(t *testing.T) {
big := uint64(1) << 50 big := uint64(1) << 50
v := &sessionRecord{} v := &sessionRecord{numLevel: opt.DefaultNumLevel}
i := uint64(0) i := uint64(0)
test := func() { test := func() {
res, err := decodeEncode(v) res, err := decodeEncode(v)
+1 -1
View File
@@ -182,7 +182,7 @@ func (s *session) newManifest(rec *sessionRecord, v *version) (err error) {
defer v.release() defer v.release()
} }
if rec == nil { if rec == nil {
rec = &sessionRecord{} rec = &sessionRecord{numLevel: s.o.GetNumLevel()}
} }
s.fillRecord(rec, true) s.fillRecord(rec, true)
v.fillRecord(rec) v.fillRecord(rec)
@@ -42,8 +42,6 @@ type tsOp uint
const ( const (
tsOpOpen tsOp = iota tsOpOpen tsOp = iota
tsOpCreate tsOpCreate
tsOpReplace
tsOpRemove
tsOpRead tsOpRead
tsOpReadAt tsOpReadAt
tsOpWrite tsOpWrite
@@ -243,10 +241,6 @@ func (tf tsFile) Replace(newfile storage.File) (err error) {
if err != nil { if err != nil {
return return
} }
if tf.shouldErr(tsOpReplace) {
err = errors.New("leveldb.testStorage: emulated create error")
return
}
err = tf.File.Replace(newfile.(tsFile).File) err = tf.File.Replace(newfile.(tsFile).File)
if err != nil { if err != nil {
ts.t.Errorf("E: cannot replace file, num=%d type=%v: %v", tf.Num(), tf.Type(), err) ts.t.Errorf("E: cannot replace file, num=%d type=%v: %v", tf.Num(), tf.Type(), err)
@@ -264,10 +258,6 @@ func (tf tsFile) Remove() (err error) {
if err != nil { if err != nil {
return return
} }
if tf.shouldErr(tsOpRemove) {
err = errors.New("leveldb.testStorage: emulated create error")
return
}
err = tf.File.Remove() err = tf.File.Remove()
if err != nil { if err != nil {
ts.t.Errorf("E: cannot remove file, num=%d type=%v: %v", tf.Num(), tf.Type(), err) ts.t.Errorf("E: cannot remove file, num=%d type=%v: %v", tf.Num(), tf.Type(), err)
+2 -6
View File
@@ -441,26 +441,22 @@ func newTableOps(s *session) *tOps {
var ( var (
cacher cache.Cacher cacher cache.Cacher
bcache *cache.Cache bcache *cache.Cache
bpool *util.BufferPool
) )
if s.o.GetOpenFilesCacheCapacity() > 0 { if s.o.GetOpenFilesCacheCapacity() > 0 {
cacher = cache.NewLRU(s.o.GetOpenFilesCacheCapacity()) cacher = cache.NewLRU(s.o.GetOpenFilesCacheCapacity())
} }
if !s.o.GetDisableBlockCache() { if !s.o.DisableBlockCache {
var bcacher cache.Cacher var bcacher cache.Cacher
if s.o.GetBlockCacheCapacity() > 0 { if s.o.GetBlockCacheCapacity() > 0 {
bcacher = cache.NewLRU(s.o.GetBlockCacheCapacity()) bcacher = cache.NewLRU(s.o.GetBlockCacheCapacity())
} }
bcache = cache.NewCache(bcacher) bcache = cache.NewCache(bcacher)
} }
if !s.o.GetDisableBufferPool() {
bpool = util.NewBufferPool(s.o.GetBlockSize() + 5)
}
return &tOps{ return &tOps{
s: s, s: s,
cache: cache.NewCache(cacher), cache: cache.NewCache(cacher),
bcache: bcache, bcache: bcache,
bpool: bpool, bpool: util.NewBufferPool(s.o.GetBlockSize() + 5),
} }
} }
+1 -1
View File
@@ -14,7 +14,7 @@ import (
"strings" "strings"
"sync" "sync"
"github.com/golang/snappy" "github.com/syndtr/gosnappy/snappy"
"github.com/syndtr/goleveldb/leveldb/cache" "github.com/syndtr/goleveldb/leveldb/cache"
"github.com/syndtr/goleveldb/leveldb/comparer" "github.com/syndtr/goleveldb/leveldb/comparer"
+6 -2
View File
@@ -12,7 +12,7 @@ import (
"fmt" "fmt"
"io" "io"
"github.com/golang/snappy" "github.com/syndtr/gosnappy/snappy"
"github.com/syndtr/goleveldb/leveldb/comparer" "github.com/syndtr/goleveldb/leveldb/comparer"
"github.com/syndtr/goleveldb/leveldb/filter" "github.com/syndtr/goleveldb/leveldb/filter"
@@ -167,7 +167,11 @@ func (w *Writer) writeBlock(buf *util.Buffer, compression opt.Compression) (bh b
if n := snappy.MaxEncodedLen(buf.Len()) + blockTrailerLen; len(w.compressionScratch) < n { if n := snappy.MaxEncodedLen(buf.Len()) + blockTrailerLen; len(w.compressionScratch) < n {
w.compressionScratch = make([]byte, n) w.compressionScratch = make([]byte, n)
} }
compressed := snappy.Encode(w.compressionScratch, buf.Bytes()) var compressed []byte
compressed, err = snappy.Encode(w.compressionScratch, buf.Bytes())
if err != nil {
return
}
n := len(compressed) n := len(compressed)
b = compressed[:n+blockTrailerLen] b = compressed[:n+blockTrailerLen]
b[n] = blockTypeSnappyCompression b[n] = blockTypeSnappyCompression
+3 -24
View File
@@ -136,8 +136,9 @@ func (v *version) get(ikey iKey, ro *opt.ReadOptions, noValue bool) (value []byt
if !tseek { if !tseek {
if tset == nil { if tset == nil {
tset = &tSet{level, t} tset = &tSet{level, t}
} else { } else if tset.table.consumeSeek() <= 0 {
tseek = true tseek = true
tcomp = atomic.CompareAndSwapPointer(&v.cSeek, nil, unsafe.Pointer(tset))
} }
} }
@@ -202,28 +203,6 @@ func (v *version) get(ikey iKey, ro *opt.ReadOptions, noValue bool) (value []byt
return true return true
}) })
if tseek && tset.table.consumeSeek() <= 0 {
tcomp = atomic.CompareAndSwapPointer(&v.cSeek, nil, unsafe.Pointer(tset))
}
return
}
func (v *version) sampleSeek(ikey iKey) (tcomp bool) {
var tset *tSet
v.walkOverlapping(ikey, func(level int, t *tFile) bool {
if tset == nil {
tset = &tSet{level, t}
return true
} else {
if tset.table.consumeSeek() <= 0 {
tcomp = atomic.CompareAndSwapPointer(&v.cSeek, nil, unsafe.Pointer(tset))
}
return false
}
}, nil)
return return
} }
@@ -300,7 +279,7 @@ func (v *version) offsetOf(ikey iKey) (n uint64, err error) {
return return
} }
func (v *version) pickMemdbLevel(umin, umax []byte) (level int) { func (v *version) pickLevel(umin, umax []byte) (level int) {
if !v.tables[0].overlaps(v.s.icmp, umin, umax, true) { if !v.tables[0].overlaps(v.s.icmp, umin, umax, true) {
var overlaps tFiles var overlaps tFiles
maxLevel := v.s.o.GetMaxMemCompationLevel() maxLevel := v.s.o.GetMaxMemCompationLevel()
@@ -27,7 +27,7 @@ func DecodedLen(src []byte) (int, error) {
// that the length header occupied. // that the length header occupied.
func decodedLen(src []byte) (blockLen, headerLen int, err error) { func decodedLen(src []byte) (blockLen, headerLen int, err error) {
v, n := binary.Uvarint(src) v, n := binary.Uvarint(src)
if n <= 0 { if n == 0 {
return 0, 0, ErrCorrupt return 0, 0, ErrCorrupt
} }
if uint64(int(v)) != v { if uint64(int(v)) != v {
@@ -56,7 +56,7 @@ func Decode(dst, src []byte) ([]byte, error) {
x := uint(src[s] >> 2) x := uint(src[s] >> 2)
switch { switch {
case x < 60: case x < 60:
s++ s += 1
case x == 60: case x == 60:
s += 2 s += 2
if s > len(src) { if s > len(src) {
@@ -130,7 +130,7 @@ func Decode(dst, src []byte) ([]byte, error) {
// NewReader returns a new Reader that decompresses from r, using the framing // NewReader returns a new Reader that decompresses from r, using the framing
// format described at // format described at
// https://github.com/google/snappy/blob/master/framing_format.txt // https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
func NewReader(r io.Reader) *Reader { func NewReader(r io.Reader) *Reader {
return &Reader{ return &Reader{
r: r, r: r,
@@ -200,7 +200,7 @@ func (r *Reader) Read(p []byte) (int, error) {
} }
// The chunk types are specified at // The chunk types are specified at
// https://github.com/google/snappy/blob/master/framing_format.txt // https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
switch chunkType { switch chunkType {
case chunkTypeCompressedData: case chunkTypeCompressedData:
// Section 4.2. Compressed data (chunk type 0x00). // Section 4.2. Compressed data (chunk type 0x00).
@@ -280,11 +280,13 @@ func (r *Reader) Read(p []byte) (int, error) {
// Section 4.5. Reserved unskippable chunks (chunk types 0x02-0x7f). // Section 4.5. Reserved unskippable chunks (chunk types 0x02-0x7f).
r.err = ErrUnsupported r.err = ErrUnsupported
return 0, r.err return 0, r.err
}
// Section 4.4 Padding (chunk type 0xfe). } else {
// Section 4.6. Reserved skippable chunks (chunk types 0x80-0xfd). // Section 4.4 Padding (chunk type 0xfe).
if !r.readFull(r.buf[:chunkLen]) { // Section 4.6. Reserved skippable chunks (chunk types 0x80-0xfd).
return 0, r.err if !r.readFull(r.buf[:chunkLen]) {
return 0, r.err
}
} }
} }
} }
@@ -79,7 +79,7 @@ func emitCopy(dst []byte, offset, length int) int {
// slice of dst if dst was large enough to hold the entire encoded block. // slice of dst if dst was large enough to hold the entire encoded block.
// Otherwise, a newly allocated slice will be returned. // Otherwise, a newly allocated slice will be returned.
// It is valid to pass a nil dst. // It is valid to pass a nil dst.
func Encode(dst, src []byte) []byte { func Encode(dst, src []byte) ([]byte, error) {
if n := MaxEncodedLen(len(src)); len(dst) < n { if n := MaxEncodedLen(len(src)); len(dst) < n {
dst = make([]byte, n) dst = make([]byte, n)
} }
@@ -92,7 +92,7 @@ func Encode(dst, src []byte) []byte {
if len(src) != 0 { if len(src) != 0 {
d += emitLiteral(dst[d:], src) d += emitLiteral(dst[d:], src)
} }
return dst[:d] return dst[:d], nil
} }
// Initialize the hash table. Its size ranges from 1<<8 to 1<<14 inclusive. // Initialize the hash table. Its size ranges from 1<<8 to 1<<14 inclusive.
@@ -145,7 +145,7 @@ func Encode(dst, src []byte) []byte {
if lit != len(src) { if lit != len(src) {
d += emitLiteral(dst[d:], src[lit:]) d += emitLiteral(dst[d:], src[lit:])
} }
return dst[:d] return dst[:d], nil
} }
// MaxEncodedLen returns the maximum length of a snappy block, given its // MaxEncodedLen returns the maximum length of a snappy block, given its
@@ -176,7 +176,7 @@ func MaxEncodedLen(srcLen int) int {
// NewWriter returns a new Writer that compresses to w, using the framing // NewWriter returns a new Writer that compresses to w, using the framing
// format described at // format described at
// https://github.com/google/snappy/blob/master/framing_format.txt // https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
func NewWriter(w io.Writer) *Writer { func NewWriter(w io.Writer) *Writer {
return &Writer{ return &Writer{
w: w, w: w,
@@ -226,7 +226,11 @@ func (w *Writer) Write(p []byte) (n int, errRet error) {
// Compress the buffer, discarding the result if the improvement // Compress the buffer, discarding the result if the improvement
// isn't at least 12.5%. // isn't at least 12.5%.
chunkType := uint8(chunkTypeCompressedData) chunkType := uint8(chunkTypeCompressedData)
chunkBody := Encode(w.enc, uncompressed) chunkBody, err := Encode(w.enc, uncompressed)
if err != nil {
w.err = err
return n, err
}
if len(chunkBody) >= len(uncompressed)-len(uncompressed)/8 { if len(chunkBody) >= len(uncompressed)-len(uncompressed)/8 {
chunkType, chunkBody = chunkTypeUncompressedData, uncompressed chunkType, chunkBody = chunkTypeUncompressedData, uncompressed
} }
@@ -240,11 +244,11 @@ func (w *Writer) Write(p []byte) (n int, errRet error) {
w.buf[5] = uint8(checksum >> 8) w.buf[5] = uint8(checksum >> 8)
w.buf[6] = uint8(checksum >> 16) w.buf[6] = uint8(checksum >> 16)
w.buf[7] = uint8(checksum >> 24) w.buf[7] = uint8(checksum >> 24)
if _, err := w.w.Write(w.buf[:]); err != nil { if _, err = w.w.Write(w.buf[:]); err != nil {
w.err = err w.err = err
return n, err return n, err
} }
if _, err := w.w.Write(chunkBody); err != nil { if _, err = w.w.Write(chunkBody); err != nil {
w.err = err w.err = err
return n, err return n, err
} }
@@ -5,7 +5,7 @@
// Package snappy implements the snappy block-based compression format. // Package snappy implements the snappy block-based compression format.
// It aims for very high speeds and reasonable compression. // It aims for very high speeds and reasonable compression.
// //
// The C++ snappy implementation is at https://github.com/google/snappy // The C++ snappy implementation is at http://code.google.com/p/snappy/
package snappy package snappy
import ( import (
@@ -46,7 +46,7 @@ const (
chunkHeaderSize = 4 chunkHeaderSize = 4
magicChunk = "\xff\x06\x00\x00" + magicBody magicChunk = "\xff\x06\x00\x00" + magicBody
magicBody = "sNaPpY" magicBody = "sNaPpY"
// https://github.com/google/snappy/blob/master/framing_format.txt says // https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt says
// that "the uncompressed data in a chunk must be no longer than 65536 bytes". // that "the uncompressed data in a chunk must be no longer than 65536 bytes".
maxUncompressedChunkLen = 65536 maxUncompressedChunkLen = 65536
) )
@@ -61,7 +61,7 @@ const (
var crcTable = crc32.MakeTable(crc32.Castagnoli) var crcTable = crc32.MakeTable(crc32.Castagnoli)
// crc implements the checksum specified in section 3 of // crc implements the checksum specified in section 3 of
// https://github.com/google/snappy/blob/master/framing_format.txt // https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
func crc(b []byte) uint32 { func crc(b []byte) uint32 {
c := crc32.Update(0, crcTable, b) c := crc32.Update(0, crcTable, b)
return uint32(c>>15|c<<17) + 0xa282ead8 return uint32(c>>15|c<<17) + 0xa282ead8
@@ -24,7 +24,11 @@ var (
) )
func roundtrip(b, ebuf, dbuf []byte) error { func roundtrip(b, ebuf, dbuf []byte) error {
d, err := Decode(dbuf, Encode(ebuf, b)) e, err := Encode(ebuf, b)
if err != nil {
return fmt.Errorf("encoding error: %v", err)
}
d, err := Decode(dbuf, e)
if err != nil { if err != nil {
return fmt.Errorf("decoding error: %v", err) return fmt.Errorf("decoding error: %v", err)
} }
@@ -78,16 +82,6 @@ func TestSmallRegular(t *testing.T) {
} }
} }
func TestInvalidVarint(t *testing.T) {
data := []byte("\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00")
if _, err := DecodedLen(data); err != ErrCorrupt {
t.Errorf("DecodedLen: got %v, want ErrCorrupt", err)
}
if _, err := Decode(nil, data); err != ErrCorrupt {
t.Errorf("Decode: got %v, want ErrCorrupt", err)
}
}
func cmp(a, b []byte) error { func cmp(a, b []byte) error {
if len(a) != len(b) { if len(a) != len(b) {
return fmt.Errorf("got %d bytes, want %d", len(a), len(b)) return fmt.Errorf("got %d bytes, want %d", len(a), len(b))
@@ -203,7 +197,10 @@ func TestWriterReset(t *testing.T) {
} }
func benchDecode(b *testing.B, src []byte) { func benchDecode(b *testing.B, src []byte) {
encoded := Encode(nil, src) encoded, err := Encode(nil, src)
if err != nil {
b.Fatal(err)
}
// Bandwidth is in amount of uncompressed data. // Bandwidth is in amount of uncompressed data.
b.SetBytes(int64(len(src))) b.SetBytes(int64(len(src)))
b.ResetTimer() b.ResetTimer()
@@ -225,7 +222,7 @@ func benchEncode(b *testing.B, src []byte) {
func readFile(b testing.TB, filename string) []byte { func readFile(b testing.TB, filename string) []byte {
src, err := ioutil.ReadFile(filename) src, err := ioutil.ReadFile(filename)
if err != nil { if err != nil {
b.Skipf("skipping benchmark: %v", err) b.Fatalf("failed reading %s: %s", filename, err)
} }
if len(src) == 0 { if len(src) == 0 {
b.Fatalf("%s has zero length", filename) b.Fatalf("%s has zero length", filename)
@@ -287,14 +284,14 @@ var testFiles = []struct {
// The test data files are present at this canonical URL. // The test data files are present at this canonical URL.
const baseURL = "https://raw.githubusercontent.com/google/snappy/master/testdata/" const baseURL = "https://raw.githubusercontent.com/google/snappy/master/testdata/"
func downloadTestdata(b *testing.B, basename string) (errRet error) { func downloadTestdata(basename string) (errRet error) {
filename := filepath.Join(*testdata, basename) filename := filepath.Join(*testdata, basename)
if stat, err := os.Stat(filename); err == nil && stat.Size() != 0 { if stat, err := os.Stat(filename); err == nil && stat.Size() != 0 {
return nil return nil
} }
if !*download { if !*download {
b.Skipf("test data not found; skipping benchmark without the -download flag") return fmt.Errorf("test data not found; skipping benchmark without the -download flag")
} }
// Download the official snappy C++ implementation reference test data // Download the official snappy C++ implementation reference test data
// files for benchmarking. // files for benchmarking.
@@ -329,7 +326,7 @@ func downloadTestdata(b *testing.B, basename string) (errRet error) {
} }
func benchFile(b *testing.B, n int, decode bool) { func benchFile(b *testing.B, n int, decode bool) {
if err := downloadTestdata(b, testFiles[n].filename); err != nil { if err := downloadTestdata(testFiles[n].filename); err != nil {
b.Fatalf("failed to download testdata: %s", err) b.Fatalf("failed to download testdata: %s", err)
} }
data := readFile(b, filepath.Join(*testdata, testFiles[n].filename)) data := readFile(b, filepath.Join(*testdata, testFiles[n].filename))
+1 -1
View File
@@ -1,4 +1,4 @@
Copyright (c) 2014-2015 Barracuda Networks, Inc. Copyright (c) 2014 Barracuda Networks, Inc.
Permission is hereby granted, free of charge, to any person obtaining a copy Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal of this software and associated documentation files (the "Software"), to deal
+9 -30
View File
@@ -36,13 +36,6 @@ to your Supervisor. Supervisors are also services, so you can create a
tree structure here, depending on the exact combination of restarts tree structure here, depending on the exact combination of restarts
you want to create. you want to create.
As a special case, when adding Supervisors to Supervisors, the "sub"
supervisor will have the "super" supervisor's Log function copied.
This allows you to set one log function on the "top" supervisor, and
have it propagate down to all the sub-supervisors. This also allows
libraries or modules to provide Supervisors without having to commit
their users to a particular logging method.
Finally, as what is probably the last line of your main() function, call Finally, as what is probably the last line of your main() function, call
.Serve() on your top level supervisor. This will start all the services .Serve() on your top level supervisor. This will start all the services
you've defined. you've defined.
@@ -133,10 +126,8 @@ type Supervisor struct {
// If you ever come up with some need to get into these, submit a pull // If you ever come up with some need to get into these, submit a pull
// request to make them public and some smidge of justification, and // request to make them public and some smidge of justification, and
// I'll happily do it. // I'll happily do it.
// But since I've now changed the signature on these once, I'm glad I logBadStop func(Service)
// didn't start with them public... :) logFailure func(service Service, currentFailures float64, failureThreshold float64, restarting bool, error interface{}, stacktrace []byte)
logBadStop func(*Supervisor, Service)
logFailure func(supervisor *Supervisor, service Service, currentFailures float64, failureThreshold float64, restarting bool, error interface{}, stacktrace []byte)
logBackoff func(*Supervisor, bool) logBackoff func(*Supervisor, bool)
// avoid a dependency on github.com/thejerf/abtime by just implementing // avoid a dependency on github.com/thejerf/abtime by just implementing
@@ -242,10 +233,10 @@ func New(name string, spec Spec) (s *Supervisor) {
s.resumeTimer = make(chan time.Time) s.resumeTimer = make(chan time.Time)
// set up the default logging handlers // set up the default logging handlers
s.logBadStop = func(supervisor *Supervisor, service Service) { s.logBadStop = func(service Service) {
s.log(fmt.Sprintf("%s: Service %s failed to terminate in a timely manner", serviceName(supervisor), serviceName(service))) s.log(fmt.Sprintf("Service %s failed to terminate in a timely manner", serviceName(service)))
} }
s.logFailure = func(supervisor *Supervisor, service Service, failures float64, threshold float64, restarting bool, err interface{}, st []byte) { s.logFailure = func(service Service, failures float64, threshold float64, restarting bool, err interface{}, st []byte) {
var errString string var errString string
e, canError := err.(error) e, canError := err.(error)
@@ -255,7 +246,7 @@ func New(name string, spec Spec) (s *Supervisor) {
errString = fmt.Sprintf("%#v", err) errString = fmt.Sprintf("%#v", err)
} }
s.log(fmt.Sprintf("%s: Failed service '%s' (%f failures of %f), restarting: %#v, error: %s, stacktrace: %s", serviceName(supervisor), serviceName(service), failures, threshold, restarting, errString, string(st))) s.log(fmt.Sprintf("Failed service '%s' (%f failures of %f), restarting: %#v, error: %s, stacktrace: %s", serviceName(service), failures, threshold, restarting, errString, string(st)))
} }
s.logBackoff = func(s *Supervisor, entering bool) { s.logBackoff = func(s *Supervisor, entering bool) {
if entering { if entering {
@@ -355,24 +346,12 @@ will be started when the supervisor is.
The returned ServiceID may be passed to the Remove method of the Supervisor The returned ServiceID may be passed to the Remove method of the Supervisor
to terminate the service. to terminate the service.
As a special behavior, if the service added is itself a supervisor, the
supervisor being added will copy the Log function from the Supervisor it
is being added to. This allows factoring out providing a Supervisor
from its logging.
*/ */
func (s *Supervisor) Add(service Service) ServiceToken { func (s *Supervisor) Add(service Service) ServiceToken {
if s == nil { if s == nil {
panic("can't add service to nil *suture.Supervisor") panic("can't add service to nil *suture.Supervisor")
} }
if supervisor, isSupervisor := service.(*Supervisor); isSupervisor {
supervisor.logBadStop = s.logBadStop
supervisor.logFailure = s.logFailure
supervisor.logBackoff = s.logBackoff
}
if s.state == notRunning { if s.state == notRunning {
id := s.serviceCounter id := s.serviceCounter
s.serviceCounter++ s.serviceCounter++
@@ -513,12 +492,12 @@ func (s *Supervisor) handleFailedService(id serviceID, err interface{}, stacktra
if monitored { if monitored {
if s.state == normal { if s.state == normal {
s.runService(failedService, id) s.runService(failedService, id)
s.logFailure(s, failedService, s.failures, s.failureThreshold, true, err, stacktrace) s.logFailure(failedService, s.failures, s.failureThreshold, true, err, stacktrace)
} else { } else {
// FIXME: When restarting, check that the service still // FIXME: When restarting, check that the service still
// exists (it may have been stopped in the meantime) // exists (it may have been stopped in the meantime)
s.restartQueue = append(s.restartQueue, id) s.restartQueue = append(s.restartQueue, id)
s.logFailure(s, failedService, s.failures, s.failureThreshold, false, err, stacktrace) s.logFailure(failedService, s.failures, s.failureThreshold, false, err, stacktrace)
} }
} }
} }
@@ -557,7 +536,7 @@ func (s *Supervisor) removeService(id serviceID) {
case <-successChan: case <-successChan:
// Life is good! // Life is good!
case <-failChan: case <-failChan:
s.logBadStop(s, service) s.logBadStop(service)
} }
}() }()
} }
+6 -14
View File
@@ -77,7 +77,7 @@ func TestFailures(t *testing.T) {
// to avoid deadlocks during shutdown, we have to not try to send // to avoid deadlocks during shutdown, we have to not try to send
// things out on channels while we're shutting down (this undoes the // things out on channels while we're shutting down (this undoes the
// logFailure overide about 25 lines down) // logFailure overide about 25 lines down)
s.logFailure = func(*Supervisor, Service, float64, float64, bool, interface{}, []byte) {} s.logFailure = func(Service, float64, float64, bool, interface{}, []byte) {}
s.Stop() s.Stop()
}() }()
s.sync() s.sync()
@@ -102,7 +102,7 @@ func TestFailures(t *testing.T) {
failNotify := make(chan bool) failNotify := make(chan bool)
// use this to synchronize on here // use this to synchronize on here
s.logFailure = func(supervisor *Supervisor, s Service, cf float64, ft float64, r bool, error interface{}, stacktrace []byte) { s.logFailure = func(s Service, cf float64, ft float64, r bool, error interface{}, stacktrace []byte) {
failNotify <- r failNotify <- r
} }
@@ -276,8 +276,8 @@ func TestDefaultLogging(t *testing.T) {
serviceName(&BarelyService{}) serviceName(&BarelyService{})
s.logBadStop(s, service) s.logBadStop(service)
s.logFailure(s, service, 1, 1, true, errors.New("test error"), []byte{}) s.logFailure(service, 1, 1, true, errors.New("test error"), []byte{})
s.Stop() s.Stop()
} }
@@ -289,17 +289,9 @@ func TestNestedSupervisors(t *testing.T) {
super2 := NewSimple("Nested5") super2 := NewSimple("Nested5")
service := NewService("Service5") service := NewService("Service5")
super2.logBadStop = func(*Supervisor, Service) {
panic("Failed to copy logBadStop")
}
super1.Add(super2) super1.Add(super2)
super2.Add(service) super2.Add(service)
// test the functions got copied from super1; if this panics, it didn't
// get copied
super2.logBadStop(super2, service)
go super1.Serve() go super1.Serve()
super1.sync() super1.sync()
@@ -348,7 +340,7 @@ func TestStoppingStillWorksWithHungServices(t *testing.T) {
return resumeChan return resumeChan
} }
failNotify := make(chan struct{}) failNotify := make(chan struct{})
s.logBadStop = func(supervisor *Supervisor, s Service) { s.logBadStop = func(s Service) {
failNotify <- struct{}{} failNotify <- struct{}{}
} }
@@ -446,7 +438,7 @@ func TestFailingSupervisors(t *testing.T) {
} }
failNotify := make(chan string) failNotify := make(chan string)
// use this to synchronize on here // use this to synchronize on here
s1.logFailure = func(supervisor *Supervisor, s Service, cf float64, ft float64, r bool, error interface{}, stacktrace []byte) { s1.logFailure = func(s Service, cf float64, ft float64, r bool, error interface{}, stacktrace []byte) {
failNotify <- fmt.Sprintf("%s", s) failNotify <- fmt.Sprintf("%s", s)
} }
+16 -15
View File
@@ -471,22 +471,29 @@ func computeNonStarterCounts() {
if exp := c.forms[FCompatibility].expandedDecomp; len(exp) > 0 { if exp := c.forms[FCompatibility].expandedDecomp; len(exp) > 0 {
runes = exp runes = exp
} }
// We consider runes that combine backwards to be non-starters for the
// purpose of Stream-Safe Text Processing.
for _, r := range runes { for _, r := range runes {
if cr := &chars[r]; cr.ccc == 0 && !cr.forms[FCompatibility].combinesBackward { if chars[r].ccc == 0 {
break break
} }
c.nLeadingNonStarters++ c.nLeadingNonStarters++
} }
for i := len(runes) - 1; i >= 0; i-- { for i := len(runes) - 1; i >= 0; i-- {
if cr := &chars[runes[i]]; cr.ccc == 0 && !cr.forms[FCompatibility].combinesBackward { if chars[runes[i]].ccc == 0 {
break break
} }
c.nTrailingNonStarters++ c.nTrailingNonStarters++
} }
if c.nTrailingNonStarters > 3 {
log.Fatalf("%U: Decomposition with more than 3 (%d) trailing modifiers (%U)", i, c.nTrailingNonStarters, runes) // We consider runes that combine backwards to be non-starters for the
// purpose of Stream-Safe Text Processing.
for _, f := range c.forms {
if c.ccc == 0 && f.combinesBackward {
if len(c.forms[FCompatibility].expandedDecomp) > 0 {
log.Fatalf("%U: CCC==0 modifier with an expansion is not supported.", i)
}
c.nTrailingNonStarters = 1
c.nLeadingNonStarters = 1
}
} }
if isHangul(rune(i)) { if isHangul(rune(i)) {
@@ -832,16 +839,10 @@ func verifyComputed() {
continue continue
} }
if a, b := c.nLeadingNonStarters > 0, (c.ccc > 0 || f.combinesBackward); a != b { if a, b := c.nLeadingNonStarters > 0, (c.ccc > 0 || f.combinesBackward); a != b {
// We accept these runes to be treated differently (it only affects // We accept these two runes to be treated differently (it only affects
// segment breaking in iteration, most likely on improper use), but // segment breaking in iteration, most likely on inproper use), but
// reconsider if more characters are added. // reconsider if more characters are added.
// U+FF9E HALFWIDTH KATAKANA VOICED SOUND MARK;Lm;0;L;<narrow> 3099;;;;N;;;;; if i != 0xFF9E && i != 0xFF9F {
// U+FF9F HALFWIDTH KATAKANA SEMI-VOICED SOUND MARK;Lm;0;L;<narrow> 309A;;;;N;;;;;
// U+3133 HANGUL LETTER KIYEOK-SIOS;Lo;0;L;<compat> 11AA;;;;N;HANGUL LETTER GIYEOG SIOS;;;;
// U+318E HANGUL LETTER ARAEAE;Lo;0;L;<compat> 11A1;;;;N;HANGUL LETTER ALAE AE;;;;
// U+FFA3 HALFWIDTH HANGUL LETTER KIYEOK-SIOS;Lo;0;L;<narrow> 3133;;;;N;HALFWIDTH HANGUL LETTER GIYEOG SIOS;;;;
// U+FFDC HALFWIDTH HANGUL LETTER I;Lo;0;L;<narrow> 3163;;;;N;;;;;
if i != 0xFF9E && i != 0xFF9F && !(0x3133 <= i && i <= 0x318E) && !(0xFFA3 <= i && i <= 0xFFDC) {
log.Fatalf("%U: nLead was %v; want %v", i, a, b) log.Fatalf("%U: nLead was %v; want %v", i, a, b)
} }
} }
-56
View File
@@ -113,25 +113,7 @@ var decomposeSegmentTests = []PositionTest{
{"\u00C0b", 2, "A\u0300"}, {"\u00C0b", 2, "A\u0300"},
// long // long
{grave(31), 60, grave(30) + cgj}, {grave(31), 60, grave(30) + cgj},
{"a" + grave(31), 61, "a" + grave(30) + cgj},
// Stability tests: see http://www.unicode.org/review/pr-29.html.
// U+0300 COMBINING GRAVE ACCENT;Mn;230;NSM;;;;;N;NON-SPACING GRAVE;;;;
// U+0B47 ORIYA VOWEL SIGN E;Mc;0;L;;;;;N;;;;;
// U+0B3E ORIYA VOWEL SIGN AA;Mc;0;L;;;;;N;;;;;
// U+1100 HANGUL CHOSEONG KIYEOK;Lo;0;L;;;;;N;;;;;
// U+1161 HANGUL JUNGSEONG A;Lo;0;L;;;;;N;;;;;
{"\u0B47\u0300\u0B3E", 8, "\u0B47\u0300\u0B3E"},
{"\u1100\u0300\u1161", 8, "\u1100\u0300\u1161"},
{"\u0B47\u0B3E", 6, "\u0B47\u0B3E"},
{"\u1100\u1161", 6, "\u1100\u1161"},
// U+04DA MALAYALAM VOWEL SIGN O;Mc;0;L;0D46 0D3E;;;;N;;;;;
// Sequence of decomposing characters that are starters and modifiers.
{"\u0d4a" + strings.Repeat("\u0d3e", 31), 90, "\u0d46" + strings.Repeat("\u0d3e", 30) + cgj},
{grave(30), 60, grave(30)}, {grave(30), 60, grave(30)},
// U+FF9E is a starter, but decomposes to U+3099, which is not.
{grave(30) + "\uff9e", 60, grave(30) + cgj}, {grave(30) + "\uff9e", 60, grave(30) + cgj},
// ends with incomplete UTF-8 encoding // ends with incomplete UTF-8 encoding
{"\xCC", 0, ""}, {"\xCC", 0, ""},
@@ -570,44 +552,6 @@ var appendTestsNFC = []AppendTest{
"a" + rep(0x0305, maxNonStarters+4) + "\u0316", "a" + rep(0x0305, maxNonStarters+4) + "\u0316",
"a" + rep(0x0305, maxNonStarters) + cgj + "\u0316" + rep(0x305, 4), "a" + rep(0x0305, maxNonStarters) + cgj + "\u0316" + rep(0x305, 4),
}, },
{ // Combine across non-blocking non-starters.
// U+0327 COMBINING CEDILLA;Mn;202;NSM;;;;;N;NON-SPACING CEDILLA;;;;
// U+0325 COMBINING RING BELOW;Mn;220;NSM;;;;;N;NON-SPACING RING BELOW;;;;
"", "a\u0327\u0325", "\u1e01\u0327",
},
{ // Jamo V+T does not combine.
"",
"\u1161\u11a8",
"\u1161\u11a8",
},
// Stability tests: see http://www.unicode.org/review/pr-29.html.
{"", "\u0b47\u0300\u0b3e", "\u0b47\u0300\u0b3e"},
{"", "\u1100\u0300\u1161", "\u1100\u0300\u1161"},
{"", "\u0b47\u0b3e", "\u0b4b"},
{"", "\u1100\u1161", "\uac00"},
// U+04DA MALAYALAM VOWEL SIGN O;Mc;0;L;0D46 0D3E;;;;N;;;;;
{ // 0d4a starts a new segment.
"",
"\u0d4a" + strings.Repeat("\u0d3e", 15) + "\u0d4a" + strings.Repeat("\u0d3e", 15),
"\u0d4a" + strings.Repeat("\u0d3e", 15) + "\u0d4a" + strings.Repeat("\u0d3e", 15),
},
{ // Split combining characters.
// TODO: don't insert CGJ before starters.
"",
"\u0d46" + strings.Repeat("\u0d3e", 31),
"\u0d4a" + strings.Repeat("\u0d3e", 29) + cgj + "\u0d3e",
},
{ // Split combining characters.
"",
"\u0d4a" + strings.Repeat("\u0d3e", 30),
"\u0d4a" + strings.Repeat("\u0d3e", 29) + cgj + "\u0d3e",
},
} }
var appendTestsNFD = []AppendTest{ var appendTestsNFD = []AppendTest{
File diff suppressed because it is too large Load Diff
+1 -10
View File
@@ -14,24 +14,17 @@ andrew-d <andrew@du.nham.ca>
asdil12 <dominik@heidler.eu> asdil12 <dominik@heidler.eu>
bencurthoys <ben@bencurthoys.com> bencurthoys <ben@bencurthoys.com>
bigbear2nd <bigbear2nd@gmail.com> bigbear2nd <bigbear2nd@gmail.com>
brbecker <brbecker@gmail.com>
brendanlong <self@brendanlong.com> brendanlong <self@brendanlong.com>
brgmnn <dan.arne.bergmann@gmail.com> <brgmnn@users.noreply.github.com>
bsidhom <bsidhom@gmail.com> bsidhom <bsidhom@gmail.com>
calmh <jakob@nym.se> calmh <jakob@nym.se>
canton7 <antony.male@gmail.com>
cdata <chris@scriptolo.gy> cdata <chris@scriptolo.gy>
cdhowie <me@chrishowie.com>
ceh <emil@hessman.se> ceh <emil@hessman.se>
cqcallaw <enlightened.despot@gmail.com> cqcallaw <enlightened.despot@gmail.com>
dva <denisva@gmail.com>
dzarda <dzardacz@gmail.com> dzarda <dzardacz@gmail.com>
facastagnini <federico.castagnini@gmail.com> facastagnini <federico.castagnini@gmail.com>
filoozoom <philippe@schommers.be> filoozoom <philippe@schommers.be>
frioux <frew@afoolishmanifesto.com> <frioux@gmail.com> frioux <frew@afoolishmanifesto.com> <frioux@gmail.com>
fti7 <frank@isemann.name>
gillisig <gilli@vx.is> gillisig <gilli@vx.is>
hadogenes <szafar@linux.pl>
jarlebring <jarlebring@gmail.com> jarlebring <jarlebring@gmail.com>
jedie <github.com@jensdiemer.de> <git@jensdiemer.de> jedie <github.com@jensdiemer.de> <git@jensdiemer.de>
jpjp <jamespatterson@operamail.com> <jpjp@users.noreply.github.com> jpjp <jamespatterson@operamail.com> <jpjp@users.noreply.github.com>
@@ -41,7 +34,6 @@ kozec <kozec@kozec.com>
krozycki <rozycki.karol@gmail.com> krozycki <rozycki.karol@gmail.com>
marcindziadus <dziadus.marcin@gmail.com> marcindziadus <dziadus.marcin@gmail.com>
marclaporte <marc@marclaporte.com> marclaporte <marc@marclaporte.com>
mogwa1 <devriesb@gmail.com>
moshen <moshen.colin@gmail.com> moshen <moshen.colin@gmail.com>
mvdan <mvdan@mvdan.cc> mvdan <mvdan@mvdan.cc>
pascalj <github@pascalj.com> <mail@pascal-jungblut.com> pascalj <github@pascalj.com> <mail@pascal-jungblut.com>
@@ -50,9 +42,9 @@ philips <brandon@ifup.org>
piobpl <piotrb10@gmail.com> piobpl <piotrb10@gmail.com>
pluby <phill.luby@newredo.com> pluby <phill.luby@newredo.com>
pyfisch <pyfisch@gmail.com> pyfisch <pyfisch@gmail.com>
qbit <qbit@deftly.net>
ralder <ralder@yandex.ru> ralder <ralder@yandex.ru>
rumpelsepp <stefan@sevenbyte.org> rumpelsepp <stefan@sevenbyte.org>
qbit <qbit@deftly.net>
sciurius <jvromans@squirrel.nl> sciurius <jvromans@squirrel.nl>
seehuhn <voss@seehuhn.de> seehuhn <voss@seehuhn.de>
snnd <dw@risu.io> snnd <dw@risu.io>
@@ -61,5 +53,4 @@ tnn2 <tnn@nygren.pp.se>
tojrobinson <tully@tojr.org> tojrobinson <tully@tojr.org>
uok <ueomkail@gmail.com> <uok@users.noreply.github.com> uok <ueomkail@gmail.com> <uok@users.noreply.github.com>
veeti <veeti.paananen@rojekti.fi> veeti <veeti.paananen@rojekti.fi>
wsgcsysadmin <e.meitner@willystreet.coo>
zukoo <fxgsell@gmail.com> zukoo <fxgsell@gmail.com>
+9 -8
View File
@@ -1,11 +1,11 @@
Syncthing syncthing
========= =========
[![Latest Build](http://img.shields.io/jenkins/s/http/build.syncthing.net/syncthing.svg?style=flat-square)](http://build.syncthing.net/job/syncthing/lastBuild/) [![Latest Build](http://img.shields.io/jenkins/s/http/build.syncthing.net/syncthing.svg?style=flat-square)](http://build.syncthing.net/job/syncthing/lastBuild/)
[![API Documentation](http://img.shields.io/badge/api-Godoc-blue.svg?style=flat-square)](http://godoc.org/github.com/syncthing/syncthing) [![API Documentation](http://img.shields.io/badge/api-Godoc-blue.svg?style=flat-square)](http://godoc.org/github.com/syncthing/syncthing)
[![MPLv2 License](http://img.shields.io/badge/license-MPLv2-blue.svg?style=flat-square)](https://www.mozilla.org/MPL/2.0/) [![MPLv2 License](http://img.shields.io/badge/license-MPLv2-blue.svg?style=flat-square)](https://www.mozilla.org/MPL/2.0/)
This is the Syncthing project which pursues the following goals: This is the `syncthing` project which pursues the following goals:
1. Define a protocol for synchronization of a folder between a number of 1. Define a protocol for synchronization of a folder between a number of
collaborating devices. This protocol should be well defined, unambiguous, collaborating devices. This protocol should be well defined, unambiguous,
@@ -18,16 +18,16 @@ This is the Syncthing project which pursues the following goals:
alternative, compatible implementations of the protocol will arise. alternative, compatible implementations of the protocol will arise.
The two are evolving together; the protocol is not to be considered The two are evolving together; the protocol is not to be considered
stable until Syncthing 1.0 is released, at which point it is locked down stable until syncthing 1.0 is released, at which point it is locked down
for incompatible changes. for incompatible changes.
Getting Started Getting Started
--------------- ---------------
Take a look at the [getting started Take a look at the [getting started
guide](http://docs.syncthing.net/intro/getting-started.html). guide](https://github.com/syncthing/syncthing/wiki/Getting-Started).
There are a few examples for keeping Syncthing running in the background There are a few examples for keeping syncthing running in the background
on your system in [the etc directory](https://github.com/syncthing/syncthing/blob/master/etc). on your system in [the etc directory](https://github.com/syncthing/syncthing/blob/master/etc).
There is an IRC channel, `#syncthing` on Freenode, for talking directly There is an IRC channel, `#syncthing` on Freenode, for talking directly
@@ -37,7 +37,7 @@ Building
-------- --------
Building Syncthing from source is easy, and there's a Building Syncthing from source is easy, and there's a
[guide](http://docs.syncthing.net/dev/building.html). [guide](https://github.com/syncthing/syncthing/wiki/Building).
that describes it for both Unix and Windows systems. that describes it for both Unix and Windows systems.
Signed Releases Signed Releases
@@ -51,8 +51,9 @@ available in the md5sum.txt.asc and sha1sum.txt.asc files.
Documentation Documentation
============= =============
Please see the [Syncthing The [syncthing
documentation site](http://docs.syncthing.net/). documentation](https://github.com/syncthing/syncthing/wiki/) is on the
Github wiki.
All code is licensed under the All code is licensed under the
[MPLv2 License](https://github.com/syncthing/syncthing/blob/master/LICENSE). [MPLv2 License](https://github.com/syncthing/syncthing/blob/master/LICENSE).
+4 -6
View File
@@ -20,8 +20,6 @@ import (
"strings" "strings"
) )
const htmlFile = "gui/syncthing/core/aboutModalView.html"
func main() { func main() {
bs := readAll("AUTHORS") bs := readAll("AUTHORS")
lines := strings.Split(string(bs), "\n") lines := strings.Split(string(bs), "\n")
@@ -29,17 +27,17 @@ func main() {
authors := make([]string, 0, len(lines)) authors := make([]string, 0, len(lines))
for _, line := range lines { for _, line := range lines {
if m := nameRe.FindStringSubmatch(line); len(m) == 2 { if m := nameRe.FindStringSubmatch(line); len(m) == 2 {
authors = append(authors, " <li class=\"auto-generated\">"+m[1]+"</li>") authors = append(authors, " <li class=\"auto-generated\">"+m[1]+"</li>")
} }
} }
sort.Strings(authors) sort.Strings(authors)
replacement := strings.Join(authors, "\n") replacement := strings.Join(authors, "\n")
authorsRe := regexp.MustCompile(`(?s)id="contributor-list">.*?</ul>`) authorsRe := regexp.MustCompile(`(?s)id="contributor-list">.*?</ul>`)
bs = readAll(htmlFile) bs = readAll("gui/index.html")
bs = authorsRe.ReplaceAll(bs, []byte("id=\"contributor-list\">\n"+replacement+"\n </ul>")) bs = authorsRe.ReplaceAll(bs, []byte("id=\"contributor-list\">\n"+replacement+"\n </ul>"))
if err := ioutil.WriteFile(htmlFile, bs, 0644); err != nil { if err := ioutil.WriteFile("gui/index.html", bs, 0644); err != nil {
log.Fatal(err) log.Fatal(err)
} }
} }
+26 -66
View File
@@ -36,7 +36,6 @@ var (
goos string goos string
noupgrade bool noupgrade bool
version string version string
goVersion float64
race bool race bool
) )
@@ -71,7 +70,7 @@ func main() {
log.Printf("Unknown goarch %q; proceed with caution!", goarch) log.Printf("Unknown goarch %q; proceed with caution!", goarch)
} }
goVersion, _ = checkRequiredGoVersion() checkRequiredGoVersion()
if flag.NArg() == 0 { if flag.NArg() == 0 {
var tags []string var tags []string
@@ -81,9 +80,9 @@ func main() {
install("./cmd/...", tags) install("./cmd/...", tags)
vet("./cmd/syncthing") vet("./cmd/syncthing")
vet("./lib/...") vet("./internal/...")
lint("./cmd/syncthing") lint("./cmd/syncthing")
lint("./lib/...") lint("./internal/...")
return return
} }
@@ -111,9 +110,6 @@ func main() {
case "test": case "test":
test("./...") test("./...")
case "bench":
bench("./...")
case "assets": case "assets":
assets() assets()
@@ -143,11 +139,11 @@ func main() {
case "vet": case "vet":
vet("./cmd/syncthing") vet("./cmd/syncthing")
vet("./lib/...") vet("./internal/...")
case "lint": case "lint":
lint("./cmd/syncthing") lint("./cmd/syncthing")
lint("./lib/...") lint("./internal/...")
default: default:
log.Fatalf("Unknown command %q", cmd) log.Fatalf("Unknown command %q", cmd)
@@ -155,7 +151,7 @@ func main() {
} }
} }
func checkRequiredGoVersion() (float64, bool) { func checkRequiredGoVersion() {
ver := run("go", "version") ver := run("go", "version")
re := regexp.MustCompile(`go version go(\d+\.\d+)`) re := regexp.MustCompile(`go version go(\d+\.\d+)`)
if m := re.FindSubmatch(ver); len(m) == 2 { if m := re.FindSubmatch(ver); len(m) == 2 {
@@ -164,16 +160,14 @@ func checkRequiredGoVersion() (float64, bool) {
f, err := strconv.ParseFloat(vs, 64) f, err := strconv.ParseFloat(vs, 64)
if err != nil { if err != nil {
log.Printf("*** Couldn't parse Go version out of %q.\n*** This isn't known to work, proceed on your own risk.", vs) log.Printf("*** Couldn't parse Go version out of %q.\n*** This isn't known to work, proceed on your own risk.", vs)
return 0, false return
} }
if f < minGoVersion { if f < minGoVersion {
log.Fatalf("*** Go version %.01f is less than required %.01f.\n*** This is known not to work, not proceeding.", f, minGoVersion) log.Fatalf("*** Go version %.01f is less than required %.01f.\n*** This is known not to work, not proceeding.", f, minGoVersion)
} }
return f, true } else {
log.Printf("*** Unknown Go version %q.\n*** This isn't known to work, proceed on your own risk.", ver)
} }
log.Printf("*** Unknown Go version %q.\n*** This isn't known to work, proceed on your own risk.", ver)
return 0, false
} }
func setup() { func setup() {
@@ -181,8 +175,6 @@ func setup() {
runPrint("go", "get", "-v", "golang.org/x/tools/cmd/vet") runPrint("go", "get", "-v", "golang.org/x/tools/cmd/vet")
runPrint("go", "get", "-v", "golang.org/x/net/html") runPrint("go", "get", "-v", "golang.org/x/net/html")
runPrint("go", "get", "-v", "github.com/tools/godep") runPrint("go", "get", "-v", "github.com/tools/godep")
runPrint("go", "get", "-v", "github.com/axw/gocov/gocov")
runPrint("go", "get", "-v", "github.com/AlekSi/gocov-xml")
} }
func test(pkg string) { func test(pkg string) {
@@ -190,11 +182,6 @@ func test(pkg string) {
runPrint("go", "test", "-short", "-timeout", "60s", pkg) runPrint("go", "test", "-short", "-timeout", "60s", pkg)
} }
func bench(pkg string) {
setBuildEnv()
runPrint("go", "test", "-run", "NONE", "-bench", ".", pkg)
}
func install(pkg string, tags []string) { func install(pkg string, tags []string) {
os.Setenv("GOBIN", "./bin") os.Setenv("GOBIN", "./bin")
args := []string{"install", "-v", "-ldflags", ldflags()} args := []string{"install", "-v", "-ldflags", ldflags()}
@@ -291,17 +278,6 @@ func buildZip() {
func buildDeb() { func buildDeb() {
os.RemoveAll("deb") os.RemoveAll("deb")
// "goarch" here is set to whatever the Debian packages expect. We correct
// "it to what we actually know how to build and keep the Debian variant
// "name in "debarch".
debarch := goarch
switch goarch {
case "i386":
goarch = "386"
case "armel", "armhf":
goarch = "arm"
}
build("./cmd/syncthing", []string{"noupgrade"}) build("./cmd/syncthing", []string{"noupgrade"})
files := []archiveFile{ files := []archiveFile{
@@ -309,18 +285,6 @@ func buildDeb() {
{src: "LICENSE", dst: "deb/usr/share/doc/syncthing/LICENSE.txt", perm: 0644}, {src: "LICENSE", dst: "deb/usr/share/doc/syncthing/LICENSE.txt", perm: 0644},
{src: "AUTHORS", dst: "deb/usr/share/doc/syncthing/AUTHORS.txt", perm: 0644}, {src: "AUTHORS", dst: "deb/usr/share/doc/syncthing/AUTHORS.txt", perm: 0644},
{src: "syncthing", dst: "deb/usr/bin/syncthing", perm: 0755}, {src: "syncthing", dst: "deb/usr/bin/syncthing", perm: 0755},
{src: "man/syncthing.1", dst: "deb/usr/share/man/man1/syncthing.1", perm: 0644},
{src: "man/syncthing-config.5", dst: "deb/usr/share/man/man5/syncthing-config.5", perm: 0644},
{src: "man/syncthing-stignore.5", dst: "deb/usr/share/man/man5/syncthing-stignore.5", perm: 0644},
{src: "man/syncthing-device-ids.7", dst: "deb/usr/share/man/man7/syncthing-device-ids.7", perm: 0644},
{src: "man/syncthing-event-api.7", dst: "deb/usr/share/man/man7/syncthing-event-api.7", perm: 0644},
{src: "man/syncthing-faq.7", dst: "deb/usr/share/man/man7/syncthing-faq.7", perm: 0644},
{src: "man/syncthing-networking.7", dst: "deb/usr/share/man/man7/syncthing-networking.7", perm: 0644},
{src: "man/syncthing-rest-api.7", dst: "deb/usr/share/man/man7/syncthing-rest-api.7", perm: 0644},
{src: "man/syncthing-security.7", dst: "deb/usr/share/man/man7/syncthing-security.7", perm: 0644},
{src: "man/syncthing-versioning.7", dst: "deb/usr/share/man/man7/syncthing-versioning.7", perm: 0644},
{src: "etc/linux-systemd/system/syncthing@.service", dst: "deb/lib/systemd/system/syncthing@.service", perm: 0644},
{src: "etc/linux-systemd/user/syncthing.service", dst: "deb/usr/lib/systemd/user/syncthing.service", perm: 0644},
} }
for _, file := range listFiles("extra") { for _, file := range listFiles("extra") {
@@ -333,6 +297,11 @@ func buildDeb() {
} }
} }
debarch := goarch
if debarch == "386" {
debarch = "i386"
}
control := `Package: syncthing control := `Package: syncthing
Architecture: {{arch}} Architecture: {{arch}}
Depends: libc6 Depends: libc6
@@ -407,16 +376,16 @@ func setBuildEnv() {
func assets() { func assets() {
setBuildEnv() setBuildEnv()
runPipe("lib/auto/gui.files.go", "go", "run", "script/genassets.go", "gui") runPipe("internal/auto/gui.files.go", "go", "run", "cmd/genassets/main.go", "gui")
} }
func xdr() { func xdr() {
runPrint("go", "generate", "./lib/discover", "./lib/db") runPrint("go", "generate", "./internal/discover", "./internal/db")
} }
func translate() { func translate() {
os.Chdir("gui/assets/lang") os.Chdir("gui/assets/lang")
runPipe("lang-en-new.json", "go", "run", "../../../script/translate.go", "lang-en.json", "../../") runPipe("lang-en-new.json", "go", "run", "../../../cmd/translate/main.go", "lang-en.json", "../../index.html")
os.Remove("lang-en.json") os.Remove("lang-en.json")
err := os.Rename("lang-en-new.json", "lang-en.json") err := os.Rename("lang-en-new.json", "lang-en.json")
if err != nil { if err != nil {
@@ -427,7 +396,7 @@ func translate() {
func transifex() { func transifex() {
os.Chdir("gui/assets/lang") os.Chdir("gui/assets/lang")
runPrint("go", "run", "../../../script/transifexdl.go") runPrint("go", "run", "../../../cmd/transifexdl/main.go")
os.Chdir("../../..") os.Chdir("../../..")
assets() assets()
} }
@@ -443,18 +412,13 @@ func clean() {
} }
func ldflags() string { func ldflags() string {
sep := ' ' var b bytes.Buffer
if goVersion > 1.4 {
sep = '='
}
b := new(bytes.Buffer)
b.WriteString("-w") b.WriteString("-w")
fmt.Fprintf(b, " -X main.Version%c%s", sep, version) b.WriteString(fmt.Sprintf(" -X main.Version %s", version))
fmt.Fprintf(b, " -X main.BuildStamp%c%d", sep, buildStamp()) b.WriteString(fmt.Sprintf(" -X main.BuildStamp %d", buildStamp()))
fmt.Fprintf(b, " -X main.BuildUser%c%s", sep, buildUser()) b.WriteString(fmt.Sprintf(" -X main.BuildUser %s", buildUser()))
fmt.Fprintf(b, " -X main.BuildHost%c%s", sep, buildHost()) b.WriteString(fmt.Sprintf(" -X main.BuildHost %s", buildHost()))
fmt.Fprintf(b, " -X main.BuildEnv%c%s", sep, buildEnvironment()) b.WriteString(fmt.Sprintf(" -X main.BuildEnv %s", buildEnvironment()))
return b.String() return b.String()
} }
@@ -752,9 +716,7 @@ func vet(pkg string) {
if falseAlarmComposites.Match(line) || exitStatus.Match(line) { if falseAlarmComposites.Match(line) || exitStatus.Match(line) {
continue continue
} }
if len(line) > 0 { log.Printf("%s", line)
log.Printf("%s", line)
}
} }
} }
@@ -770,8 +732,6 @@ func lint(pkg string) {
if analCommentPolicy.Match(line) { if analCommentPolicy.Match(line) {
continue continue
} }
if len(line) > 0 { log.Printf("%s", line)
log.Printf("%s", line)
}
} }
} }
+55 -75
View File
@@ -4,97 +4,79 @@ IFS=$'\n\t'
STTRACE=${STTRACE:-} STTRACE=${STTRACE:-}
script() {
name="$1"
shift
go run "script/$name.go" "$@"
}
build() {
go run build.go "$@"
}
case "${1:-default}" in case "${1:-default}" in
default) default)
build go run build.go
;; ;;
clean) clean)
build "$@" go run build.go "$1"
;;
tar)
build "$@"
;;
deps)
build "$@"
;;
assets)
build "$@"
;;
xdr)
build "$@"
;;
translate)
build "$@"
;;
deb)
build "$@"
;;
setup)
build "$@"
;; ;;
test) test)
ulimit -t 600 &>/dev/null || true ulimit -t 60 &>/dev/null || true
ulimit -d 512000 &>/dev/null || true ulimit -d 512000 &>/dev/null || true
ulimit -m 512000 &>/dev/null || true ulimit -m 512000 &>/dev/null || true
LOGGER_DISCARD=1 build test
go run build.go "$1"
;; ;;
bench) tar)
LOGGER_DISCARD=1 build bench | script benchfilter go run build.go "$1"
;; ;;
prerelease) deps)
build transifex go run build.go "$1"
git add -A gui/assets/ lib/auto/ ;;
pushd man ; ./refresh.sh ; popd
git add -A man assets)
go run build.go "$1"
;;
xdr)
go run build.go "$1"
;;
translate)
go run build.go "$1"
;;
transifex)
go run build.go "$1"
;; ;;
noupgrade) noupgrade)
build -no-upgrade tar go run build.go -no-upgrade tar
;; ;;
all) all)
build -goos darwin -goarch amd64 tar go run build.go -goos darwin -goarch amd64 tar
go run build.go -goos darwin -goarch 386 tar
build -goos dragonfly -goarch amd64 tar go run build.go -goos dragonfly -goarch 386 tar
go run build.go -goos dragonfly -goarch amd64 tar
build -goos freebsd -goarch 386 tar go run build.go -goos freebsd -goarch 386 tar
build -goos freebsd -goarch amd64 tar go run build.go -goos freebsd -goarch amd64 tar
build -goos linux -goarch 386 tar go run build.go -goos linux -goarch 386 tar
build -goos linux -goarch amd64 tar go run build.go -goos linux -goarch amd64 tar
build -goos linux -goarch arm tar go run build.go -goos linux -goarch arm tar
build -goos netbsd -goarch 386 tar go run build.go -goos netbsd -goarch 386 tar
build -goos netbsd -goarch amd64 tar go run build.go -goos netbsd -goarch amd64 tar
build -goos openbsd -goarch 386 tar go run build.go -goos openbsd -goarch 386 tar
build -goos openbsd -goarch amd64 tar go run build.go -goos openbsd -goarch amd64 tar
build -goos solaris -goarch amd64 tar go run build.go -goos solaris -goarch amd64 tar
build -goos windows -goarch 386 zip go run build.go -goos windows -goarch 386 zip
build -goos windows -goarch amd64 zip go run build.go -goos windows -goarch amd64 zip
;;
setup)
echo "Don't worry, just build."
;; ;;
test-cov) test-cov)
@@ -102,6 +84,9 @@ case "${1:-default}" in
ulimit -d 512000 &>/dev/null || true ulimit -d 512000 &>/dev/null || true
ulimit -m 512000 &>/dev/null || true ulimit -m 512000 &>/dev/null || true
go get github.com/axw/gocov/gocov
go get github.com/AlekSi/gocov-xml
echo "mode: set" > coverage.out echo "mode: set" > coverage.out
fail=0 fail=0
@@ -127,25 +112,22 @@ case "${1:-default}" in
;; ;;
docker-all) docker-all)
img=${DOCKERIMG:-syncthing/build:latest}
docker run --rm -h syncthing-builder -u $(id -u) -t \ docker run --rm -h syncthing-builder -u $(id -u) -t \
-v $(pwd):/go/src/github.com/syncthing/syncthing \ -v $(pwd):/go/src/github.com/syncthing/syncthing \
-w /go/src/github.com/syncthing/syncthing \ -w /go/src/github.com/syncthing/syncthing \
-e "STTRACE=$STTRACE" \ -e "STTRACE=$STTRACE" \
"$img" \ syncthing/build:latest \
sh -c './build.sh clean \ sh -c './build.sh clean \
&& ./build.sh test-cov \ && ./build.sh all \
&& ./build.sh bench \ && STTRACE=all ./build.sh test-cov'
&& ./build.sh all'
;; ;;
docker-test) docker-test)
img=${DOCKERIMG:-syncthing/build:latest}
docker run --rm -h syncthing-builder -u $(id -u) -t \ docker run --rm -h syncthing-builder -u $(id -u) -t \
-v $(pwd):/go/src/github.com/syncthing/syncthing \ -v $(pwd):/go/src/github.com/syncthing/syncthing \
-w /go/src/github.com/syncthing/syncthing \ -w /go/src/github.com/syncthing/syncthing \
-e "STTRACE=$STTRACE" \ -e "STTRACE=$STTRACE" \
"$img" \ syncthing/build:latest \
sh -euxc './build.sh clean \ sh -euxc './build.sh clean \
&& go run build.go -race \ && go run build.go -race \
&& export GOPATH=$(pwd)/Godeps/_workspace:$GOPATH \ && export GOPATH=$(pwd)/Godeps/_workspace:$GOPATH \
@@ -155,23 +137,21 @@ case "${1:-default}" in
;; ;;
docker-lint) docker-lint)
img=${DOCKERIMG:-syncthing/build:latest}
docker run --rm -h syncthing-builder -u $(id -u) -t \ docker run --rm -h syncthing-builder -u $(id -u) -t \
-v $(pwd):/go/src/github.com/syncthing/syncthing \ -v $(pwd):/go/src/github.com/syncthing/syncthing \
-w /go/src/github.com/syncthing/syncthing \ -w /go/src/github.com/syncthing/syncthing \
-e "STTRACE=$STTRACE" \ -e "STTRACE=$STTRACE" \
"$img" \ syncthing/build:latest \
sh -euxc 'go run build.go lint' sh -euxc 'go run build.go lint'
;; ;;
docker-vet) docker-vet)
img=${DOCKERIMG:-syncthing/build:latest}
docker run --rm -h syncthing-builder -u $(id -u) -t \ docker run --rm -h syncthing-builder -u $(id -u) -t \
-v $(pwd):/go/src/github.com/syncthing/syncthing \ -v $(pwd):/go/src/github.com/syncthing/syncthing \
-w /go/src/github.com/syncthing/syncthing \ -w /go/src/github.com/syncthing/syncthing \
-e "STTRACE=$STTRACE" \ -e "STTRACE=$STTRACE" \
"$img" \ syncthing/build:latest \
sh -euxc 'go run build.go vet' sh -euxc 'go run build.go vet'
;; ;;
+52
View File
@@ -0,0 +1,52 @@
#!/bin/bash
missing-authors() {
for email in $(git log --format=%ae HEAD | sort | uniq) ; do
grep -q "$email" AUTHORS || echo $email
done
}
no-docs-typos() {
# Commits that are known to not change code
grep -v 63bd0136fb40a91efaa279cb4b4159d82e8e6904 |\
grep -v 4e2feb6fbc791bb8a2daf0ab8efb10775d66343e |\
grep -v f2459ef3319b2f060dbcdacd0c35a1788a94b8bd |\
grep -v b61f418bf2d1f7d5a9d7088a20a2a448e5e66801 |\
grep -v f0621207e3953711f9ab86d99724f1d0faac45b1 |\
grep -v f1120d7aa936c0658429edef0037792520b46334 |\
grep -v a9339d0627fff439879d157c75077f02c9fac61b |\
grep -v 254c63763a3ad42fd82259f1767db526cff94a14 |\
grep -v 4b76ec40c07078beaa2c5e250ed7d9bd6276a718 |\
grep -v ffc39dfbcb34eacc3ea12327a02b6e7741a2c207 |\
grep -v 32a76901a91ff0f663db6f0830e0aedec946e4d0 |\
grep -v af3288043a49bcc28f8ae3060852a09de552fe5f
}
print-missing-authors() {
for email in $(missing-authors) ; do
git log --author="$email" --format="%H %ae %s" | no-docs-typos
done
}
print-missing-copyright() {
find . -name \*.go | xargs egrep -L 'Copyright \(C\)|automatically generated' | grep -v Godeps | grep -v internal/auto/
}
authors=$(print-missing-authors)
if [[ ! -z $authors ]] ; then
echo '***'
echo Author emails not in AUTHORS:
echo $authors
echo '***'
exit 1
fi
copy=$(print-missing-copyright)
if [[ ! -z $copy ]] ; then
echo ***
echo Files missing copyright notice:
echo $copy
echo ***
exit 1
fi
@@ -36,7 +36,8 @@ const (
func Assets() map[string][]byte { func Assets() map[string][]byte {
var assets = make(map[string][]byte, {{.Assets | len}}) var assets = make(map[string][]byte, {{.Assets | len}})
{{range $asset := .Assets}} {{range $asset := .Assets}}
assets["{{$asset.Name}}"], _ = base64.StdEncoding.DecodeString("{{$asset.Data}}"){{end}} assets["{{$asset.Name}}"], _ = base64.StdEncoding.DecodeString("{{$asset.Data}}")
{{end}}
return assets return assets
} }
+1 -1
View File
@@ -16,7 +16,7 @@ import (
"os" "os"
"path/filepath" "path/filepath"
"github.com/syncthing/syncthing/lib/symlinks" "github.com/syncthing/syncthing/internal/symlinks"
) )
func main() { func main() {
+1 -1
View File
@@ -13,7 +13,7 @@ import (
"path/filepath" "path/filepath"
"github.com/syncthing/protocol" "github.com/syncthing/protocol"
"github.com/syncthing/syncthing/lib/scanner" "github.com/syncthing/syncthing/internal/scanner"
) )
func main() { func main() {
+1 -1
View File
@@ -12,7 +12,7 @@ import (
"os" "os"
"github.com/syncthing/protocol" "github.com/syncthing/protocol"
"github.com/syncthing/syncthing/lib/discover" "github.com/syncthing/syncthing/internal/discover"
) )
func main() { func main() {
+4 -4
View File
@@ -14,7 +14,7 @@ import (
"os" "os"
"github.com/syncthing/protocol" "github.com/syncthing/protocol"
"github.com/syncthing/syncthing/lib/db" "github.com/syncthing/syncthing/internal/db"
"github.com/syndtr/goleveldb/leveldb" "github.com/syndtr/goleveldb/leveldb"
"github.com/syndtr/goleveldb/leveldb/opt" "github.com/syndtr/goleveldb/leveldb/opt"
) )
@@ -65,13 +65,13 @@ func main() {
fmt.Printf("[block] F:%q H:%x N:%q I:%d\n", folder, hash, name, binary.BigEndian.Uint32(it.Value())) fmt.Printf("[block] F:%q H:%x N:%q I:%d\n", folder, hash, name, binary.BigEndian.Uint32(it.Value()))
case db.KeyTypeDeviceStatistic: case db.KeyTypeDeviceStatistic:
fmt.Printf("[dstat]\n %x\n %x\n", it.Key(), it.Value()) fmt.Printf("[dstat]\n %x\n %x", it.Key(), it.Value())
case db.KeyTypeFolderStatistic: case db.KeyTypeFolderStatistic:
fmt.Printf("[fstat]\n %x\n %x\n", it.Key(), it.Value()) fmt.Printf("[fstat]\n %x\n %x", it.Key(), it.Value())
default: default:
fmt.Printf("[???]\n %x\n %x\n", it.Key(), it.Value()) fmt.Printf("[???]\n %x\n %x", it.Key(), it.Value())
} }
} }
} }
-98
View File
@@ -1,98 +0,0 @@
// Copyright (C) 2015 The Syncthing Authors.
//
// This Source Code Form is subject to the terms of the Mozilla Public
// License, v. 2.0. If a copy of the MPL was not distributed with this file,
// You can obtain one at http://mozilla.org/MPL/2.0/.
package main
import (
"bytes"
"crypto/md5"
"flag"
"fmt"
"io"
"os"
"time"
)
func getmd5(filePath string) ([]byte, error) {
var result []byte
file, err := os.Open(filePath)
if err != nil {
return result, err
}
defer file.Close()
hash := md5.New()
if _, err := io.Copy(hash, file); err != nil {
return result, err
}
return hash.Sum(result), nil
}
func main() {
period := flag.Duration("period", 200*time.Millisecond, "Sleep period between checks")
flag.Parse()
file := flag.Arg(0)
if file == "" {
fmt.Println("Expects a path as an argument")
return
}
exists := true
size := int64(0)
mtime := time.Time{}
hash := []byte{}
for {
time.Sleep(*period)
newExists := true
fi, err := os.Stat(file)
if err != nil && os.IsNotExist(err) {
newExists = false
} else if err != nil {
fmt.Println("stat:", err)
return
}
if newExists != exists {
exists = newExists
if !newExists {
fmt.Println(file, "does not exist")
} else {
fmt.Println(file, "appeared")
}
}
if !exists {
size = 0
mtime = time.Time{}
hash = []byte{}
continue
}
if fi.IsDir() {
fmt.Println(file, "is directory")
return
}
newSize := fi.Size()
newMtime := fi.ModTime()
newHash, err := getmd5(file)
if err != nil {
fmt.Println("getmd5:", err)
}
if newSize != size || newMtime != mtime || !bytes.Equal(newHash, hash) {
fmt.Println(file, "Size:", newSize, "Mtime:", newMtime, "Hash:", fmt.Sprintf("%x", newHash))
hash = newHash
size = newSize
mtime = newMtime
}
}
}
+1 -1
View File
@@ -10,7 +10,7 @@ import (
"encoding/json" "encoding/json"
"io" "io"
"github.com/syncthing/syncthing/lib/events" "github.com/syncthing/syncthing/internal/events"
) )
// The auditSvc subscribes to events and writes these in JSON format, one // The auditSvc subscribes to events and writes these in JSON format, one
+1 -1
View File
@@ -12,7 +12,7 @@ import (
"testing" "testing"
"time" "time"
"github.com/syncthing/syncthing/lib/events" "github.com/syncthing/syncthing/internal/events"
) )
func TestAuditService(t *testing.T) { func TestAuditService(t *testing.T) {
+7 -24
View File
@@ -15,9 +15,9 @@ import (
"time" "time"
"github.com/syncthing/protocol" "github.com/syncthing/protocol"
"github.com/syncthing/syncthing/lib/config" "github.com/syncthing/syncthing/internal/config"
"github.com/syncthing/syncthing/lib/events" "github.com/syncthing/syncthing/internal/events"
"github.com/syncthing/syncthing/lib/model" "github.com/syncthing/syncthing/internal/model"
"github.com/thejerf/suture" "github.com/thejerf/suture"
) )
@@ -177,6 +177,10 @@ next:
if debugNet { if debugNet {
l.Debugf("cipher suite: %04X in lan: %t", conn.ConnectionState().CipherSuite, !limit) l.Debugf("cipher suite: %04X in lan: %t", conn.ConnectionState().CipherSuite, !limit)
} }
events.Default.Log(events.DeviceConnected, map[string]string{
"id": remoteID.String(),
"addr": conn.RemoteAddr().String(),
})
s.model.AddConnection(conn, protoConn) s.model.AddConnection(conn, protoConn)
continue next continue next
@@ -349,24 +353,3 @@ func (s *connectionSvc) shouldLimit(addr net.Addr) bool {
} }
return !tcpaddr.IP.IsLoopback() return !tcpaddr.IP.IsLoopback()
} }
func (s *connectionSvc) VerifyConfiguration(from, to config.Configuration) error {
return nil
}
func (s *connectionSvc) CommitConfiguration(from, to config.Configuration) bool {
// We require a restart if a device as been removed.
newDevices := make(map[protocol.DeviceID]bool, len(to.Devices))
for _, dev := range to.Devices {
newDevices[dev.DeviceID] = true
}
for _, dev := range from.Devices {
if !newDevices[dev.DeviceID] {
return false
}
}
return true
}
+2 -3
View File
@@ -12,7 +12,6 @@ import (
) )
var ( var (
debugNet = strings.Contains(os.Getenv("STTRACE"), "net") || os.Getenv("STTRACE") == "all" debugNet = strings.Contains(os.Getenv("STTRACE"), "net") || os.Getenv("STTRACE") == "all"
debugHTTP = strings.Contains(os.Getenv("STTRACE"), "http") || os.Getenv("STTRACE") == "all" debugHTTP = strings.Contains(os.Getenv("STTRACE"), "http") || os.Getenv("STTRACE") == "all"
debugSuture = strings.Contains(os.Getenv("STTRACE"), "suture") || os.Getenv("STTRACE") == "all"
) )
+66 -130
View File
@@ -26,15 +26,15 @@ import (
"github.com/calmh/logger" "github.com/calmh/logger"
"github.com/syncthing/protocol" "github.com/syncthing/protocol"
"github.com/syncthing/syncthing/lib/auto" "github.com/syncthing/syncthing/internal/auto"
"github.com/syncthing/syncthing/lib/config" "github.com/syncthing/syncthing/internal/config"
"github.com/syncthing/syncthing/lib/db" "github.com/syncthing/syncthing/internal/db"
"github.com/syncthing/syncthing/lib/discover" "github.com/syncthing/syncthing/internal/discover"
"github.com/syncthing/syncthing/lib/events" "github.com/syncthing/syncthing/internal/events"
"github.com/syncthing/syncthing/lib/model" "github.com/syncthing/syncthing/internal/model"
"github.com/syncthing/syncthing/lib/osutil" "github.com/syncthing/syncthing/internal/osutil"
"github.com/syncthing/syncthing/lib/sync" "github.com/syncthing/syncthing/internal/sync"
"github.com/syncthing/syncthing/lib/upgrade" "github.com/syncthing/syncthing/internal/upgrade"
"github.com/vitrun/qart/qr" "github.com/vitrun/qart/qr"
"golang.org/x/crypto/bcrypt" "golang.org/x/crypto/bcrypt"
) )
@@ -49,36 +49,31 @@ var (
guiErrors = []guiError{} guiErrors = []guiError{}
guiErrorsMut = sync.NewMutex() guiErrorsMut = sync.NewMutex()
startTime = time.Now() startTime = time.Now()
eventSub *events.BufferedSubscription
) )
type apiSvc struct { type apiSvc struct {
id protocol.DeviceID cfg config.GUIConfiguration
cfg config.GUIConfiguration assetDir string
assetDir string model *model.Model
model *model.Model fss *folderSummarySvc
listener net.Listener listener net.Listener
fss *folderSummarySvc
stop chan struct{}
systemConfigMut sync.Mutex
eventSub *events.BufferedSubscription
} }
func newAPISvc(id protocol.DeviceID, cfg config.GUIConfiguration, assetDir string, m *model.Model, eventSub *events.BufferedSubscription) (*apiSvc, error) { func newAPISvc(cfg config.GUIConfiguration, assetDir string, m *model.Model) (*apiSvc, error) {
svc := &apiSvc{ svc := &apiSvc{
id: id, cfg: cfg,
cfg: cfg, assetDir: assetDir,
assetDir: assetDir, model: m,
model: m, fss: newFolderSummarySvc(m),
systemConfigMut: sync.NewMutex(),
eventSub: eventSub,
} }
var err error var err error
svc.listener, err = svc.getListener(cfg) svc.listener, err = svc.getListener()
return svc, err return svc, err
} }
func (s *apiSvc) getListener(cfg config.GUIConfiguration) (net.Listener, error) { func (s *apiSvc) getListener() (net.Listener, error) {
cert, err := tls.LoadX509KeyPair(locations[locHTTPSCertFile], locations[locHTTPSKeyFile]) cert, err := tls.LoadX509KeyPair(locations[locHTTPSCertFile], locations[locHTTPSKeyFile])
if err != nil { if err != nil {
l.Infoln("Loading HTTPS certificate:", err) l.Infoln("Loading HTTPS certificate:", err)
@@ -115,7 +110,7 @@ func (s *apiSvc) getListener(cfg config.GUIConfiguration) (net.Listener, error)
}, },
} }
rawListener, err := net.Listen("tcp", cfg.Address) rawListener, err := net.Listen("tcp", s.cfg.Address)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -125,9 +120,10 @@ func (s *apiSvc) getListener(cfg config.GUIConfiguration) (net.Listener, error)
} }
func (s *apiSvc) Serve() { func (s *apiSvc) Serve() {
s.stop = make(chan struct{})
l.AddHandler(logger.LevelWarn, s.showGuiError) l.AddHandler(logger.LevelWarn, s.showGuiError)
sub := events.Default.Subscribe(events.AllEvents)
eventSub = events.NewBufferedSubscription(sub, 1000)
defer events.Default.Unsubscribe(sub)
// The GET handlers // The GET handlers
getRestMux := http.NewServeMux() getRestMux := http.NewServeMux()
@@ -190,14 +186,14 @@ func (s *apiSvc) Serve() {
// Wrap everything in CSRF protection. The /rest prefix should be // Wrap everything in CSRF protection. The /rest prefix should be
// protected, other requests will grant cookies. // protected, other requests will grant cookies.
handler := csrfMiddleware(s.id.String()[:5], "/rest", s.cfg.APIKey, mux) handler := csrfMiddleware("/rest", s.cfg.APIKey, mux)
// Add our version and ID as a header to responses // Add our version as a header to responses
handler = withDetailsMiddleware(s.id, handler) handler = withVersionMiddleware(handler)
// Wrap everything in basic auth, if user/password is set. // Wrap everything in basic auth, if user/password is set.
if len(s.cfg.User) > 0 && len(s.cfg.Password) > 0 { if len(s.cfg.User) > 0 && len(s.cfg.Password) > 0 {
handler = basicAuthAndSessionMiddleware("sessionid-"+s.id.String()[:5], s.cfg, handler) handler = basicAuthAndSessionMiddleware(s.cfg, handler)
} }
// Redirect to HTTPS if we are supposed to // Redirect to HTTPS if we are supposed to
@@ -214,63 +210,15 @@ func (s *apiSvc) Serve() {
ReadTimeout: 10 * time.Second, ReadTimeout: 10 * time.Second,
} }
s.fss = newFolderSummarySvc(s.model)
defer s.fss.Stop()
s.fss.ServeBackground() s.fss.ServeBackground()
l.Infoln("API listening on", s.listener.Addr())
err := srv.Serve(s.listener) err := srv.Serve(s.listener)
l.Warnln("API:", err)
// The return could be due to an intentional close. Wait for the stop
// signal before returning. IF there is no stop signal within a second, we
// assume it was unintentional and log the error before retrying.
select {
case <-s.stop:
case <-time.After(time.Second):
l.Warnln("API:", err)
}
} }
func (s *apiSvc) Stop() { func (s *apiSvc) Stop() {
close(s.stop)
s.listener.Close() s.listener.Close()
} s.fss.Stop()
func (s *apiSvc) String() string {
return fmt.Sprintf("apiSvc@%p", s)
}
func (s *apiSvc) VerifyConfiguration(from, to config.Configuration) error {
return nil
}
func (s *apiSvc) CommitConfiguration(from, to config.Configuration) bool {
if to.GUI == from.GUI {
return true
}
// Order here is important. We must close the listener to stop Serve(). We
// must create a new listener before Serve() starts again. We can't create
// a new listener on the same port before the previous listener is closed.
// To assist in this little dance the Serve() method will wait for a
// signal on the stop channel after the listener has closed.
s.listener.Close()
var err error
s.listener, err = s.getListener(to.GUI)
if err != nil {
// Ideally this should be a verification error, but we check it by
// creating a new listener which requires shutting down the previous
// one first, which is too destructive for the VerifyConfiguration
// method.
return false
}
s.cfg = to.GUI
close(s.stop)
return true
} }
func getPostHandler(get, post http.Handler) http.Handler { func getPostHandler(get, post http.Handler) http.Handler {
@@ -336,10 +284,9 @@ func noCacheMiddleware(h http.Handler) http.Handler {
}) })
} }
func withDetailsMiddleware(id protocol.DeviceID, h http.Handler) http.Handler { func withVersionMiddleware(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("X-Syncthing-Version", Version) w.Header().Set("X-Syncthing-Version", Version)
w.Header().Set("X-Syncthing-ID", id.String())
h.ServeHTTP(w, r) h.ServeHTTP(w, r)
}) })
} }
@@ -428,10 +375,7 @@ func folderSummary(m *model.Model, folder string) map[string]interface{} {
res["error"] = err.Error() res["error"] = err.Error()
} }
lv, _ := m.CurrentLocalVersion(folder) res["version"] = m.CurrentLocalVersion(folder) + m.RemoteLocalVersion(folder)
rv, _ := m.RemoteLocalVersion(folder)
res["version"] = lv + rv
ignorePatterns, _, _ := m.GetIgnores(folder) ignorePatterns, _, _ := m.GetIgnores(folder)
res["ignorePatterns"] = false res["ignorePatterns"] = false
@@ -520,46 +464,49 @@ func (s *apiSvc) getSystemConfig(w http.ResponseWriter, r *http.Request) {
} }
func (s *apiSvc) postSystemConfig(w http.ResponseWriter, r *http.Request) { func (s *apiSvc) postSystemConfig(w http.ResponseWriter, r *http.Request) {
s.systemConfigMut.Lock() var newCfg config.Configuration
defer s.systemConfigMut.Unlock() err := json.NewDecoder(r.Body).Decode(&newCfg)
var to config.Configuration
err := json.NewDecoder(r.Body).Decode(&to)
if err != nil { if err != nil {
l.Warnln("decoding posted config:", err) l.Warnln("decoding posted config:", err)
http.Error(w, err.Error(), 500) http.Error(w, err.Error(), 500)
return return
} }
if to.GUI.Password != cfg.GUI().Password { if newCfg.GUI.Password != cfg.GUI().Password {
if to.GUI.Password != "" { if newCfg.GUI.Password != "" {
hash, err := bcrypt.GenerateFromPassword([]byte(to.GUI.Password), 0) hash, err := bcrypt.GenerateFromPassword([]byte(newCfg.GUI.Password), 0)
if err != nil { if err != nil {
l.Warnln("bcrypting password:", err) l.Warnln("bcrypting password:", err)
http.Error(w, err.Error(), 500) http.Error(w, err.Error(), 500)
return return
} }
to.GUI.Password = string(hash) newCfg.GUI.Password = string(hash)
} }
} }
// Fixup usage reporting settings // Start or stop usage reporting as appropriate
if curAcc := cfg.Options().URAccepted; to.Options.URAccepted > curAcc { if curAcc := cfg.Options().URAccepted; newCfg.Options.URAccepted > curAcc {
// UR was enabled // UR was enabled
to.Options.URAccepted = usageReportVersion newCfg.Options.URAccepted = usageReportVersion
to.Options.URUniqueID = randomString(8) newCfg.Options.URUniqueID = randomString(8)
} else if to.Options.URAccepted < curAcc { err := sendUsageReport(s.model)
if err != nil {
l.Infoln("Usage report:", err)
}
go usageReportingLoop(s.model)
} else if newCfg.Options.URAccepted < curAcc {
// UR was disabled // UR was disabled
to.Options.URAccepted = -1 newCfg.Options.URAccepted = -1
to.Options.URUniqueID = "" newCfg.Options.URUniqueID = ""
stopUsageReporting()
} }
// Activate and save // Activate and save
resp := cfg.Replace(to) configInSync = !config.ChangeRequiresRestart(cfg.Raw(), newCfg)
configInSync = !resp.RequiresRestart cfg.Replace(newCfg)
cfg.Save() cfg.Save()
} }
@@ -576,26 +523,21 @@ func (s *apiSvc) postSystemRestart(w http.ResponseWriter, r *http.Request) {
func (s *apiSvc) postSystemReset(w http.ResponseWriter, r *http.Request) { func (s *apiSvc) postSystemReset(w http.ResponseWriter, r *http.Request) {
var qs = r.URL.Query() var qs = r.URL.Query()
folder := qs.Get("folder") folder := qs.Get("folder")
var err error
if len(folder) > 0 { if len(folder) == 0 {
if _, ok := cfg.Folders()[folder]; !ok { err = resetDB()
http.Error(w, "Invalid folder ID", 500) } else {
return err = s.model.ResetFolder(folder)
}
} }
if err != nil {
http.Error(w, err.Error(), 500)
return
}
if len(folder) == 0 { if len(folder) == 0 {
// Reset all folders.
for folder := range cfg.Folders() {
s.model.ResetFolder(folder)
}
s.flushResponse(`{"ok": "resetting database"}`, w) s.flushResponse(`{"ok": "resetting database"}`, w)
} else { } else {
// Reset a specific folder, assuming it's supposed to exist. s.flushResponse(`{"ok": "resetting folder " + folder}`, w)
s.model.ResetFolder(folder)
s.flushResponse(`{"ok": "resetting folder `+folder+`"}`, w)
} }
go restart() go restart()
} }
@@ -752,7 +694,7 @@ func (s *apiSvc) getEvents(w http.ResponseWriter, r *http.Request) {
f := w.(http.Flusher) f := w.(http.Flusher)
f.Flush() f.Flush()
evs := s.eventSub.Since(since, nil) evs := eventSub.Since(since, nil)
if 0 < limit && limit < len(evs) { if 0 < limit && limit < len(evs) {
evs = evs[len(evs)-limit:] evs = evs[len(evs)-limit:]
} }
@@ -955,11 +897,6 @@ func (s embeddedStatic) ServeHTTP(w http.ResponseWriter, r *http.Request) {
return return
} }
if r.Header.Get("If-Modified-Since") == auto.AssetsBuildDate {
w.WriteHeader(http.StatusNotModified)
return
}
mtype := s.mimeTypeForFile(file) mtype := s.mimeTypeForFile(file)
if len(mtype) != 0 { if len(mtype) != 0 {
w.Header().Set("Content-Type", mtype) w.Header().Set("Content-Type", mtype)
@@ -975,7 +912,6 @@ func (s embeddedStatic) ServeHTTP(w http.ResponseWriter, r *http.Request) {
} }
w.Header().Set("Content-Length", fmt.Sprintf("%d", len(bs))) w.Header().Set("Content-Length", fmt.Sprintf("%d", len(bs)))
w.Header().Set("Last-Modified", auto.AssetsBuildDate) w.Header().Set("Last-Modified", auto.AssetsBuildDate)
w.Header().Set("Cache-Control", "public")
w.Write(bs) w.Write(bs)
} }
+5 -6
View File
@@ -14,8 +14,8 @@ import (
"strings" "strings"
"time" "time"
"github.com/syncthing/syncthing/lib/config" "github.com/syncthing/syncthing/internal/config"
"github.com/syncthing/syncthing/lib/sync" "github.com/syncthing/syncthing/internal/sync"
"golang.org/x/crypto/bcrypt" "golang.org/x/crypto/bcrypt"
) )
@@ -24,15 +24,14 @@ var (
sessionsMut = sync.NewMutex() sessionsMut = sync.NewMutex()
) )
func basicAuthAndSessionMiddleware(cookieName string, cfg config.GUIConfiguration, next http.Handler) http.Handler { func basicAuthAndSessionMiddleware(cfg config.GUIConfiguration, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if cfg.APIKey != "" && r.Header.Get("X-API-Key") == cfg.APIKey { if cfg.APIKey != "" && r.Header.Get("X-API-Key") == cfg.APIKey {
next.ServeHTTP(w, r) next.ServeHTTP(w, r)
return return
} }
cookie, err := r.Cookie(cookieName) cookie, err := r.Cookie("sessionid")
if err == nil && cookie != nil { if err == nil && cookie != nil {
sessionsMut.Lock() sessionsMut.Lock()
_, ok := sessions[cookie.Value] _, ok := sessions[cookie.Value]
@@ -87,7 +86,7 @@ func basicAuthAndSessionMiddleware(cookieName string, cfg config.GUIConfiguratio
sessions[sessionid] = true sessions[sessionid] = true
sessionsMut.Unlock() sessionsMut.Unlock()
http.SetCookie(w, &http.Cookie{ http.SetCookie(w, &http.Cookie{
Name: cookieName, Name: "sessionid",
Value: sessionid, Value: sessionid,
MaxAge: 0, MaxAge: 0,
}) })
+24 -15
View File
@@ -12,9 +12,10 @@ import (
"net/http" "net/http"
"os" "os"
"strings" "strings"
"time"
"github.com/syncthing/syncthing/lib/osutil" "github.com/syncthing/syncthing/internal/osutil"
"github.com/syncthing/syncthing/lib/sync" "github.com/syncthing/syncthing/internal/sync"
) )
var csrfTokens []string var csrfTokens []string
@@ -23,7 +24,7 @@ var csrfMut = sync.NewMutex()
// Check for CSRF token on /rest/ URLs. If a correct one is not given, reject // Check for CSRF token on /rest/ URLs. If a correct one is not given, reject
// the request with 403. For / and /index.html, set a new CSRF cookie if none // the request with 403. For / and /index.html, set a new CSRF cookie if none
// is currently set. // is currently set.
func csrfMiddleware(unique, prefix, apiKey string, next http.Handler) http.Handler { func csrfMiddleware(prefix, apiKey string, next http.Handler) http.Handler {
loadCsrfTokens() loadCsrfTokens()
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Allow requests carrying a valid API key // Allow requests carrying a valid API key
@@ -34,10 +35,10 @@ func csrfMiddleware(unique, prefix, apiKey string, next http.Handler) http.Handl
// Allow requests for the front page, and set a CSRF cookie if there isn't already a valid one. // Allow requests for the front page, and set a CSRF cookie if there isn't already a valid one.
if !strings.HasPrefix(r.URL.Path, prefix) { if !strings.HasPrefix(r.URL.Path, prefix) {
cookie, err := r.Cookie("CSRF-Token-" + unique) cookie, err := r.Cookie("CSRF-Token")
if err != nil || !validCsrfToken(cookie.Value) { if err != nil || !validCsrfToken(cookie.Value) {
cookie = &http.Cookie{ cookie = &http.Cookie{
Name: "CSRF-Token-" + unique, Name: "CSRF-Token",
Value: newCsrfToken(), Value: newCsrfToken(),
} }
http.SetCookie(w, cookie) http.SetCookie(w, cookie)
@@ -53,7 +54,7 @@ func csrfMiddleware(unique, prefix, apiKey string, next http.Handler) http.Handl
} }
// Verify the CSRF token // Verify the CSRF token
token := r.Header.Get("X-CSRF-Token-" + unique) token := r.Header.Get("X-CSRF-Token")
if !validCsrfToken(token) { if !validCsrfToken(token) {
http.Error(w, "CSRF Error", 403) http.Error(w, "CSRF Error", 403)
return return
@@ -90,20 +91,28 @@ func newCsrfToken() string {
} }
func saveCsrfTokens() { func saveCsrfTokens() {
// We're ignoring errors in here. It's not super critical and there's
// nothing relevant we can do about them anyway...
name := locations[locCsrfTokens] name := locations[locCsrfTokens]
f, err := osutil.CreateAtomic(name, 0600) tmp := fmt.Sprintf("%s.tmp.%d", name, time.Now().UnixNano())
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0644)
if err != nil {
return
}
defer os.Remove(tmp)
for _, t := range csrfTokens {
_, err := fmt.Fprintln(f, t)
if err != nil {
return
}
}
err = f.Close()
if err != nil { if err != nil {
return return
} }
for _, t := range csrfTokens { osutil.Rename(tmp, name)
fmt.Fprintln(f, t)
}
f.Close()
} }
func loadCsrfTokens() { func loadCsrfTokens() {
+1 -3
View File
@@ -13,7 +13,7 @@ import (
"strings" "strings"
"time" "time"
"github.com/syncthing/syncthing/lib/osutil" "github.com/syncthing/syncthing/internal/osutil"
) )
type locationEnum string type locationEnum string
@@ -31,7 +31,6 @@ const (
locCsrfTokens = "csrfTokens" locCsrfTokens = "csrfTokens"
locPanicLog = "panicLog" locPanicLog = "panicLog"
locAuditLog = "auditLog" locAuditLog = "auditLog"
locGUIAssets = "GUIAssets"
locDefFolder = "defFolder" locDefFolder = "defFolder"
) )
@@ -53,7 +52,6 @@ var locations = map[locationEnum]string{
locCsrfTokens: "${config}/csrftokens.txt", locCsrfTokens: "${config}/csrftokens.txt",
locPanicLog: "${config}/panic-${timestamp}.log", locPanicLog: "${config}/panic-${timestamp}.log",
locAuditLog: "${config}/audit-${timestamp}.log", locAuditLog: "${config}/audit-${timestamp}.log",
locGUIAssets: "${config}/gui",
locDefFolder: "${home}/Sync", locDefFolder: "${home}/Sync",
} }
+60 -89
View File
@@ -28,14 +28,14 @@ import (
"github.com/calmh/logger" "github.com/calmh/logger"
"github.com/juju/ratelimit" "github.com/juju/ratelimit"
"github.com/syncthing/protocol" "github.com/syncthing/protocol"
"github.com/syncthing/syncthing/lib/config" "github.com/syncthing/syncthing/internal/config"
"github.com/syncthing/syncthing/lib/db" "github.com/syncthing/syncthing/internal/db"
"github.com/syncthing/syncthing/lib/discover" "github.com/syncthing/syncthing/internal/discover"
"github.com/syncthing/syncthing/lib/events" "github.com/syncthing/syncthing/internal/events"
"github.com/syncthing/syncthing/lib/model" "github.com/syncthing/syncthing/internal/model"
"github.com/syncthing/syncthing/lib/osutil" "github.com/syncthing/syncthing/internal/osutil"
"github.com/syncthing/syncthing/lib/symlinks" "github.com/syncthing/syncthing/internal/symlinks"
"github.com/syncthing/syncthing/lib/upgrade" "github.com/syncthing/syncthing/internal/upgrade"
"github.com/syndtr/goleveldb/leveldb" "github.com/syndtr/goleveldb/leveldb"
"github.com/syndtr/goleveldb/leveldb/errors" "github.com/syndtr/goleveldb/leveldb/errors"
"github.com/syndtr/goleveldb/leveldb/opt" "github.com/syndtr/goleveldb/leveldb/opt"
@@ -155,7 +155,6 @@ are mostly useful for developers. Use with care.
- "model" (the model package) - "model" (the model package)
- "scanner" (the scanner package) - "scanner" (the scanner package)
- "stats" (the stats package) - "stats" (the stats package)
- "suture" (the suture package; service management)
- "upnp" (the upnp package) - "upnp" (the upnp package)
- "xdr" (the xdr package) - "xdr" (the xdr package)
- "all" (all of the above) - "all" (all of the above)
@@ -252,10 +251,6 @@ func main() {
l.Fatalln(err) l.Fatalln(err)
} }
if guiAssets == "" {
guiAssets = locations[locGUIAssets]
}
if runtime.GOOS == "windows" { if runtime.GOOS == "windows" {
if logFile == "" { if logFile == "" {
// Use the default log file location // Use the default log file location
@@ -284,7 +279,7 @@ func main() {
l.Fatalln(dir, "is not a directory") l.Fatalln(dir, "is not a directory")
} }
if err != nil && os.IsNotExist(err) { if err != nil && os.IsNotExist(err) {
err = osutil.MkdirAll(dir, 0700) err = os.MkdirAll(dir, 0700)
if err != nil { if err != nil {
l.Fatalln("generate:", err) l.Fatalln("generate:", err)
} }
@@ -425,12 +420,11 @@ func upgradeViaRest() error {
func syncthingMain() { func syncthingMain() {
// Create a main service manager. We'll add things to this as we go along. // Create a main service manager. We'll add things to this as we go along.
// We want any logging it does to go through our log system. // We want any logging it does to go through our log system, with INFO
// severity.
mainSvc := suture.New("main", suture.Spec{ mainSvc := suture.New("main", suture.Spec{
Log: func(line string) { Log: func(line string) {
if debugSuture { l.Infoln(line)
l.Debugln(line)
}
}, },
}) })
mainSvc.ServeBackground() mainSvc.ServeBackground()
@@ -447,17 +441,11 @@ func syncthingMain() {
mainSvc.Add(newVerboseSvc()) mainSvc.Add(newVerboseSvc())
} }
// Event subscription for the API; must start early to catch the early events.
apiSub := events.NewBufferedSubscription(events.Default.Subscribe(events.AllEvents), 1000)
if len(os.Getenv("GOMAXPROCS")) == 0 { if len(os.Getenv("GOMAXPROCS")) == 0 {
runtime.GOMAXPROCS(runtime.NumCPU()) runtime.GOMAXPROCS(runtime.NumCPU())
} }
// Attempt to increase the limit on number of open files to the maximum events.Default.Log(events.Starting, map[string]string{"home": baseDirs["config"]})
// allowed, in case we have many peers. We don't really care enough to
// report the error if there is one.
osutil.MaximizeOpenFileLimit()
// Ensure that that we have a certificate and key. // Ensure that that we have a certificate and key.
cert, err := tls.LoadX509KeyPair(locations[locCertFile], locations[locKeyFile]) cert, err := tls.LoadX509KeyPair(locations[locCertFile], locations[locKeyFile])
@@ -478,13 +466,6 @@ func syncthingMain() {
l.Infoln(LongVersion) l.Infoln(LongVersion)
l.Infoln("My ID:", myID) l.Infoln("My ID:", myID)
// Emit the Starting event, now that we know who we are.
events.Default.Log(events.Starting, map[string]string{
"home": baseDirs["config"],
"myID": myID.String(),
})
// Prepare to be able to save configuration // Prepare to be able to save configuration
cfgFile := locations[locConfigFile] cfgFile := locations[locConfigFile]
@@ -570,9 +551,6 @@ func syncthingMain() {
symlinks.Supported = false symlinks.Supported = false
} }
protocol.PingTimeout = time.Duration(opts.PingTimeoutS) * time.Second
protocol.PingIdleTime = time.Duration(opts.PingIdleTimeS) * time.Second
if opts.MaxSendKbps > 0 { if opts.MaxSendKbps > 0 {
writeRateLimit = ratelimit.NewBucketWithRate(float64(1000*opts.MaxSendKbps), int64(5*1000*opts.MaxSendKbps)) writeRateLimit = ratelimit.NewBucketWithRate(float64(1000*opts.MaxSendKbps), int64(5*1000*opts.MaxSendKbps))
} }
@@ -608,7 +586,6 @@ func syncthingMain() {
} }
m := model.NewModel(cfg, myID, myName, "syncthing", Version, ldb) m := model.NewModel(cfg, myID, myName, "syncthing", Version, ldb)
cfg.Subscribe(m)
if t := os.Getenv("STDEADLOCKTIMEOUT"); len(t) > 0 { if t := os.Getenv("STDEADLOCKTIMEOUT"); len(t) > 0 {
it, err := strconv.Atoi(t) it, err := strconv.Atoi(t)
@@ -619,6 +596,10 @@ func syncthingMain() {
m.StartDeadlockDetector(20 * 60 * time.Second) m.StartDeadlockDetector(20 * 60 * time.Second)
} }
// GUI
setupGUI(mainSvc, cfg, m)
// Clear out old indexes for other devices. Otherwise we'll start up and // Clear out old indexes for other devices. Otherwise we'll start up and
// start needing a bunch of files which are nowhere to be found. This // start needing a bunch of files which are nowhere to be found. This
// needs to be changed when we correctly do persistent indexes. // needs to be changed when we correctly do persistent indexes.
@@ -630,21 +611,8 @@ func syncthingMain() {
} }
m.Index(device, folderCfg.ID, nil, 0, nil) m.Index(device, folderCfg.ID, nil, 0, nil)
} }
// Routine to pull blocks from other devices to synchronize the local
// folder. Does not run when we are in read only (publish only) mode.
if folderCfg.ReadOnly {
m.StartFolderRO(folderCfg.ID)
} else {
m.StartFolderRW(folderCfg.ID)
}
} }
mainSvc.Add(m)
// GUI
setupGUI(mainSvc, cfg, m, apiSub)
// The default port we announce, possibly modified by setupUPnP next. // The default port we announce, possibly modified by setupUPnP next.
addr, err := net.ResolveTCPAddr("tcp", opts.ListenAddress[0]) addr, err := net.ResolveTCPAddr("tcp", opts.ListenAddress[0])
@@ -666,15 +634,27 @@ func syncthingMain() {
} }
connectionSvc := newConnectionSvc(cfg, myID, m, tlsCfg) connectionSvc := newConnectionSvc(cfg, myID, m, tlsCfg)
cfg.Subscribe(connectionSvc)
mainSvc.Add(connectionSvc) mainSvc.Add(connectionSvc)
for _, folder := range cfg.Folders() {
// Routine to pull blocks from other devices to synchronize the local
// folder. Does not run when we are in read only (publish only) mode.
if folder.ReadOnly {
l.Okf("Ready to synchronize %s (read only; no external updates accepted)", folder.ID)
m.StartFolderRO(folder.ID)
} else {
l.Okf("Ready to synchronize %s (read-write)", folder.ID)
m.StartFolderRW(folder.ID)
}
}
if cpuProfile { if cpuProfile {
f, err := os.Create(fmt.Sprintf("cpu-%d.pprof", os.Getpid())) f, err := os.Create(fmt.Sprintf("cpu-%d.pprof", os.Getpid()))
if err != nil { if err != nil {
log.Fatal(err) log.Fatal(err)
} }
pprof.StartCPUProfile(f) pprof.StartCPUProfile(f)
defer pprof.StopCPUProfile()
} }
for _, device := range cfg.Devices() { for _, device := range cfg.Devices() {
@@ -697,13 +677,16 @@ func syncthingMain() {
cfg.SetOptions(opts) cfg.SetOptions(opts)
cfg.Save() cfg.Save()
} }
go usageReportingLoop(m)
go func() {
time.Sleep(10 * time.Minute)
err := sendUsageReport(m)
if err != nil {
l.Infoln("Usage report:", err)
}
}()
} }
// The usageReportingManager registers itself to listen to configuration
// changes, and there's nothing more we need to tell it from the outside.
// Hence we don't keep the returned pointer.
newUsageReportingManager(m, cfg)
if opts.RestartOnWakeup { if opts.RestartOnWakeup {
go standbyMonitor() go standbyMonitor()
} }
@@ -718,9 +701,7 @@ func syncthingMain() {
} }
} }
events.Default.Log(events.StartupComplete, map[string]string{ events.Default.Log(events.StartupComplete, nil)
"myID": myID.String(),
})
go generatePingEvents() go generatePingEvents()
cleanConfigDirectory() cleanConfigDirectory()
@@ -730,37 +711,29 @@ func syncthingMain() {
mainSvc.Stop() mainSvc.Stop()
l.Okln("Exiting") l.Okln("Exiting")
if cpuProfile {
pprof.StopCPUProfile()
}
os.Exit(code) os.Exit(code)
} }
func dbOpts() *opt.Options { func dbOpts() *opt.Options {
// Calculate a suitable database block cache capacity. // Calculate a sutiable database block cache capacity. We start at the
// default of 8 MiB and use larger values for machines with more memory.
// In reality, the database will use twice the amount we calculate here,
// as it also has two write buffers each sized at half the block cache.
// Default is 8 MiB.
blockCacheCapacity := 8 << 20 blockCacheCapacity := 8 << 20
// Increase block cache up to this maximum: if bytes, err := memorySize(); err == nil {
const maxCapacity = 64 << 20 if bytes > 74<<30 {
// ... which we reach when the box has this much RAM: // At 74 GiB of RAM, we hit a 256 MiB block cache (per the
const maxAtRAM = 8 << 30 // calculations below). There's probably no point in growing the
// cache beyond this point.
if v := cfg.Options().DatabaseBlockCacheMiB; v != 0 { blockCacheCapacity = 256 << 20
// Use the value from the config, if it's set. } else if bytes > 8<<30 {
blockCacheCapacity = v << 20 // Slowly grow from 128 MiB at 8 GiB of RAM up to 256 MiB for a
} else if bytes, err := memorySize(); err == nil { // ~74 GiB RAM machine
// We start at the default of 8 MiB and use larger values for machines blockCacheCapacity = int(bytes/512) + 128 - 16
// with more memory. } else if bytes > 512<<20 {
// Grow from 8 MiB at start to 128 MiB of cache at 8 GiB of RAM.
if bytes > maxAtRAM { blockCacheCapacity = int(bytes / 64)
// Cap the cache at maxCapacity when we reach maxAtRam amount of memory
blockCacheCapacity = maxCapacity
} else if bytes > maxAtRAM/maxCapacity*int64(blockCacheCapacity) {
// Grow from the default to maxCapacity at maxAtRam amount of memory
blockCacheCapacity = int(bytes * maxCapacity / maxAtRAM)
} }
l.Infoln("Database block cache capacity", blockCacheCapacity/1024, "KiB") l.Infoln("Database block cache capacity", blockCacheCapacity/1024, "KiB")
} }
@@ -768,7 +741,7 @@ func dbOpts() *opt.Options {
return &opt.Options{ return &opt.Options{
OpenFilesCacheCapacity: 100, OpenFilesCacheCapacity: 100,
BlockCacheCapacity: blockCacheCapacity, BlockCacheCapacity: blockCacheCapacity,
WriteBuffer: 4 << 20, WriteBuffer: blockCacheCapacity / 2,
} }
} }
@@ -789,7 +762,7 @@ func startAuditing(mainSvc *suture.Supervisor) {
l.Infoln("Audit log in", auditFile) l.Infoln("Audit log in", auditFile)
} }
func setupGUI(mainSvc *suture.Supervisor, cfg *config.Wrapper, m *model.Model, apiSub *events.BufferedSubscription) { func setupGUI(mainSvc *suture.Supervisor, cfg *config.Wrapper, m *model.Model) {
opts := cfg.Options() opts := cfg.Options()
guiCfg := overrideGUIConfig(cfg.GUI(), guiAddress, guiAuthentication, guiAPIKey) guiCfg := overrideGUIConfig(cfg.GUI(), guiAddress, guiAuthentication, guiAPIKey)
@@ -818,11 +791,10 @@ func setupGUI(mainSvc *suture.Supervisor, cfg *config.Wrapper, m *model.Model, a
urlShow := fmt.Sprintf("%s://%s/", proto, net.JoinHostPort(hostShow, strconv.Itoa(addr.Port))) urlShow := fmt.Sprintf("%s://%s/", proto, net.JoinHostPort(hostShow, strconv.Itoa(addr.Port)))
l.Infoln("Starting web GUI on", urlShow) l.Infoln("Starting web GUI on", urlShow)
api, err := newAPISvc(myID, guiCfg, guiAssets, m, apiSub) api, err := newAPISvc(guiCfg, guiAssets, m)
if err != nil { if err != nil {
l.Fatalln("Cannot start GUI:", err) l.Fatalln("Cannot start GUI:", err)
} }
cfg.Subscribe(api)
mainSvc.Add(api) mainSvc.Add(api)
if opts.StartBrowser && !noBrowser && !stRestarting { if opts.StartBrowser && !noBrowser && !stRestarting {
@@ -842,7 +814,6 @@ func defaultConfig(myName string) config.Configuration {
ID: "default", ID: "default",
RawPath: locations[locDefFolder], RawPath: locations[locDefFolder],
RescanIntervalS: 60, RescanIntervalS: 60,
MinDiskFreePct: 1,
Devices: []config.FolderDeviceConfiguration{{DeviceID: myID}}, Devices: []config.FolderDeviceConfiguration{{DeviceID: myID}},
}, },
} }
@@ -909,7 +880,7 @@ func discovery(extPort int) *discover.Discoverer {
func ensureDir(dir string, mode int) { func ensureDir(dir string, mode int) {
fi, err := os.Stat(dir) fi, err := os.Stat(dir)
if os.IsNotExist(err) { if os.IsNotExist(err) {
err := osutil.MkdirAll(dir, 0700) err := os.MkdirAll(dir, 0700)
if err != nil { if err != nil {
l.Fatalln(err) l.Fatalln(err)
} }
+3 -3
View File
@@ -11,9 +11,9 @@ import (
"testing" "testing"
"github.com/syncthing/protocol" "github.com/syncthing/protocol"
"github.com/syncthing/syncthing/lib/config" "github.com/syncthing/syncthing/internal/config"
"github.com/syncthing/syncthing/lib/db" "github.com/syncthing/syncthing/internal/db"
"github.com/syncthing/syncthing/lib/model" "github.com/syncthing/syncthing/internal/model"
"github.com/syndtr/goleveldb/leveldb" "github.com/syndtr/goleveldb/leveldb"
"github.com/syndtr/goleveldb/leveldb/storage" "github.com/syndtr/goleveldb/leveldb/storage"
+3 -4
View File
@@ -17,8 +17,8 @@ import (
"syscall" "syscall"
"time" "time"
"github.com/syncthing/syncthing/lib/osutil" "github.com/syncthing/syncthing/internal/osutil"
"github.com/syncthing/syncthing/lib/sync" "github.com/syncthing/syncthing/internal/sync"
) )
var ( var (
@@ -182,8 +182,7 @@ func copyStderr(stderr io.Reader, dst io.Writer) {
} }
l.Warnf("Panic detected, writing to \"%s\"", panicFd.Name()) l.Warnf("Panic detected, writing to \"%s\"", panicFd.Name())
l.Warnln("Please check for existing issues with similar panic message at https://github.com/syncthing/syncthing/issues/") l.Warnln("Please create an issue at https://github.com/syncthing/syncthing/issues/ with the panic log attached")
l.Warnln("If no issue with similar panic message exists, please create a new issue with the panic log attached")
stdoutMut.Lock() stdoutMut.Lock()
for _, line := range stdoutFirstLines { for _, line := range stdoutFirstLines {
+2 -6
View File
@@ -7,7 +7,6 @@
package main package main
import ( import (
"runtime"
"sync" "sync"
"testing" "testing"
) )
@@ -15,13 +14,10 @@ import (
var predictableRandomTest sync.Once var predictableRandomTest sync.Once
func TestPredictableRandom(t *testing.T) { func TestPredictableRandom(t *testing.T) {
if runtime.GOARCH != "amd64" {
t.Skip("Test only for 64 bit platforms; but if it works there, it should work on 32 bit")
}
predictableRandomTest.Do(func() { predictableRandomTest.Do(func() {
// predictable random sequence is predictable // predictable random sequence is predictable
e := int64(3440579354231278675) e := 3440579354231278675
if v := int64(predictableRandom.Int()); v != e { if v := predictableRandom.Int(); v != e {
t.Errorf("Unexpected random value %d != %d", v, e) t.Errorf("Unexpected random value %d != %d", v, e)
} }
}) })
+3 -3
View File
@@ -9,9 +9,9 @@ package main
import ( import (
"time" "time"
"github.com/syncthing/syncthing/lib/events" "github.com/syncthing/syncthing/internal/events"
"github.com/syncthing/syncthing/lib/model" "github.com/syncthing/syncthing/internal/model"
"github.com/syncthing/syncthing/lib/sync" "github.com/syncthing/syncthing/internal/sync"
"github.com/thejerf/suture" "github.com/thejerf/suture"
) )
-2
View File
@@ -102,9 +102,7 @@ func (l *DowngradingListener) Accept() (net.Conn, error) {
} }
br := bufio.NewReader(conn) br := bufio.NewReader(conn)
conn.SetReadDeadline(time.Now().Add(1 * time.Second))
bs, err := br.Peek(1) bs, err := br.Peek(1)
conn.SetReadDeadline(time.Time{})
if err != nil { if err != nil {
// We hit a read error here, but the Accept() call succeeded so we must not return an error. // We hit a read error here, but the Accept() call succeeded so we must not return an error.
// We return the connection as is and let whoever tries to use it deal with the error. // We return the connection as is and let whoever tries to use it deal with the error.
+4 -6
View File
@@ -10,8 +10,8 @@ import (
"fmt" "fmt"
"time" "time"
"github.com/syncthing/syncthing/lib/config" "github.com/syncthing/syncthing/internal/config"
"github.com/syncthing/syncthing/lib/upnp" "github.com/syncthing/syncthing/internal/upnp"
) )
// The UPnP service runs a loop for discovery of IGDs (Internet Gateway // The UPnP service runs a loop for discovery of IGDs (Internet Gateway
@@ -79,10 +79,8 @@ func (s *upnpSvc) tryIGDs(igds []upnp.IGD, prevExtPort int) int {
// External port changed; refresh the discovery announcement. // External port changed; refresh the discovery announcement.
// TODO: Don't reach out to some magic global here? // TODO: Don't reach out to some magic global here?
l.Infof("New UPnP port mapping: external port %d to local port %d.", extPort, s.localPort) l.Infof("New UPnP port mapping: external port %d to local port %d.", extPort, s.localPort)
if s.cfg.Options().GlobalAnnEnabled { discoverer.StopGlobal()
discoverer.StopGlobal() discoverer.StartGlobal(s.cfg.Options().GlobalAnnServers, uint16(extPort))
discoverer.StartGlobal(s.cfg.Options().GlobalAnnServers, uint16(extPort))
}
} }
if debugNet { if debugNet {
l.Debugf("Created/updated UPnP port mapping for external port %d on device %s.", extPort, igd.FriendlyIdentifier()) l.Debugf("Created/updated UPnP port mapping for external port %d on device %s.", extPort, igd.FriendlyIdentifier())
+19 -73
View File
@@ -11,15 +11,12 @@ import (
"crypto/rand" "crypto/rand"
"crypto/sha256" "crypto/sha256"
"encoding/json" "encoding/json"
"fmt"
"net" "net"
"net/http" "net/http"
"runtime" "runtime"
"time" "time"
"github.com/syncthing/syncthing/lib/config" "github.com/syncthing/syncthing/internal/model"
"github.com/syncthing/syncthing/lib/model"
"github.com/thejerf/suture"
) )
// Current version number of the usage report, for acceptance purposes. If // Current version number of the usage report, for acceptance purposes. If
@@ -27,54 +24,8 @@ import (
// are prompted for acceptance of the new report. // are prompted for acceptance of the new report.
const usageReportVersion = 1 const usageReportVersion = 1
type usageReportingManager struct { var stopUsageReportingCh = make(chan struct{})
model *model.Model
sup *suture.Supervisor
}
func newUsageReportingManager(m *model.Model, cfg *config.Wrapper) *usageReportingManager {
mgr := &usageReportingManager{
model: m,
}
// Start UR if it's enabled.
mgr.CommitConfiguration(config.Configuration{}, cfg.Raw())
// Listen to future config changes so that we can start and stop as
// appropriate.
cfg.Subscribe(mgr)
return mgr
}
func (m *usageReportingManager) VerifyConfiguration(from, to config.Configuration) error {
return nil
}
func (m *usageReportingManager) CommitConfiguration(from, to config.Configuration) bool {
if to.Options.URAccepted >= usageReportVersion && m.sup == nil {
// Usage reporting was turned on; lets start it.
svc := &usageReportingService{
model: m.model,
}
m.sup = suture.NewSimple("usageReporting")
m.sup.Add(svc)
m.sup.ServeBackground()
} else if to.Options.URAccepted < usageReportVersion && m.sup != nil {
// Usage reporting was turned off
m.sup.Stop()
m.sup = nil
}
return true
}
func (m *usageReportingManager) String() string {
return fmt.Sprintf("usageReportingManager@%p", m)
}
// reportData returns the data to be sent in a usage report. It's used in
// various places, so not part of the usageReportingSvc object.
func reportData(m *model.Model) map[string]interface{} { func reportData(m *model.Model) map[string]interface{} {
res := make(map[string]interface{}) res := make(map[string]interface{})
res["uniqueID"] = cfg.Options().URUniqueID res["uniqueID"] = cfg.Options().URUniqueID
@@ -124,13 +75,8 @@ func reportData(m *model.Model) map[string]interface{} {
return res return res
} }
type usageReportingService struct { func sendUsageReport(m *model.Model) error {
model *model.Model d := reportData(m)
stop chan struct{}
}
func (s *usageReportingService) sendUsageReport() error {
d := reportData(s.model)
var b bytes.Buffer var b bytes.Buffer
json.NewEncoder(&b).Encode(d) json.NewEncoder(&b).Encode(d)
@@ -148,32 +94,32 @@ func (s *usageReportingService) sendUsageReport() error {
return err return err
} }
func (s *usageReportingService) Serve() { func usageReportingLoop(m *model.Model) {
s.stop = make(chan struct{})
l.Infoln("Starting usage reporting") l.Infoln("Starting usage reporting")
defer l.Infoln("Stopping usage reporting") t := time.NewTicker(86400 * time.Second)
loop:
t := time.NewTimer(10 * time.Minute) // time to initial report at start
for { for {
select { select {
case <-s.stop: case <-stopUsageReportingCh:
return break loop
case <-t.C: case <-t.C:
err := s.sendUsageReport() err := sendUsageReport(m)
if err != nil { if err != nil {
l.Infoln("Usage report:", err) l.Infoln("Usage report:", err)
} }
t.Reset(24 * time.Hour) // next report tomorrow
} }
} }
l.Infoln("Stopping usage reporting")
}
func stopUsageReporting() {
select {
case stopUsageReportingCh <- struct{}{}:
default:
}
} }
func (s *usageReportingService) Stop() { // Returns CPU performance as a measure of single threaded SHA-256 MiB/s
close(s.stop)
}
// cpuBench returns CPU performance as a measure of single threaded SHA-256 MiB/s
func cpuBench() float64 { func cpuBench() float64 {
chunkSize := 100 * 1 << 10 chunkSize := 100 * 1 << 10
h := sha256.New() h := sha256.New()
+3 -6
View File
@@ -9,7 +9,7 @@ package main
import ( import (
"fmt" "fmt"
"github.com/syncthing/syncthing/lib/events" "github.com/syncthing/syncthing/internal/events"
) )
// The verbose logging service subscribes to events and prints these in // The verbose logging service subscribes to events and prints these in
@@ -102,11 +102,8 @@ func (s *verboseSvc) formatEvent(ev events.Event) string {
return fmt.Sprintf("Started syncing %q / %q (%v %v)", data["folder"], data["item"], data["action"], data["type"]) return fmt.Sprintf("Started syncing %q / %q (%v %v)", data["folder"], data["item"], data["action"], data["type"])
case events.ItemFinished: case events.ItemFinished:
data := ev.Data.(map[string]interface{}) data := ev.Data.(map[string]interface{})
if err, ok := data["error"].(*string); ok && err != nil { if err := data["err"]; err != nil {
// If the err interface{} is not nil, it is a string pointer. return fmt.Sprintf("Finished syncing %q / %q (%v %v): %v", data["folder"], data["item"], data["action"], data["type"], err)
// Dereference it to get the actual error or Sprintf will print
// the pointer value....
return fmt.Sprintf("Finished syncing %q / %q (%v %v): %v", data["folder"], data["item"], data["action"], data["type"], *err)
} }
return fmt.Sprintf("Finished syncing %q / %q (%v %v): Success", data["folder"], data["item"], data["action"], data["type"]) return fmt.Sprintf("Finished syncing %q / %q (%v %v): Success", data["folder"], data["item"], data["action"], data["type"])
@@ -51,8 +51,6 @@ func main() {
} }
resp.Body.Close() resp.Body.Close()
names := make(map[string]string)
var langs []string var langs []string
for code, stat := range stats { for code, stat := range stats {
code = strings.Replace(code, "_", "-", 1) code = strings.Replace(code, "_", "-", 1)
@@ -64,7 +62,6 @@ func main() {
} }
langs = append(langs, code) langs = append(langs, code)
names[code] = languageName(code)
if code == "en" { if code == "en" {
continue continue
} }
@@ -88,7 +85,6 @@ func main() {
} }
saveValidLangs(langs) saveValidLangs(langs)
saveLanguageNames(names)
} }
func saveValidLangs(langs []string) { func saveValidLangs(langs []string) {
@@ -102,16 +98,6 @@ func saveValidLangs(langs []string) {
fd.Close() fd.Close()
} }
func saveLanguageNames(names map[string]string) {
fd, err := os.Create("prettyprint.js")
if err != nil {
log.Fatal(err)
}
fmt.Fprint(fd, "var langPrettyprint = ")
json.NewEncoder(fd).Encode(names)
fd.Close()
}
func userPass() (string, string) { func userPass() (string, string) {
user := os.Getenv("TRANSIFEX_USER") user := os.Getenv("TRANSIFEX_USER")
pass := os.Getenv("TRANSIFEX_PASS") pass := os.Getenv("TRANSIFEX_PASS")
@@ -145,7 +131,7 @@ func loadValidLangs() []string {
} }
var langs []string var langs []string
exp := regexp.MustCompile(`\[([a-zA-Z@",-]+)\]`) exp := regexp.MustCompile(`\[([a-zA-Z",-]+)\]`)
if matches := exp.FindSubmatch(bs); len(matches) == 2 { if matches := exp.FindSubmatch(bs); len(matches) == 2 {
langs = strings.Split(string(matches[1]), ",") langs = strings.Split(string(matches[1]), ",")
for i := range langs { for i := range langs {
@@ -156,19 +142,3 @@ func loadValidLangs() []string {
return langs return langs
} }
type languageResponse struct {
Code string
Name string
}
func languageName(code string) string {
var lang languageResponse
resp := req("https://www.transifex.com/api/2/language/" + code)
defer resp.Body.Close()
json.NewDecoder(resp.Body).Decode(&lang)
if lang.Name == "" {
return code
}
return lang.Name
}
+24 -45
View File
@@ -12,7 +12,6 @@ import (
"encoding/json" "encoding/json"
"log" "log"
"os" "os"
"path/filepath"
"regexp" "regexp"
"strings" "strings"
@@ -22,48 +21,44 @@ import (
var trans = make(map[string]string) var trans = make(map[string]string)
var attrRe = regexp.MustCompile(`\{\{'([^']+)'\s+\|\s+translate\}\}`) var attrRe = regexp.MustCompile(`\{\{'([^']+)'\s+\|\s+translate\}\}`)
func generalNode(n *html.Node, filename string) { func generalNode(n *html.Node) {
translate := false translate := false
if n.Type == html.ElementNode { if n.Type == html.ElementNode {
if n.Data == "translate" { // for <translate>Text</translate> for _, a := range n.Attr {
translate = true if a.Key == "translate" {
} else { translate = true
for _, a := range n.Attr { break
if a.Key == "translate" { } else {
translate = true if matches := attrRe.FindStringSubmatch(a.Val); len(matches) == 2 {
break translation(matches[1])
} else {
if matches := attrRe.FindStringSubmatch(a.Val); len(matches) == 2 {
translation(matches[1])
}
} }
} }
} }
} else if n.Type == html.TextNode { } else if n.Type == html.TextNode {
v := strings.TrimSpace(n.Data) v := strings.TrimSpace(n.Data)
if len(v) > 1 && !(strings.HasPrefix(v, "{{") && strings.HasSuffix(v, "}}")) { if len(v) > 1 && !(strings.HasPrefix(v, "{{") && strings.HasSuffix(v, "}}")) {
log.Println("Untranslated text node (" + filename + "):") log.Println("Untranslated text node:")
log.Print("\t" + v) log.Print("\t" + v)
} }
} }
for c := n.FirstChild; c != nil; c = c.NextSibling { for c := n.FirstChild; c != nil; c = c.NextSibling {
if translate { if translate {
inTranslate(c, filename) inTranslate(c)
} else { } else {
generalNode(c, filename) generalNode(c)
} }
} }
} }
func inTranslate(n *html.Node, filename string) { func inTranslate(n *html.Node) {
if n.Type == html.TextNode { if n.Type == html.TextNode {
translation(n.Data) translation(n.Data)
} else { } else {
log.Println("translate node with non-text child < (" + filename + ")") log.Println("translate node with non-text child <")
log.Println(n) log.Println(n)
} }
if n.FirstChild != nil { if n.FirstChild != nil {
log.Println("translate node has children (" + filename + "):") log.Println("translate node has children:")
log.Println(n.Data) log.Println(n.Data)
} }
} }
@@ -77,29 +72,6 @@ func translation(v string) {
} }
} }
func walkerFor(basePath string) filepath.WalkFunc {
return func(name string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if filepath.Ext(name) == ".html" && info.Mode().IsRegular() {
fd, err := os.Open(name)
if err != nil {
log.Fatal(err)
}
doc, err := html.Parse(fd)
if err != nil {
log.Fatal(err)
}
fd.Close()
generalNode(doc, filepath.Base(name))
}
return nil
}
}
func main() { func main() {
fd, err := os.Open(os.Args[1]) fd, err := os.Open(os.Args[1])
if err != nil { if err != nil {
@@ -111,10 +83,17 @@ func main() {
} }
fd.Close() fd.Close()
var guiDir = os.Args[2] fd, err = os.Open(os.Args[2])
if err != nil {
filepath.Walk(guiDir, walkerFor(guiDir)) log.Fatal(err)
}
doc, err := html.Parse(fd)
if err != nil {
log.Fatal(err)
}
fd.Close()
generalNode(doc)
bs, err := json.MarshalIndent(trans, "", " ") bs, err := json.MarshalIndent(trans, "", " ")
if err != nil { if err != nil {
log.Fatal(err) log.Fatal(err)
+1 -1
View File
@@ -5,4 +5,4 @@ This directory contains configuration files for running syncthing under the
systemd user service. For further documentation take a look at the [systemd systemd user service. For further documentation take a look at the [systemd
section][1] on the Github Wiki. section][1] on the Github Wiki.
[1]: http://docs.syncthing.net/users/autostart.html#systemd [1]: https://github.com/syncthing/syncthing/wiki/Autostart-syncthing#systemd
+1 -1
View File
@@ -1,6 +1,6 @@
[Unit] [Unit]
Description=Syncthing - Open Source Continuous File Synchronization for %I Description=Syncthing - Open Source Continuous File Synchronization for %I
Documentation=http://docs.syncthing.net/ Documentation=https://github.com/syncthing/syncthing/wiki
After=network.target After=network.target
[Service] [Service]
+1 -1
View File
@@ -1,6 +1,6 @@
[Unit] [Unit]
Description=Syncthing - Open Source Continuous File Synchronization Description=Syncthing - Open Source Continuous File Synchronization
Documentation=http://docs.syncthing.net/ Documentation=https://github.com/syncthing/syncthing/wiki
After=network.target After=network.target
[Service] [Service]
+3 -5
View File
@@ -4,8 +4,9 @@ background under Mac OS X.
1. Install the `syncthing` binary in a directory called `bin` in your 1. Install the `syncthing` binary in a directory called `bin` in your
home directory. home directory.
2. Edit the `syncthing.plist` by replacing `USERNAME` with your actual 2. Edit the `syncthing.plist` file in the two places that refer to your
username such as `jb`. home directory; that is, replace `/Users/jb` with your actual home
directory location.
3. Copy the `syncthing.plist` file to `~/Library/LaunchAgents`. 3. Copy the `syncthing.plist` file to `~/Library/LaunchAgents`.
@@ -14,6 +15,3 @@ background under Mac OS X.
You probably want to turn off "Start Browser" among the settings to You probably want to turn off "Start Browser" among the settings to
avoid it opening a browser window on each login. avoid it opening a browser window on each login.
Logs are in `~/Library/Logs/Syncthing.log` and, for crashes and exceptions,
`~/Library/Logs/Syncthing-Error.log`.

Some files were not shown because too many files have changed in this diff Show More