mirrors
/
restic
mirror of https://github.com/restic/restic.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
3,408 Commits 7 Branches 45 Tags 82 MiB
Commit Graph

267 Commits

Author SHA1 Message Date
Alexander Neumann 8e812b7ac0 errors: Make Cause() unwrap *url.Error 2018-01-23 23:14:05 +01:00
Alexander Neumann 685ce719ad Add comments for List() 2018-01-23 23:14:05 +01:00
Alexander Neumann b0c6e53241 Fix calls to repo/backend.List() everywhere 2018-01-21 21:15:09 +01:00
Alexander Neumann e9ea268847 Change List() implementation for all backends 2018-01-21 21:15:09 +01:00
Alexander Neumann c4e9d5d11e backend: Add tests for new List() function 2018-01-21 18:35:37 +01:00
Alexander Neumann 52230b8f07 backend: Rework List() 
For a discussion see #1567
 2018-01-21 18:35:37 +01:00
Alexander Neumann 2130897ce0 rest: Add test for external server 2018-01-20 10:25:47 +01:00
Alexander Neumann 67da240068 rest: Refactor backend tests 2018-01-20 10:25:37 +01:00
Alexander Neumann 1046eabf95 rest: Remove unneeded tempdir 2018-01-20 10:13:04 +01:00
Alexander Neumann 200415e0a1 Merge pull request #1564 from restic/sftp-password-prompt 
sftp: Prompt for password, don't terminate on SIGINT
 2018-01-20 09:48:17 +01:00
Alexander Neumann bcb6881ffb Merge pull request #1548 from ifedorenko/checker-backend-Test 
checker: Optimize checker.Packs()
 2018-01-19 13:05:35 +01:00
Igor Fedorenko 231076fa4a checker: Optimize checker.Packs() 
Use result of single repository.List() to find both missing and
orphaned data packs. For 500GB repository this eliminates ~100K
repository.Test() calls and improves check time by >30M in my
environment (~45min before this change and ~7min after).

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-18 20:50:39 -05:00
Alexander Neumann 0bdb131521 Remove SuspendSignalHandler 2018-01-17 23:14:47 +01:00
Alexander Neumann c31a5e7e5c Add argument to Skipf() 2018-01-17 23:14:37 +01:00
Alexander Neumann 05958caf6e sftp: Prompt for password, don't terminate on SIGINT 
This is a follow-up on fb9729fdb9, which
runs the `ssh` in its own process group and selects that process group
as the foreground group. After the sftp connection is established,
restic switches back to the previous foreground process group.

This allows `ssh` to prompt for the password, but it won't receive
the interrupt signal (SIGINT, ^C) later on, because it is not in the
foreground process group any more, allowing a clean tear down.
 2018-01-17 23:02:47 +01:00
Alexander Neumann 97565d68a2 fuse/mount: Add option for snapshot template 2018-01-15 14:47:17 +01:00
Alexander Neumann f49f5c5903 fuse: Don't return an error for EOF 2018-01-13 10:14:10 +01:00
Alexander Neumann d89f2e5226 fuse: Remove debug log message 2018-01-13 10:10:07 +01:00
Matthew Dawson 539599d1f1 repository/index: Optimize index.Has() 
When backing up several million files (>14M tested here) with few changes,
a large amount of time is spent failing to find an id in an index and creating
an error to signify this.  Since this is checked using the Has method,
which doesn't use this error, this time creating the error is wasted.

Instead, directly check if the given id and type are present in the index.
This also avoids reporting all the packs containing this blob, further
reducing cpu usage.
 2018-01-08 21:46:17 +01:00
Alexander Neumann d77a326bb0 Add benchmark for Index.Has() 2018-01-08 21:10:30 +01:00
Alexander Neumann b8620429e2 restorer: Set directory mode as last step 
Closes #1512
 2018-01-07 15:13:24 +01:00
Alexander Neumann aab414b368 Add test case for #1512 2018-01-07 14:50:47 +01:00
Alexander Neumann 5f43e17918 Merge pull request #1535 from ifedorenko/check-retry 
fixed restic-check does not retry backend.Test failures
 2018-01-07 09:43:49 +01:00
Igor Fedorenko 8c550ca011 fixed restic-check does not retry backend.Test failures 
added missing RetryBackend.Test implementation

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-06 23:22:35 -05:00
Alexander Neumann 032621289b errors: Add "Fatal" prefix for Fatalf() 2018-01-06 23:27:47 +01:00
Alexander Neumann b45fc89512 local/sftp: Create repo dirs on demand in Save() 2018-01-05 17:51:09 +01:00
Alexander Neumann aabc0ccaa7 fs: Make HasPathPrefix work with relative paths 2018-01-05 17:38:52 +01:00
Alexander Neumann 18ecd9df30 Improve limiting HTTP based backends 2017-12-29 12:43:49 +01:00
Alexander Neumann 3f7d85360a fuse: Only reload list of snapshots once per minute 2017-12-28 13:18:27 +01:00
Alexander Neumann 6c2b2a58ad backend: Retry deletes 2017-12-22 22:41:28 +01:00
Alexander Neumann bc04ce8e6b Merge pull request #1482 from restic/fix-1478 
backend: Only return top-level files for most dirs
 2017-12-20 18:06:47 +01:00
Alexander Neumann 7d8765a937 backend: Only return top-level files for most dirs 
Fixes #1478
 2017-12-14 19:14:16 +01:00
Alexander Neumann 81a04656c5 fuse: Cleanup node name 2017-12-13 20:21:18 +01:00
Harshavardhana 27ccea6371 Since upgrade to minio-go 4.0 remove workaround 
We added previously a code to fix the issue of chaining
credentials, we do not need this anymore since the
upstream minio-go already has this relevant change.
 2017-12-09 02:01:42 -08:00
Alexander Neumann 8b3b7bc5ef s3: Use context 2017-12-08 22:04:55 +01:00
Alexander Neumann 934ae1b559 Update to minio-go 4 2017-12-08 21:52:50 +01:00
Alexander Neumann 3a5e040b7e Merge pull request #1439 from armhold/propagate-context 
replace ad-hoc context.TODO() with gopts.ctx
 2017-12-08 20:27:36 +01:00
George Armhold 1695c8ed55 use global context for check, debug, dump, find, forget, init, key, 
list, mount, tag, unlock commands

gh-1434
 2017-12-06 07:02:55 -05:00
George Armhold 0dc31c03e1 remove check for context.Canceled 
gh-1434
 2017-12-06 05:38:29 -05:00
Alexander Neumann 82ded35706 node: Correctly compare times 2017-12-03 18:33:35 +01:00
Alexander Neumann 88607fc625 Correctly format all node types 2017-12-03 17:38:55 +01:00
Alexander Neumann 7092af6329 fuse: Handle sockets/fifos/devs correctly 
Closes #1463
 2017-12-03 17:25:00 +01:00
George Armhold 20d78ab0d9 add context to repo.Flush() call in test code 
gh-1434
 2017-12-03 07:32:50 -05:00
George Armhold be24237063 make retry code context-aware. 
detect cancellation in backend, so that retry code does not keep trying
once user has hit ^c

gh-1434
 2017-12-03 07:22:14 -05:00
George Armhold d886cb5c27 replace ad-hoc context.TODO() with gopts.ctx, so that cancellation 
can properly trickle down from cmd_*.

gh-1434
 2017-12-03 07:22:14 -05:00
Alexander Neumann 0b44c629f2 retry: Remove file after failed save 2017-11-30 22:05:14 +01:00
Alexander Neumann 410efe0694 Merge pull request #1454 from restic/fix-cache-dir 
Fix cache dir detection
 2017-11-29 18:43:07 +01:00
Alexander Neumann eddb8549ef backup: By default, do not save the access time 
This can be re-enabled with `--with-atime`.
 2017-11-28 21:31:35 +01:00
Alexander Neumann dc38265b54 Merge pull request #1436 from restic/remove-old-cache 
Remove old cache directories
 2017-11-27 21:37:05 +01:00
Alexander Neumann 0f5e38609f restore: Fix restore to relative path 2017-11-26 18:36:48 +01:00
Alexander Neumann 27d29b9853 restorer: Make sure node names are clean 2017-11-26 15:28:45 +01:00
Alexander Neumann 8a171731ba restorer: Add tests for invalid node names 2017-11-26 15:28:37 +01:00
Alexander Neumann 134abbd82b rest: Use client for creating the repository 
Before, creating a new repo via REST would use the defaut HTTP client,
which is not a problem unless the server uses HTTPS and a TLS
certificate which isn't signed by a CA in the system's CA store. In this
case, all commands work except the 'init' command, which fails with a
message like "invalid certificate".
 2017-11-25 20:56:40 +01:00
Alexander Neumann 1ebf0e8de8 Merge pull request #1437 from restic/fix-1292 
s3: Document and remove default prefix
 2017-11-25 11:34:26 +01:00
Alexander Neumann 47b326b7b5 Merge pull request #1423 from harshavardhana/creds 
Fix chaining of credentials for minio-go
 2017-11-24 21:57:52 +01:00
Alexander Neumann f79698dcdd Merge pull request #1410 from armhold/deadlock2 
unify behavior for max http connections across backends
 2017-11-24 21:32:56 +01:00
Alexander Neumann f4bab789b8 cache: Simplify cache dir creation 2017-11-24 20:53:26 +01:00
Alexander Neumann fa893ee477 cache: Add detection code for old cache dirs 2017-11-24 20:53:26 +01:00
Alexander Neumann 262b0cd9d4 s3: Remove default prefix "/restic" 2017-11-21 21:33:09 +01:00
Alexander Neumann e83ec17e95 s3: Correct comment 2017-11-20 22:21:39 +01:00
Alexander Neumann ea593fca1b cache: Correctly return dir for Windows/darwin 2017-11-20 06:11:18 +01:00
Alexander Neumann fe1f151ae1 cache: Return error during default dir detection 2017-11-20 06:10:42 +01:00
Harshavardhana 41c8c946ba Fix chaining of credentials for minio-go 
chaining failed because chaining provider
was only looking for subsequent credentials
provider after an error. Writer a new
chaining provider which proceeds to fetch
new credentials also under situations where
providers do not return but instead return
no keys at all.

Fixes https://github.com/restic/restic/issues/1422
 2017-11-18 02:51:12 -08:00
George Armhold 0268d0e7d6 swift backend: limit http concurrency in Save(), Stat(), Test(), Remove(), 
List().

move comment regarding problematic List() backend api (it's s3's ListObjects
that has a problem, NOT swift's ObjectsWalk).

As per discussion in PR #1399.
 2017-11-02 18:29:32 -04:00
George Armhold 8515d093e0 swift backend: fix premature release of semaphore in Load() & document 
concurrency issue in List().

refactor wrapReader from b2 -> semaphore so it can be used elsewhere.

As per discussion in PR #1399.
 2017-11-02 12:38:17 -04:00
George Armhold 99ac0da4bc s3 backend: limit http concurrency in Save(), Stat(), Test(), Remove() 
NB: List() is NOT currently limited, as it would cause deadlock due to
be.client.ListObjects() implementation.

as per discussion in PR #1399
 2017-11-01 09:40:54 -04:00
Alexander Neumann 6d46824fb0 Pass in a nil buffer to Open() 2017-11-01 10:30:40 +01:00
Alexander Neumann bb435b39d9 crypto: Rework Seal/Open to use sliceForAppend 2017-11-01 10:30:40 +01:00
Alexander Neumann 2a67d7a6c2 crypto: Correct test function names 2017-11-01 10:30:40 +01:00
Alexander Neumann ba43c8bab5 crypto: Fix nonce test, make it faster 2017-11-01 10:30:40 +01:00
Alexander Neumann 931e6ed2ac Use Seal/Open everywhere 2017-11-01 10:30:40 +01:00
Alexander Neumann a5f0e9ab65 Remove custom Encrypt/Decrypt methods 2017-11-01 10:30:40 +01:00
Alexander Neumann 6fc133ad6a Run tests on Seal/Open 2017-11-01 10:30:40 +01:00
Alexander Neumann e1b80859f2 Make crypto.Key implement cipher.AEAD 2017-11-01 10:30:39 +01:00
George Armhold d069ee31b2 GS backend: limit http concurrency in Save(), Stat(), Test(), Remove(), List() 
as per discussion in PR #1399
 2017-10-31 08:01:43 -04:00
George Armhold 981752ade0 Azure backend: limit http concurrency in Stat(), Test(), Remove() 
as per discussion in PR #1399
 2017-10-31 07:32:30 -04:00
George Armhold 2f8147af59 log unexpected errs from b2 ListCurrentObject() 
gh-1385
 2017-10-29 08:53:39 -04:00
Alexander Neumann f854a41ba9 Merge pull request #1399 from armhold/deadlock2 
prevent deadlock in List() for B2 when b2.connections=1
 2017-10-29 09:26:46 +01:00
George Armhold 3304b0fcf0 prevent deadlock in List() for B2 when b2.connections=1 
This is a fix for the following situation (gh-1188):

List() grabs a semaphore token upon entry, starts a goroutine, and
does not release the token until the routine exits (via a defer).

The goroutine iterates over the results from ListCurrentObjects(),
sending them one at a time to a channel, where they are ultimately
processed by be.Load().

Since be.Load() also needs a token, this will result in deadlock if
b2.connections=1.

This fix changes List() so that the token is only held during the call
to ListCurrentObjects().
 2017-10-28 18:46:47 -04:00
George Armhold d8938e259a sftp ReadDir: add path to return error messages (gh-1323) 
fix missing "Close" string in debug log fmt
 2017-10-28 14:16:27 -04:00
Alexander Neumann e71db01230 Improve crypto test 2017-10-28 12:09:33 +02:00
Alexander Neumann 178e946fc7 Rename KDFParams -> Params 2017-10-28 10:28:29 +02:00
Alexander Neumann 7a99418dc5 Merge pull request #1393 from armhold/lint-errcheck 
detect errors from fs.Walk() in local backend List()
 2017-10-28 09:56:11 +02:00
Alexander Neumann c71ba466ea Merge pull request #1391 from armhold/b2-listmax 
pass in defaultListMaxItems to b2Backend constructor
 2017-10-28 09:54:57 +02:00
Alexander Neumann 8ce5d35543 Merge pull request #1390 from armhold/lint 
small cleanup
 2017-10-28 09:54:35 +02:00
George Armhold 8a37c07295 send errors from fs.Walk() to debug log 
clarify non-err returns from Walk where err is already proved to be nil
 2017-10-27 08:41:17 -04:00
George Armhold bd0ada7842 go fmt 2017-10-26 16:37:11 -04:00
George Armhold eea96f652d go fmt 2017-10-26 16:22:10 -04:00
George Armhold 38c3061df7 pass in defaultListMaxItems to b2Backend constructor 
gh-1385
 2017-10-26 14:22:16 -04:00
George Armhold f5fa602482 detect and return error from file Close() in Node.createFileAt() 
gh-1385
 2017-10-26 13:53:31 -04:00
Konrad Wojas 5b96885c6d Control progress rate with RESTIC_PROGRESS_FPS env 
Add a RESTIC_PROGRESS_FPS environment variable to limit the interval
at which the progress indicator updates (allowed values: 1-60).

The default rate of 60 FPS can cause high terminal CPU load on some
systems, like iTerm2 on macOS with font anti-aliasing enabled.

Usage:

    RESTIC_PROGRESS_FPS=1 restic ...
    RESTIC_PROGRESS_FPS=60 restic ...
 2017-10-26 14:46:56 +08:00
George Armhold bcdebfb84e small cleanup: 
- be explicit when discarding returned errors from .Close(), etc.
- remove named return values from funcs when naked return not used
- fix some "err" shadowing when redeclaration not needed
 2017-10-25 12:03:55 -04:00
Alexander Neumann 90b96d19cd Merge pull request #1365 from felix9/fix_1068 
Fix failure to detect some legacy s3 repos
 2017-10-21 12:19:58 +02:00
Alexander Neumann d63ab4e9a4 Merge pull request #1358 from prattmic/chunk_size 
gs: add option to set chunk size
 2017-10-21 11:13:48 +02:00
Felix Lee 944fc857eb Fix failure to detect some legacy s3 repos 
Sometimes s3 listobjects for a directory includes an entry for that
directory. The restic s3 backend doesn't expect that and returns
an error.

Symptom is:
  ReadDir: invalid key name restic/key/, removing prefix
     restic/key/ yielded empty string

I'm not sure when s3 does that; I'm unable to reproduce it myself.

But in any case, it seems correct to ignore that when it happens.

Fixes #1068
 2017-10-18 13:45:31 -07:00
Michael Pratt 9fa4f5eb6b gs: disable resumable uploads 
By default, the GCS Go packages have an internal "chunk size" of 8MB,
used for blob uploads.

Media().Do() will buffer a full 8MB from the io.Reader (or less if EOF
is reached) then write that full 8MB to the network all at once.

This behavior does not play nicely with --limit-upload, which only
limits the Reader passed to Media. While the long-term average upload
rate will be correctly limited, the actual network bandwidth will be
very spikey.

e.g., if an 8MB/s connection is limited to 1MB/s, Media().Do() will
spend 8s reading from the rate-limited reader (performing no network
requests), then 1s writing to the network at 8MB/s.

This is bad for network connections hurt by full-speed uploads,
particularly when writing 8MB will take several seconds.

Disable resumable uploads entirely by setting the chunk size to zero.
This causes the io.Reader to be passed further down the request stack,
where there is less (but still some) buffering.

My connection is around 1.5MB/s up, with nominal ~15ms ping times to
8.8.8.8.

Without this change, --limit-upload 1024 results in several seconds of
~200ms ping times (uploading), followed by several seconds of ~15ms ping
times (reading from rate-limited reader). A bandwidth monitor reports
this as several seconds of ~1.5MB/s followed by several seconds of
0.0MB/s.

With this change, --limit-upload 1024 results in ~20ms ping times and
the bandwidth monitor reports a constant ~1MB/s.

I've elected to make this change unconditional of --limit-upload because
the resumable uploads shouldn't be providing much benefit anyways, as
restic already uploads mostly small blobs and already has a retry
mechanism.

--limit-download is not affected by this problem, as Get().Download()
returns the real http.Response.Body without any internal buffering.

Updates #1216
 2017-10-17 21:12:04 -07:00
Alexander Neumann ce4d71d626 backend: Add partial read failure to error backend 2017-10-17 22:11:38 +02:00
Alexander Neumann 8e2ef3f38b cache: Store first, then store in the backend 
Store the file in the backend, then rewind the reader and store it
in the cache afterwards.
 2017-10-17 22:01:20 +02:00
Alexander Neumann 8dc952775e backend: Correctly retry Save() calls 
Make sure the given reader is an io.Seeker and rewind it properly each
time.
 2017-10-17 21:46:38 +02:00
Tobias Klein b57ca64275 fuse: bugfix: ignore empty tags 2017-10-16 16:22:08 +02:00