e4a48085ae
backend/retry: feature flag new retry behavior
2024-05-24 20:24:02 +02:00
98709a4372
retry: reduce total number of retries
...
Retries in restic try to solve two main problems:
- retry a temporarily failed operation
- tolerate temporary network interruptions
The first problem only requires a few retries, whereas the last one benefits
primarily from spreading the requests over a longer duration.
Increasing the default multiplier and the initial interval works for
both cases. The first few retries only take a few seconds, while later
retries quickly reach the maximum interval of one minute. This ensures
that the total number of retries issued by restic will remain at around
21 retries for a 15 minute period. As the concurrency in restic is
bounded, retries drastically reduce the number of requests sent to a
backend. This helps to prevent overloading the backend.
2024-05-24 20:24:02 +02:00
512cd6ef07
retry: ensure that there's always at least one retry
...
Previously, if an operation failed after 15 minutes, then it would never
be retried. This means that large backend requests are more unreliable
than smaller ones.
2024-05-24 20:24:02 +02:00
a60ee9b764
retry: limit retries based on elapsed time not count
...
Depending on how long an operation takes to fail, the total retry
duration can currently vary between 1.5 and 15 minutes. In particular
for temporarily interrupted network connections, the former timeout is
too short. Thus always use a limit of 15 minutes.
2024-05-24 20:24:02 +02:00
a3633cad9e
retry: explicitly log failed requests
...
This simplifies finding the request in the log output that cause an
operation to fail.
2024-05-24 20:24:02 +02:00
c56ecec9aa
azure: deduplicate cli and default credentials case
2024-05-18 22:15:54 +02:00
355f520936
Azure: add option to force use of CLI credential
2024-05-18 22:15:54 +02:00
0c1ba6d95d
backend: remove unused Location method
2024-05-18 21:38:31 +02:00
1d6d3656b0
repository: move backend.LoadAll to repository.LoadRaw
...
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.
The removed cache backend test will be added again later on.
2024-05-18 21:26:00 +02:00
47232bf8b0
backend: move LimitReadCloser to util package
...
The helper is only intended for usage by backend implementations.
2024-05-18 21:26:00 +02:00
53d15bcd1b
retry: add circuit breaker to load method
...
If a file exhausts its retry attempts, then it is likely not accessible
the next time. Thus, immediately fail all load calls for that file to
avoid useless retries.
2024-05-18 19:59:26 +02:00
394c8ca3ed
rest/rclone/s3/sftp/swift: move short file detection behind feature gate
...
These backends tend to use a large variety of server implementations.
Some of those implementations might prove problematic with the new
checks.
2024-05-18 19:59:26 +02:00
aeb7eb245c
retry: do not retry permanent errors
...
This is currently gated behind a feature flag as some unexpected
interactions might show up in the wild.
2024-05-18 19:59:26 +02:00
bf8cc59889
Use generic backend-error-redesign feature flag instead of http-timeouts
...
An individual flag for each change of the backend error handling would
be too finegrained. Thus, add a generic flag.
2024-05-18 19:54:52 +02:00
4740528a0b
backend: add tests for IsPermanentError
2024-05-18 19:54:52 +02:00
6a85df7297
backend: add IsPermanentError() method to interface
2024-05-18 19:54:52 +02:00
cfc420664a
mem: stricter handling of out of bounds requests
2024-05-18 19:54:52 +02:00
d40f23e716
azure/b2/gs/s3/swift: adapt cloud backend
2024-05-18 19:54:51 +02:00
e793c002ec
local: stricter handling of short files
2024-05-18 19:54:21 +02:00
b4895ebd76
rest: rework error reporting and report too short files
2024-05-18 19:54:21 +02:00
eaa3f81d6b
sftp: check for truncated files without an extra backend request
2024-05-18 19:54:21 +02:00
c6d74458ee
sftp: improve handling of too short files
2024-05-18 19:54:21 +02:00
b5fdb1d637
Merge pull request #4782 from MichaelEischer/fix-sftp-performance
...
Fix sftp upload performance
2024-05-12 15:28:33 +02:00
ebd01a4675
backend: add tests for watchdogRoundTripper
2024-05-09 23:46:17 +02:00
8778670232
backend: cancel stuck http requests
...
requests that make no upload or download progress within a timeout are
canceled.
2024-05-09 23:46:17 +02:00
0987c731ec
backend: configure protocol-level connection health checks
...
This should detect a connection that is stuck for more than 2 minutes.
2024-05-09 23:46:17 +02:00
935327d480
sftp: slightly increase write concurrency
...
This should increase upload throughput for high latency links a bit.
2024-04-28 11:50:09 +02:00
669a669603
sftp: Fix upload performance issue
...
Since pkg/sftp 1.13.0 files were uploaded sequentially using 32kb chunks
instead of sending 64 chunks in parallel.
2024-04-28 11:48:26 +02:00
b15d867414
Merge pull request #4763 from MichaelEischer/refactor-prune
...
Refactor repair index / prune into the repository package
2024-04-22 22:24:53 +02:00
35277b7797
backend/mem: cleanup not found error message
2024-04-14 13:45:06 +02:00
55d56db31b
Merge pull request #4743 from MichaelEischer/deprecate-s3legacy-layout
...
Deprecate s3legacy layout
2024-04-11 22:09:34 +02:00
591b421c4a
Deprecate s3legacy layout
2024-04-10 21:27:56 +02:00
6e775d3787
Enhancement: option to send HTTP over unix socket
...
add tests for unix socket connection
switch HTTP rest-server test to use any free port
allow rest-server test graceful shutdown opportunity
2024-03-28 17:41:41 +01:00
c0514dd8ba
Fix linter errors (except for tests)
2024-02-10 22:58:10 +01:00
a763a5c67d
s3: minor cleanups for archive storage class handling
2024-01-20 11:25:28 +01:00
8ca58b487c
S3: do not set storage class for metadata when using archive storage
2024-01-20 11:04:15 +01:00
bd883caae1
CI: enable bodyclose linter
2024-01-19 21:17:18 +01:00
b1a8fd1d03
rest: fix and cleanup closing of http response body
...
If client.Do returns an error, then there's no body that has to be
closed. For requests for which we are not interested in the response
body, immediately drain and close the body to make sure it isn't
forgotten later on.
This change in particular adds the missing `Close()` call for the
`List()` command.
2024-01-19 21:17:17 +01:00
7b2de84763
Merge pull request #4618 from MichaelEischer/workaround-rclone-list-errors
...
rclone: Workaround for incorrect "not found" errors while listing files
2024-01-09 18:28:31 +01:00
2e8de9edfd
rclone: Workaround for incorrect "not found" errors while listing files
...
rclone returns a "not found" error if an internal error occurs while
listing a folder. Ignoring this error lets restic erroneously think that
there are no files, which can cause `prune` to wipe the whole
repository.
2024-01-09 18:20:16 +01:00
02bc73f5eb
s3: minor code cleanups
2024-01-06 21:44:53 +01:00
20cf4777cb
s3: check for EnvAWS credentials before Static credentials
...
EnvAWS considers more environment variables, including AWS_SESSION_TOKEN
and thus should be checked first.
2024-01-06 21:43:47 +01:00
5ffb536aae
feat: support AWS assume role
2024-01-06 21:19:58 +01:00
241916d55b
Fix typos
2023-12-06 13:11:55 +01:00
c7b770eb1f
convert MemorizeList to be repository based
...
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
2023-10-25 23:01:35 +02:00
1b8a67fe76
move Backend interface to backend package
2023-10-25 23:00:18 +02:00
ceb0774af1
backend: make LoadAll independent of restic package
2023-10-25 22:58:39 +02:00
b6d79bdf6f
restic: decouple restic.Handle
2023-10-25 22:54:07 +02:00
7881309d63
backend: move backend implementation helpers to util package
...
This removes code that is only used within a backend implementation from
the backend package. The latter now only contains code that also has
external users.
2023-10-25 22:54:07 +02:00
8e6fdf5edf
Merge pull request #4520 from awannabeengineer/load-retry-nonexistent
...
retry: Do not retry Load() if file does not exist
2023-10-25 20:42:05 +00:00
Michael Eischer