mirrors
/
restic
mirror of https://github.com/restic/restic.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
8,110 Commits 7 Branches 44 Tags 81 MiB
Commit Graph

2124 Commits

Author SHA1 Message Date
aneesh-n 5cff6e084e
Add support for Windows EA in node 
Refactor Extended Attribute related functions in node files as windows apis get and set EA in bulk
 2024-05-17 14:18:20 -06:00
aneesh-n d6708505b9
Add helpers for Windows Extended Attributes 2024-05-17 14:15:03 -06:00
Michael Eischer 7ed560a201
Merge pull request #4796 from MichaelEischer/parallel-dump-load 
dump: Parallelize loading large files
 2024-05-14 22:35:44 +02:00
Michael Eischer 92221c2a6d
Merge pull request #4708 from zmanda/windows-securitydesc 
Back up and restore SecurityDescriptors on Windows
 2024-05-12 14:14:39 +00:00
Michael Eischer b5fdb1d637
Merge pull request #4782 from MichaelEischer/fix-sftp-performance 
Fix sftp upload performance
 2024-05-12 15:28:33 +02:00
Michael Eischer ebd01a4675 backend: add tests for watchdogRoundTripper 2024-05-09 23:46:17 +02:00
Michael Eischer 8778670232 backend: cancel stuck http requests 
requests that make no upload or download progress within a timeout are
canceled.
 2024-05-09 23:46:17 +02:00
Michael Eischer 0987c731ec backend: configure protocol-level connection health checks 
This should detect a connection that is stuck for more than 2 minutes.
 2024-05-09 23:46:17 +02:00
aneesh-n a4fd1b91e5
Fix review comments 
Change lowerPrivileges from bool to atomic.Bool.
Add missing cleanup from upstream go-winio.
Add handling for ERROR_NOT_ALL_ASSIGNED warning.
 2024-05-06 16:54:08 -06:00
Michael Eischer 4d55a62ada bloblru: add test for GetOrCompute 2024-05-05 12:00:25 +02:00
Michael Eischer 7cce667f92 fuse: switch to use bloblru.GetOrCompute 2024-05-05 11:38:42 +02:00
Michael Eischer bd03af2feb dump: add GetOrCompute to bloblru cache 2024-05-05 11:38:42 +02:00
Michael Eischer 45509eafc8 dump: load blobs of a file from repository in parallel 2024-05-05 11:38:42 +02:00
Michael Eischer ffe5439149
Merge pull request #4605 from MichaelEischer/better-restorer-error-handling 
Rework repository.StreamPacks & better restorer error handling
 2024-05-01 16:37:41 +02:00
aneesh-n 672f6cd776
Fix review comments for privileges and security flags 2024-04-29 17:29:51 -06:00
aneesh-n 08c6945d61
Fix review comments 2024-04-29 16:21:38 -06:00
Aneesh N 3f76b902e5
Merge branch 'master' into windows-securitydesc 2024-04-29 14:40:34 -06:00
DRON-666 125dba23c5 Rearange code 2024-04-29 01:27:34 +03:00
DRON-666 7ee889bb0d Use S_FALSE and MaxInt 2024-04-29 01:25:25 +03:00
DRON-666 90b168eb6c isMountPointExcluded to isMountPointIncluded 2024-04-29 01:23:50 +03:00
DRON-666 24330c19a8 Use kebab case in option names 2024-04-29 01:21:33 +03:00
DRON-666 5703e5a652 Fix texts and comments 2024-04-29 01:18:46 +03:00
DRON-666 0a8f9c5d9c vss: Add tests for "provider" option 2024-04-28 22:45:21 +03:00
DRON-666 bb0f93ef3d vss: Add "provider" option 2024-04-28 22:45:21 +03:00
DRON-666 3bac1f0135 vss: Fix issues reported by linters 2024-04-28 22:45:21 +03:00
DRON-666 88c509e3e9 vss: Change `ErrorHandler` signature 
We don't need `error` here: the only existing implementation
of `ErrorHandler` always call `Backup.Error` and all
implementations of `Backup.Error` always return nil.
 2024-04-28 22:44:16 +03:00
DRON-666 9d3d915e2c vss: Add some tests 2024-04-28 22:44:16 +03:00
DRON-666 c4f67c0064 vss: Add volume filtering 
Add options to exclude all mountpoints and arbitrary volumes from snapshotting.
 2024-04-28 22:44:15 +03:00
DRON-666 7470e5356e vss: Add "timeout" option 
Changing multiple "callAsyncFunctionAndWait" with fixed timeout
to calculated timeout based on deadline.
 2024-04-28 22:44:15 +03:00
DRON-666 78dbc5ec58 vss: Add initial support for extended options 2024-04-28 22:44:15 +03:00
Michael Eischer 935327d480 sftp: slightly increase write concurrency 
This should increase upload throughput for high latency links a bit.
 2024-04-28 11:50:09 +02:00
Michael Eischer 669a669603 sftp: Fix upload performance issue 
Since pkg/sftp 1.13.0 files were uploaded sequentially using 32kb chunks
instead of sending 64 chunks in parallel.
 2024-04-28 11:48:26 +02:00
Michael Eischer 940a3159b5 let index.Each() and pack.Size() return error on canceled context 
This forces a caller to actually check that the function did complete.
 2024-04-22 22:39:32 +02:00
Michael Eischer 31624aeffd Improve command shutdown on context cancellation 2024-04-22 22:31:38 +02:00
Michael Eischer b15d867414
Merge pull request #4763 from MichaelEischer/refactor-prune 
Refactor repair index / prune into the repository package
 2024-04-22 22:24:53 +02:00
Michael Eischer 2e6c43c695
Merge pull request #4761 from MichaelEischer/fix-cache-race 
cache: ignore ErrNotExist during cleanup of old files
 2024-04-22 21:46:06 +02:00
Michael Eischer f7632de3d6
Merge pull request #4772 from MichaelEischer/better-error-on-too-large-blob 
repository: Better error message if blob is larger than 4GB
 2024-04-22 21:45:06 +02:00
Michael Eischer 20d8eed400 repository: streamPack: separate requests for gap larger than 1MB 
With most cloud providers, traffic is much more expensive than API
calls. Thus slightly bias streamPack towards a bit more API calls in
exchange for slightly less traffic.
 2024-04-22 21:21:23 +02:00
Michael Eischer cf700d8794 repository: streamPack: reuse zstd decoder 2024-04-22 21:21:23 +02:00
Michael Eischer 666a0b0bdb repository: streamPack: replace streaming with chunked download 
Due to the interface of streamPack, we cannot guarantee that operations
progress fast enough that the underlying connections remains open. This
introduces partial failures which massively complicate the error
handling.

Switch to a simpler approach that retrieves the pack in chunks of 32MB.
If a blob is larger than this limit, then it is downloaded separately.

To avoid multiple copies in memory, an auxiliary interface
`discardReader` is introduced that allows directly accessing the
downloaded byte slices, while still supporting the streaming used by the
`check` command.
 2024-04-22 21:21:23 +02:00
Michael Eischer 621012dac0 repository: Add blob loading fallback to LoadBlobsFromPack 
Try to retrieve individual blobs via LoadBlob if streaming did not work.
 2024-04-21 21:35:55 +02:00
Michael Eischer 10355c3fb6 repository: Better error message if blob is larger than 4GB 2024-04-19 22:00:35 +02:00
coderwander a82ed71de7 Fix struct names 
Signed-off-by: coderwander <770732124@qq.com>
 2024-04-18 10:02:09 +08:00
Michael Eischer 09587e6c08 repository: duplicate a few blobs in prune tests 2024-04-14 13:57:19 +02:00
Michael Eischer defd7ae729 prune/repair index: reset in-memory index after command 
The current in-memory index becomes stale after prune or repair index
have run. Thus, just drop the in-memory index altogether once these
commands have finished.
 2024-04-14 13:46:24 +02:00
Michael Eischer 038586dc9d repository: add minimal test for prune 2024-04-14 13:45:17 +02:00
Michael Eischer d8622c86eb prune: clean up internal interface 2024-04-14 13:45:15 +02:00
Michael Eischer 8d507c1372 repository: add basic test for RepairIndex 2024-04-14 13:45:15 +02:00
Michael Eischer 310db03c0e repair index: improve log output if index cannot be deleted 
The operation will always fail with an error if an index cannot be
deleted. Thus, this change is purely cosmetic.
 2024-04-14 13:45:13 +02:00
Michael Eischer 7d1b9cde34 repository: use normal Init method in tests 2024-04-14 13:45:11 +02:00
Michael Eischer b25fc2c89d repository: remove redundant flushes from tests 2024-04-14 13:45:10 +02:00
Michael Eischer c65459cd8a repository: speed up tests 2024-04-14 13:45:10 +02:00
Michael Eischer eda9f7beb4 ui/progress: add helper to print messages during tests 2024-04-14 13:45:08 +02:00
Michael Eischer 35277b7797 backend/mem: cleanup not found error message 2024-04-14 13:45:06 +02:00
Michael Eischer 7ba5e95a82 check: allow tests to only verify pack&index integrity 2024-04-14 13:45:04 +02:00
Michael Eischer 4c9a10ca37 repair packs: deduplicate index rebuild 2024-04-14 13:45:02 +02:00
Michael Eischer 85e4021619 prune: move additional option checks to repository 2024-04-14 13:44:58 +02:00
Michael Eischer 55d56db31b
Merge pull request #4743 from MichaelEischer/deprecate-s3legacy-layout 
Deprecate s3legacy layout
 2024-04-11 22:09:34 +02:00
Michael Eischer fc3b548625 prune: move logic into repository package 2024-04-10 21:30:52 +02:00
Michael Eischer 866ddf5698 repair index: refactor code into repository package 2024-04-10 21:30:52 +02:00
Michael Eischer 591b421c4a Deprecate s3legacy layout 2024-04-10 21:27:56 +02:00
Michael Eischer bf054c09d2 backup: Ignore xattr.list permission error for parent directories 
On FreeBSD, limited users may not be able to even list xattrs for the
parent directories above the snapshot source paths. As this can cause
the backup to fail, just ignore those errors.
 2024-04-10 20:46:15 +02:00
Michael Eischer 0747cf5319 cache: ignore ErrNotExist during cleanup of old files 
Two restic processes running concurrently can try to remove the same
files from the cache. This could cause one process to fail with an error
if the other one has already remove a file that the current process also
tries to delete.
 2024-04-10 19:25:51 +02:00
Michael Eischer 49f98f25fc
Merge pull request #4742 from MichaelEischer/consistent-rtest-import 
Use consistent alias for interal/test package
 2024-04-03 21:47:32 +02:00
Michael Eischer 6cca1d5705
Merge pull request #4655 from ae-govau/unixsocket 
Enhancement: option to send HTTP over unix socket
 2024-04-03 19:29:21 +00:00
Michael Eischer ec2b79834a use consistent alias for interal/test package 2024-03-29 00:24:03 +01:00
Michael Eischer 5e98f1e2eb repository: fix test setup race conditions 2024-03-28 23:17:02 +01:00
Michael Eischer dc441c57a7 repository: unify repository initialization in tests 
Tests should use a helper from internal/repository/testing.go to
construct a Repository object.
 2024-03-28 23:17:02 +01:00
Michael Eischer 3ba1fa3cee repository: remove a few global variables 2024-03-28 23:17:02 +01:00
Michael Eischer 044e8bf821 repository: parallelize lock tests 2024-03-28 23:17:02 +01:00
Michael Eischer e8df50fa3c repository: remove global list of locks 2024-03-28 22:46:33 +01:00
Michael Eischer cbb5f89252 lock: move code to repository package 2024-03-28 22:46:33 +01:00
Michael Eischer 7f9ad1c3db
Merge pull request #4705 from MichaelEischer/snapshot-statistics 
Store snapshot statistics & print snapshot size
 2024-03-28 22:41:45 +01:00
Michael Eischer d705741571 backup: test that deviceID is only stored for hardlinks 2024-03-28 19:32:50 +01:00
Michael Eischer a9b3d86c4f features: remove example feature 2024-03-28 19:12:07 +01:00
Michael Eischer a26d6ffa72 archiver: move deviceID handling behind feature flag 2024-03-28 19:12:07 +01:00
Michael Eischer 2ba21fe72b archiver: only store deviceID for hardlinks 
The deviceID can change e.g. when backing up from filesystem snapshot.
It is only used for hardlink detection. Thus there it is not necessary
to store it for everything else.
 2024-03-28 19:12:07 +01:00
Michael Eischer 8876e3025b
Merge pull request #4724 from MichaelEischer/disable-old-index 
Deprecated legacy index format
 2024-03-28 18:24:47 +01:00
Adam Eijdenberg 6e775d3787 Enhancement: option to send HTTP over unix socket 
add tests for unix socket connection

switch HTTP rest-server test to use any free port

allow rest-server test graceful shutdown opportunity
 2024-03-28 17:41:41 +01:00
avoidalone ac948fccda fix some typos 
Signed-off-by: avoidalone <wuguangdong@outlook.com>
 2024-03-11 14:35:12 +08:00
Michael Eischer f8852f0eb6 repair index: fix deletion of legacy indexes 2024-03-09 18:21:22 +01:00
Michael Eischer 1a8bf358f1 index: deprecate legacy index format 2024-03-09 18:21:14 +01:00
Michael Eischer a9b64cd7ad features: print warning for stable/depreacted feature flags 2024-03-09 17:29:52 +01:00
Michael Eischer 70839155f2 features: add tests 2024-03-09 17:29:52 +01:00
Michael Eischer 5974a79497 features: add basic feature flag implementation 2024-03-09 17:29:52 +01:00
Michael Eischer e71660cd1e backup: rename data_added_in_repo statistic to data_added_packed 2024-02-25 20:40:52 +01:00
Aneesh Nireshwalia 062d408987
Clean up SecurityDescriptor helper 2024-02-24 14:23:04 -07:00
Aneesh Nireshwalia 5764300022
Add changelog and fix lint error 2024-02-24 13:47:49 -07:00
Aneesh Nireshwalia 90916f53de
Add test cases for security descriptors 2024-02-24 13:27:01 -07:00
Aneesh Nireshwalia 70cf8e3788
Add support for backup/restore of security descriptors 2024-02-24 13:25:28 -07:00
Aneesh Nireshwalia e3e59fef24
Fix CombineErrors and fillExtendedAttr error handling 2024-02-24 13:22:34 -07:00
Aneesh Nireshwalia 09ce1b4e58
Create helper for SecurityDescriptor related functions 2024-02-24 13:16:25 -07:00
Michael Eischer a8f5684f68 archiver: test statistics in snapshot 2024-02-23 22:05:15 +01:00
Michael Eischer 681395955e archiver: test backup summary calculation 2024-02-23 21:46:39 +01:00
Michael Eischer 38f91d3b5e backup: store statistics in snapshot 2024-02-23 20:27:13 +01:00
Michael Eischer 86897314d5 backup: expose data_added_in_repo in JSON output 
The value describes how much data was added after compression.
Previously, it was only available in the text output.
 2024-02-23 20:27:13 +01:00
Michael Eischer a59f654fa6 archiver: refactor summary collection from ui into the archiver 2024-02-23 20:27:13 +01:00
Michael Eischer 8b1a85711f archiver: unexport save/saveDir/saveTree methods 2024-02-23 20:24:21 +01:00
Aneesh Nireshwalia 4bbd25a37f
Add tests for generic attribute changes 2024-02-22 17:55:50 -07:00
Aneesh Nireshwalia d4be734c73
Handle readonly empty files in windows 2024-02-22 17:54:43 -07:00
Aneesh Nireshwalia eeb1aa5388
Add ability to report warnings to terminal 
Report warnings to terminal when unrecognized generic attributes are found in the repository.
 2024-02-22 17:52:26 -07:00
Aneesh Nireshwalia 0962917974
Support windows metadata using generic attribs 
Add new generic_attributes attribute in Node.
Use the generic attributes to add support for creation time and file attributes like hidden, readonly, encrypted in windows. Handle permission errors for readonly files in windows.
Handle backup and restore of encrypted attributes using windows system calls.
 2024-02-22 17:31:20 -07:00
Aneesh Nireshwalia 62a8a599f1
Add optional messages for Equals helper 2024-02-22 16:58:12 -07:00
Aneesh Nireshwalia 94de87d4b7
Add CombineErrors helper function 2024-02-22 16:57:00 -07:00
Michael Eischer 4c3218ef9f repository: include packID in StreamPack for decrypt/decompress errors 2024-02-17 19:38:01 +01:00
Michael Eischer 18b0bbbf42 repository: use fmt.Errorf in StreamPacks 2024-02-17 19:37:32 +01:00
Michael Eischer ed4a4f8748 check: exclude inaccessible files from the repair pack suggestion 2024-02-12 20:25:15 +01:00
Michael Eischer 4073299a7c check: fix missing error if blob is invalid 2024-02-12 20:20:13 +01:00
Michael Eischer 22a3cea1b3 checker: wrap all pack errors in ErrPackData 2024-02-12 20:19:32 +01:00
Michael Eischer 5b5d506472 backup: report files whose chunks failed to upload 2024-02-11 22:43:26 +01:00
Alexander Neumann c0514dd8ba Fix linter errors (except for tests) 2024-02-10 22:58:10 +01:00
Michael Eischer 5957417b1f Apply changelog entry / documentation improvements from review 2024-02-04 18:55:41 +01:00
Michael Eischer 86b38a0b17 rename `--no-verify-pack` to `--no-extra-verify` 2024-02-04 17:01:05 +01:00
Michael Eischer c32e5e2abb pack: verify integrity of pack file header 2024-02-04 15:31:42 +01:00
Michael Eischer c97a271e89 repository: ask users to report corrupted data while saving blobs 2024-02-04 15:31:42 +01:00
Michael Eischer 193140525c repository: test verification of blobs/unpacked data 2024-02-04 15:31:42 +01:00
Michael Eischer 2dbb18128c repository: Allow skipping verification for tests 
Some tests have to explicitly create pack files with blobs that don't
match their ID. For those blobs the builtin verification of the
repository must be disabled.
 2024-02-03 18:22:47 +01:00
Michael Eischer 30a84e9003 backup: verify unpacked files before upload 2024-02-03 18:22:47 +01:00
Michael Eischer c01a0c6da7 backup: verify blobs before upload 
This only covers the blobs themselves, the pack header is not verified
so far. Unpacked files are also not covered by the integrity check.
 2024-02-03 18:22:47 +01:00
Michael Eischer 16e3f79e8b repository: make repo.Options configurable for test repos 2024-02-03 18:22:47 +01:00
Michael Eischer bb92b487f7 repository: fix repack test 2024-02-03 18:22:47 +01:00
Michael Eischer e44e4b00a6
Merge pull request #4550 from ndecker/ls-ncdu 
Ls ncdu
 2024-01-27 12:27:35 +00:00
Michael Eischer d4ed7c8858 walker: add tests for leaveDir 2024-01-27 13:17:33 +01:00
Michael Eischer 2c80cfa4a5 walker: fix missing leaveDir if directory is partially skipped 2024-01-27 13:17:33 +01:00
Michael Eischer 9ecbda059c walker: add callback to inform about leaving a directory 2024-01-27 13:17:32 +01:00
Michael Eischer 246559e654 check: cleanup s3 legacy detection 2024-01-27 13:02:04 +01:00
Michael Eischer 1dfd854769 lock: switch to repository.List 2024-01-27 13:02:04 +01:00
Michael Eischer bfb56b78e1 replace some usages of restic.Repository with more specific interface 
This should eventually make it easier to test the code.
 2024-01-27 13:02:02 +01:00
Michael Eischer 3424088274
Merge pull request #4644 from MichaelEischer/refactor-repair-packs 
Refactor and test `repair packs`
 2024-01-27 13:00:51 +01:00
Michael Eischer 724ec179e3
Merge pull request #4648 from MichaelEischer/repository-removekey 
repository: Introduce RemoveKey function
 2024-01-27 13:00:42 +01:00
Michael Eischer f0e1ad2285 fix linter warning 2024-01-27 12:51:45 +01:00
Michael Eischer fd579421dd repository: deduplicate test 2024-01-27 12:51:45 +01:00
Michael Eischer 42c9318b9c repair pack: add tests 2024-01-27 12:51:45 +01:00
Michael Eischer 764b0bacd6 repair pack: add support for truncated files 2024-01-27 12:51:45 +01:00
Michael Eischer 7c351bc53c repair pack: reenable auto index updates 
The method is not available on the restic.Repository interface that is
used for testing. Drop the call as a small amount of additional index
writes is not a problem.
 2024-01-27 12:51:45 +01:00
Michael Eischer feeab84204 repair pack: extract the repair logic into the repository package 
Currently, the cmd/restic package contains a significant amount of code
that modifies repository internals. This code should in the mid-term
move into the repository package.
 2024-01-27 12:51:45 +01:00
Michael Eischer cb50832d50 index: let MasterIndex.Save also delete obsolete indexes 2024-01-27 12:51:08 +01:00
Michael Eischer bedff1ed6d split deleteFiles into UI and logic parts 2024-01-27 12:51:08 +01:00
Michael Eischer c13bf0b607 repository: Introduce RemoveKey function 
This replaces directly removing keys via the backend.
 2024-01-27 12:42:58 +01:00
Michael Eischer 446167ae80
Merge pull request #4643 from MichaelEischer/remove-redundant-poly1305-mask 
Remove redundant poly1305 key masking
 2024-01-23 19:46:40 +01:00
Michael Eischer 1419baf67a
Merge pull request #4645 from MichaelEischer/improve-lock-checking 
lock: checkForOtherLocks processes each lock at most once
 2024-01-23 19:46:00 +01:00
Michael Eischer 6bdca13603 ui/termstatus: simplify cleaning up on termination 
`writeStatus` also cleans no longer used status lines.
The old code actually cleaned one line too much. However, as that line
was never used it makes no difference.
 2024-01-21 21:27:27 +01:00
Michael Eischer f1f34eb3e5 lock: checkForOtherLocks processes each lock at most once 
If a lock could not be loaded, then restic would check all lock files
again. These repeated checks are not useful as the status of a lock file
cannot change unless its ID changes too. Thus, skip already check lock
files on retries.
 2024-01-20 22:40:12 +01:00
Michael Eischer fee83e1c09 Remove redundant poly1305 key masking 
The implementation in crypto/poly1305 already performs the exact same
masking.
 2024-01-20 12:36:59 +01:00
Michael Eischer a763a5c67d s3: minor cleanups for archive storage class handling 2024-01-20 11:25:28 +01:00
Vladislav Belous 8ca58b487c S3: do not set storage class for metadata when using archive storage 2024-01-20 11:04:15 +01:00
Michael Eischer 2c310a526e repository: Replace StreamPack function with LoadBlobsFromPack method 
LoadBlobsFromPack is now part of the repository struct. This ensures
that users of that method don't have to deal will internals of the
repository implementation.

The filerestorer tests now also contain far fewer pack file
implementation details.
 2024-01-19 21:40:43 +01:00
Michael Eischer 6b7b5c89e9 repository: prepare StreamPack refactor 2024-01-19 21:40:43 +01:00
Michael Eischer 22d0c3f8dc check: Use PackBlobIterator instead of StreamPack 
To only stream the content of a pack file once, check used StreamPack
with a custom pack load function. This combination was always brittle
and complicates using StreamPack everywhere else. Now that StreamPack
internally uses PackBlobIterator use that primitive instead, which is a
much better fit for what the check command requires.
 2024-01-19 21:40:36 +01:00
Michael Eischer fb422497af repository: split StreamPack implementation 
Move the actual decoding of the pack data into a separate iterator.
 2024-01-19 21:39:55 +01:00
Michael Eischer 54c5c72e5a
Merge pull request #4616 from MichaelEischer/fix-rest-connection-close 
rest: fix and cleanup closing of http response body
 2024-01-19 21:31:35 +01:00
Michael Eischer bd883caae1 CI: enable bodyclose linter 2024-01-19 21:17:18 +01:00
Michael Eischer b1a8fd1d03 rest: fix and cleanup closing of http response body 
If client.Do returns an error, then there's no body that has to be
closed. For requests for which we are not interested in the response
body, immediately drain and close the body to make sure it isn't
forgotten later on.

This change in particular adds the missing `Close()` call for the
`List()` command.
 2024-01-19 21:17:17 +01:00
Michael Eischer fdcbb53017 walker: test skipping for root node 2024-01-19 21:16:06 +01:00
Michael Eischer 0b39940fdb walker: Remove ignoreTrees functionality 
It was only used in two places:
- stats: apparently as a minor performance optimization, which is
  unlikely to be important
- find: filtered directories would be ignored. However, this
  optimization missed that it is possible that two directories have the
  exact same content. Such directories would be incorrectly ignored too.
  Example:
```
mkdir test test/a test/b
restic backup test
restic find latest test/b
-> incorrectly does not return anything
```

Thus, remove the functionality as it's apparently too complex to use
correctly.
 2024-01-19 21:16:06 +01:00
Michael Eischer 7b2de84763
Merge pull request #4618 from MichaelEischer/workaround-rclone-list-errors 
rclone: Workaround for incorrect "not found" errors while listing files
 2024-01-09 18:28:31 +01:00
Michael Eischer c31e9418ba
Merge pull request #4626 from MichaelEischer/reliable-large-restores 
Improve reliability of large restores
 2024-01-09 18:23:09 +01:00
Michael Eischer 2e8de9edfd rclone: Workaround for incorrect "not found" errors while listing files 
rclone returns a "not found" error if an internal error occurs while
listing a folder. Ignoring this error lets restic erroneously think that
there are no files, which can cause `prune` to wipe the whole
repository.
 2024-01-09 18:20:16 +01:00
Michael Eischer e78be75d1e restore: separately restore blobs that are frequently referenced 
Writing these blobs to their files can take a long time and consequently
cause the backend connection to time out. Avoid that by retrieving these
blobs separately.
 2024-01-08 21:00:13 +01:00
Michael Eischer 2267910418 restore: split error reporting from downloadPack 2024-01-08 20:57:00 +01:00
Michael Eischer 00d18b7a88 restore: cleanup downloadPack 2024-01-08 20:53:08 +01:00
Michael Eischer 9328f34d43 restore: split downloadPack into smaller methods 2024-01-08 20:52:36 +01:00
Michael Eischer 77434c6e2b
Merge pull request #4474 from ekristen/aws-assume-role 
Allow AWS Assume Role
 2024-01-08 19:07:17 +00:00
Michael Eischer e4a7eb09ef
Merge pull request #4624 from MichaelEischer/better-restorer-error-reporting 
Improver restorer error reporting
 2024-01-07 11:20:29 +01:00
Michael Eischer f8b4e932ef
Merge pull request #4620 from MichaelEischer/improve-irregular-file-handling 
Improve irregular file handling
 2024-01-07 11:12:07 +01:00
Michael Eischer dac3508170 restore: only report errors for blobs that actually failed to load 
Previously, errors would be reported for all blobs of a packfile that
failed to stream. Now, only the not yet processed blobs are reported.
 2024-01-07 10:54:56 +01:00
Michael Eischer 77b1c52673 repository: test that StreamPack only delivers blobs once 2024-01-07 10:54:53 +01:00
Michael Eischer fe5c337ca2 repository: StreamPack delivers blobs at most once 
If an error occurred while streaming a pack file, this could result in
passing some of the blobs multiple times to the callback function. This
significantly complicates using StreamPack correctly and is unnecessary.
Retries do not change the content of a blob and thus only deliver the
same result over and over again.
 2024-01-07 10:54:49 +01:00
Michael Eischer 0360e540af
Merge pull request #4622 from MichaelEischer/fix-outdated-windows-import 
termstatus: update import path of golang.org/x/term
 2024-01-06 23:24:16 +01:00
Michael Eischer e6dfefba13 termstatus: update import path of golang.org/x/term 2024-01-06 21:59:26 +01:00
Michael Eischer 02bc73f5eb s3: minor code cleanups 2024-01-06 21:44:53 +01:00
Michael Eischer 20cf4777cb s3: check for EnvAWS credentials before Static credentials 
EnvAWS considers more environment variables, including AWS_SESSION_TOKEN
and thus should be checked first.
 2024-01-06 21:43:47 +01:00
Erik Kristensen 5ffb536aae feat: support AWS assume role 2024-01-06 21:19:58 +01:00
Michael Eischer 33b7c84a7a deduplicate string formatting of snapshot metadata 
This removes the spurious ")" bracket at the end and normalizes the
metadata format used by the `ls` command.
 2024-01-06 20:20:51 +01:00
Michael Eischer 51419c51d3 archiver: Add filepath to error message if it is not included yet 2024-01-06 19:08:24 +01:00
Michael Eischer 6b79834cc8 archiver: improve error message for irregular files 
Since Go 1.21, most reparse points are considered as irregular files.
Depending on the underlying driver these can exhibit nearly arbitrary
behavior. When encountering such a file, restic returned an
indecipherable error message: `error: invalid node type ""`.

Add the filepath to the error message and state that the file type is
not supported.
 2024-01-06 19:03:11 +01:00
Michael Eischer 0018bb7854 restic: cleanup node type determination 
os.ModeCharDevice is already included in os.ModeType
 2024-01-06 18:43:16 +01:00
Michael Eischer 649a6409ee rewrite: cleanup tests 2023-12-24 15:36:22 +01:00
Michael Eischer 30e6ed038c
Merge pull request #4586 from Gelma/typos 
Fix typos
 2023-12-23 13:21:50 +00:00
Michael Eischer 53ebe91a50 Move changelog typo fixes to changelog files 2023-12-23 14:05:47 +01:00
Michael Eischer dbbd31bc3a CI: update golangci-lint 
Necessary to properly support Go 1.21.
 2023-12-23 13:41:30 +01:00
Andrea Gelmini 241916d55b
Fix typos 2023-12-06 13:11:55 +01:00
Michael Eischer ee305e6041 backup: rework error reporting for subcommand 2023-10-27 23:58:52 +02:00
Michael Eischer 8bceb8e359 fs: add tests for CommandReader 2023-10-27 23:58:52 +02:00
Michael Eischer 317144c1d6 fs: merge command startup into CommandReader 2023-10-27 23:58:51 +02:00
Michael Eischer 7d879705ad fs: cleanup CommandReader implementation 2023-10-27 23:58:51 +02:00
Enrico204 37a312e505 restic-from-command: use standard behavior when no output and exit code 0 from command 
The behavior of the new option should reflect the behavior of normal backups: when the command exit code is zero and there is no output in the stdout, emit a warning but create the snapshot. This commit fixes the integration tests and the ReadCloserCommand struct.
 2023-10-27 23:58:51 +02:00
Enrico204 81f8d473df restic-from-command: abort snapshot on non-zero exit codes 2023-10-27 23:58:51 +02:00
Enrico204 4e5caab114 stdin-from-command: implemented suggestions in #4254 
The code has been refactored so that the archiver is back to the original code, and the stderr is handled using a go routine to avoid deadlock.
 2023-10-27 23:58:51 +02:00
Sebastian Hoß c133065a9f Check command result before snapshotting 
Return with an error containing the stderr of the given command in case it fails. No new snapshot will be created and future prune operations on the repository will remove the unreferenced data.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
 2023-10-27 23:58:51 +02:00
Sebastian Hoß 25350a9c55 Extend SnapshotOptions w/ command data 
In order to determine whether to save a snapshot, we need to capture the exit code returned by a command. In order to provide a nice error message, we supply stderr as well.

Signed-off-by: Sebastian Hoß <seb@xn--ho-hia.de>
 2023-10-27 23:58:51 +02:00
Michael Eischer a8fdcf79b7 restorer: Make hardlink index generic 
This will allow reusing it for the stats command without regressing the
memory usage due to storing an unnecessary file path.
 2023-10-27 23:40:42 +02:00
Michael Eischer 45962c2847
Merge pull request #4499 from MichaelEischer/modular-backend-code 
Split backend code from restic package
 2023-10-27 20:19:20 +02:00
Leo R. Lundgren aafb806a8c doc: Correct two typos 2023-10-27 18:56:32 +02:00
Martin Michlmayr 41e6a02bcc Fix typos 2023-10-27 18:56:32 +02:00
Michael Eischer c7b770eb1f convert MemorizeList to be repository based 
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
 2023-10-25 23:01:35 +02:00
Michael Eischer 1b8a67fe76 move Backend interface to backend package 2023-10-25 23:00:18 +02:00
Michael Eischer ceb0774af1 backend: make LoadAll independent of restic package 2023-10-25 22:58:39 +02:00
Michael Eischer b6d79bdf6f restic: decouple restic.Handle 2023-10-25 22:54:07 +02:00
Michael Eischer 7881309d63 backend: move backend implementation helpers to util package 
This removes code that is only used within a backend implementation from
the backend package. The latter now only contains code that also has
external users.
 2023-10-25 22:54:07 +02:00