Merge branch 'master' into filter-snaps
This commit is contained in:
Christoph Klaffl
commit
e9eb05e840
14
CHANGELIST
14
CHANGELIST
|
|
@ -1,3 +1,17 @@
|
|||
2.2.0 [overall] documentation updates, small fixes (@azmodude, @deviantintegral, @jimsalterjrs, @alexhaydock, @cbreak-black, @kd8bny, @JavaScriptDude, @veeableful, @rsheasby, @Topslakr, @mavhc, @adam-stamand, @joelishness, @jsoref, @dodexahedron, @phreaker0)
|
||||
[syncoid] implemented flag for preserving properties without the zfs -p flag (@phreaker0)
|
||||
[syncoid] implemented target snapshot deletion (@mat813)
|
||||
[syncoid] support bookmarks which are taken in the same second (@delxg, @phreaker0)
|
||||
[syncoid] exit with an error if the specified src dataset doesn't exist (@phreaker0)
|
||||
[syncoid] rollback is now done implicitly instead of explicit (@jimsalterjrs, @phreaker0)
|
||||
[syncoid] append a rand int to the socket name to prevent collisions with parallel invocations (@Gryd3)
|
||||
[syncoid] implemented support for ssh_config(5) files (@endreszabo)
|
||||
[syncoid] snapshot hold/unhold support (@rbike)
|
||||
[sanoid] handle duplicate key definitions gracefully (@phreaker0)
|
||||
[syncoid] implemented removal of conflicting snapshots with force-delete option (@phreaker0)
|
||||
[sanoid] implemented pre pruning script hook (@phreaker0)
|
||||
[syncoid] implemented direct connection support (bypass ssh) for the actual data transfer (@phreaker0)
|
||||
|
||||
2.1.0 [overall] documentation updates, small fixes (@HavardLine, @croadfeldt, @jimsalterjrs, @jim-perkins, @kr4z33, @phreaker0)
|
||||
[syncoid] do not require user to be specified for syncoid (@aerusso)
|
||||
[syncoid] implemented option for keeping sync snaps (@phreaker0)
|
||||
|
|
|
|||
19
README.md
19
README.md
|
|
@ -274,7 +274,7 @@ As of 1.4.18, syncoid also automatically supports and enables resume of interrup
|
|||
|
||||
+ --identifier=
|
||||
|
||||
Adds the given identifier to the snapshot name after "syncoid_" prefix and before the hostname. This enables the use case of reliable replication to multiple targets from the same host. The following chars are allowed: a-z, A-Z, 0-9, _, -, : and . .
|
||||
Adds the given identifier to the snapshot and hold name after "syncoid_" prefix and before the hostname. This enables the use case of reliable replication to multiple targets from the same host. The following chars are allowed: a-z, A-Z, 0-9, _, -, : and . .
|
||||
|
||||
+ -r --recursive
|
||||
|
||||
|
|
@ -316,10 +316,22 @@ As of 1.4.18, syncoid also automatically supports and enables resume of interrup
|
|||
|
||||
This argument tells syncoid to create a zfs bookmark for the newest snapshot after it got replicated successfully. The bookmark name will be equal to the snapshot name. Only works in combination with the --no-sync-snap option. This can be very useful for irregular replication where the last matching snapshot on the source was already deleted but the bookmark remains so a replication is still possible.
|
||||
|
||||
+ --use-hold
|
||||
This argument tells syncoid to add a hold to the newest snapshot on the source and target after replication succeeds and to remove the hold after the next succesful replication. Setting a hold prevents the snapshots from being destroyed. The hold name incldues the identifier if set. This allows for separate holds in case of replication to multiple targets.
|
||||
|
||||
+ --preserve-recordsize
|
||||
|
||||
This argument tells syncoid to set the recordsize on the target before writing any data to it matching the one set on the replication src. This only applies to initial sends.
|
||||
|
||||
+ --preserve-properties
|
||||
|
||||
This argument tells syncoid to get all locally set dataset properties from the source and apply all supported ones on the target before writing any data. It's similar to the '-p' flag for zfs send but also works for encrypted datasets in non raw sends. This only applies to initial sends.
|
||||
|
||||
+ --delete-target-snapshots
|
||||
|
||||
With this argument snapshots which are missing on the source will be destroyed on the target. Use this if you only want to handle snapshots on the source.
|
||||
Note that snapshot deletion is only done after a successful synchronization. If no new snapshots are found, no synchronization is done and no deletion either.
|
||||
|
||||
+ --no-clone-rollback
|
||||
|
||||
Do not rollback clones on target
|
||||
|
|
@ -382,6 +394,11 @@ As of 1.4.18, syncoid also automatically supports and enables resume of interrup
|
|||
|
||||
Use specified identity file as per ssh -i.
|
||||
|
||||
+ --insecure-direct-connection=IP:PORT[,IP:PORT,[TIMEOUT,[mbuffer]]]
|
||||
|
||||
WARNING: This is an insecure option as the data is not encrypted while being sent over the network. Only use if you trust the complete network path.
|
||||
Use a direct tcp connection (with socat and busybox nc/mbuffer) for the actual zfs send/recv stream. All control commands are still executed via the ssh connection. The first address pair is used for connecting to the target host from the source host and the second pair is for listening on the target host. If the later isn't provided the same as the former is used. This can be used for saturating high throughput connection like >= 10GBe network which isn't easy with the overhead off ssh. It can also be useful for encrypted datasets to lower the cpu usage needed for replication but be aware that metadata is NOT ENCRYPTED in this case. The default timeout is 60 seconds and can be overridden by providing it as third argument. By default busybox nc is used for the listeing tcp socket, if mbuffer is preferred specify its name as fourth argument but be aware that mbuffer listens on all interfaces and uses an optionally provided ip address for access restriction (This option can't be used for relaying between two remote hosts)
|
||||
|
||||
+ --quiet
|
||||
|
||||
Suppress non-error output.
|
||||
|
|
|
|||
2
findoid
2
findoid
|
|
@ -4,7 +4,7 @@
|
|||
# from http://www.gnu.org/licenses/gpl-3.0.html on 2014-11-17. A copy should also be available in this
|
||||
# project's Git repository at https://github.com/jimsalterjrs/sanoid/blob/master/LICENSE.
|
||||
|
||||
$::VERSION = '2.1.0';
|
||||
$::VERSION = '2.2.0';
|
||||
|
||||
use strict;
|
||||
use warnings;
|
||||
|
|
|
|||
|
|
@ -1,3 +1,21 @@
|
|||
sanoid (2.2.0) unstable; urgency=medium
|
||||
|
||||
[overall] documentation updates, small fixes (@azmodude, @deviantintegral, @jimsalterjrs, @alexhaydock, @cbreak-black, @kd8bny, @JavaScriptDude, @veeableful, @rsheasby, @Topslakr, @mavhc, @adam-stamand, @joelishness, @jsoref, @dodexahedron, @phreaker0)
|
||||
[syncoid] implemented flag for preserving properties without the zfs -p flag (@phreaker0)
|
||||
[syncoid] implemented target snapshot deletion (@mat813)
|
||||
[syncoid] support bookmarks which are taken in the same second (@delxg, @phreaker0)
|
||||
[syncoid] exit with an error if the specified src dataset doesn't exist (@phreaker0)
|
||||
[syncoid] rollback is now done implicitly instead of explicit (@jimsalterjrs, @phreaker0)
|
||||
[syncoid] append a rand int to the socket name to prevent collisions with parallel invocations (@Gryd3)
|
||||
[syncoid] implemented support for ssh_config(5) files (@endreszabo)
|
||||
[syncoid] snapshot hold/unhold support (@rbike)
|
||||
[sanoid] handle duplicate key definitions gracefully (@phreaker0)
|
||||
[syncoid] implemented removal of conflicting snapshots with force-delete option (@phreaker0)
|
||||
[sanoid] implemented pre pruning script hook (@phreaker0)
|
||||
[syncoid] implemented direct connection support (bypass ssh) for the actual data transfer (@phreaker0)
|
||||
|
||||
-- Jim Salter <github@jrs-s.net> Tue, 18 Jul 2023 10:04:00 +0200
|
||||
|
||||
sanoid (2.1.0) unstable; urgency=medium
|
||||
|
||||
[overall] documentation updates, small fixes (@HavardLine, @croadfeldt, @jimsalterjrs, @jim-perkins, @kr4z33, @phreaker0)
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
%global version 2.1.0
|
||||
%global version 2.2.0
|
||||
%global git_tag v%{version}
|
||||
|
||||
# Enable with systemctl "enable sanoid.timer"
|
||||
|
|
@ -111,6 +111,8 @@ echo "* * * * * root %{_sbindir}/sanoid --cron" > %{buildroot}%{_docdir}/%{name}
|
|||
%endif
|
||||
|
||||
%changelog
|
||||
* Tue Jul 18 2023 Christoph Klaffl <christoph@phreaker.eu> - 2.2.0
|
||||
- Bump to 2.2.0
|
||||
* Tue Nov 24 2020 Christoph Klaffl <christoph@phreaker.eu> - 2.1.0
|
||||
- Bump to 2.1.0
|
||||
* Wed Oct 02 2019 Christoph Klaffl <christoph@phreaker.eu> - 2.0.3
|
||||
|
|
|
|||
6
sanoid
6
sanoid
|
|
@ -4,7 +4,7 @@
|
|||
# from http://www.gnu.org/licenses/gpl-3.0.html on 2014-11-17. A copy should also be available in this
|
||||
# project's Git repository at https://github.com/jimsalterjrs/sanoid/blob/master/LICENSE.
|
||||
|
||||
$::VERSION = '2.1.0';
|
||||
$::VERSION = '2.2.0';
|
||||
my $MINIMUM_DEFAULTS_VERSION = 2;
|
||||
|
||||
use strict;
|
||||
|
|
@ -30,7 +30,7 @@ GetOptions(\%args, "verbose", "debug", "cron", "readonly", "quiet",
|
|||
) or pod2usage(2);
|
||||
|
||||
# If only config directory (or nothing) has been specified, default to --cron --verbose
|
||||
if (keys %args < 2) {
|
||||
if (keys %args < 4) {
|
||||
$args{'cron'} = 1;
|
||||
$args{'verbose'} = 1;
|
||||
}
|
||||
|
|
@ -905,6 +905,8 @@ sub init {
|
|||
warn "duplicate key '$key' in section '$section', using the value from the first occurence and ignoring the others.\n";
|
||||
$ini{$section}{$key} = $value->[0];
|
||||
}
|
||||
# trim
|
||||
$ini{$section}{$key} =~ s/^\s+|\s+$//g;
|
||||
}
|
||||
|
||||
if ($section =~ /^template_/) { next; } # don't process templates directly
|
||||
|
|
|
|||
242
syncoid
242
syncoid
|
|
@ -4,7 +4,7 @@
|
|||
# from http://www.gnu.org/licenses/gpl-3.0.html on 2014-11-17. A copy should also be available in this
|
||||
# project's Git repository at https://github.com/jimsalterjrs/sanoid/blob/master/LICENSE.
|
||||
|
||||
$::VERSION = '2.1.0';
|
||||
$::VERSION = '2.2.0';
|
||||
|
||||
use strict;
|
||||
use warnings;
|
||||
|
|
@ -24,9 +24,11 @@ my %args = ('sshconfig' => '', 'sshkey' => '', 'sshport' => '', 'sshcipher' => '
|
|||
GetOptions(\%args, "no-command-checks", "monitor-version", "compress=s", "dumpsnaps", "recursive|r", "sendoptions=s", "recvoptions=s",
|
||||
"source-bwlimit=s", "target-bwlimit=s", "sshconfig=s", "sshkey=s", "sshport=i", "sshcipher|c=s", "sshoption|o=s@",
|
||||
"debug", "quiet", "no-stream", "no-sync-snap", "no-resume", "exclude=s@", "skip-parent", "identifier=s",
|
||||
"no-clone-handling", "no-privilege-elevation", "force-delete", "no-rollback", "create-bookmark",
|
||||
"no-clone-handling", "no-privilege-elevation", "force-delete", "no-rollback", "create-bookmark", "use-hold",
|
||||
"pv-options=s" => \$pvoptions, "keep-sync-snap", "preserve-recordsize", "mbuffer-size=s" => \$mbuffer_size,
|
||||
"include-snaps=s@", "exclude-snaps=s@", "exclude-datasets=s@") or pod2usage(2);
|
||||
"delete-target-snapshots", "insecure-direct-connection=s", "preserve-properties",
|
||||
"include-snaps=s@", "exclude-snaps=s@", "exclude-datasets=s@")
|
||||
or pod2usage(2);
|
||||
|
||||
my %compressargs = %{compressargset($args{'compress'} || 'default')}; # Can't be done with GetOptions arg, as default still needs to be set
|
||||
|
||||
|
|
@ -106,8 +108,9 @@ my $pscmd = 'ps';
|
|||
|
||||
my $pvcmd = 'pv';
|
||||
my $mbuffercmd = 'mbuffer';
|
||||
my $socatcmd = 'socat';
|
||||
my $sudocmd = 'sudo';
|
||||
my $mbufferoptions = "-q -s 128k -m $mbuffer_size 2>/dev/null";
|
||||
my $mbufferoptions = "-q -s 128k -m $mbuffer_size";
|
||||
# currently using POSIX compatible command to check for program existence because we aren't depending on perl
|
||||
# being present on remote machines.
|
||||
my $checkcmd = 'command -v';
|
||||
|
|
@ -146,6 +149,46 @@ my ($targethost,$targetfs,$targetisroot) = getssh($rawtargetfs);
|
|||
my $sourcesudocmd = $sourceisroot ? '' : $sudocmd;
|
||||
my $targetsudocmd = $targetisroot ? '' : $sudocmd;
|
||||
|
||||
if (!defined $sourcehost) { $sourcehost = ''; }
|
||||
if (!defined $targethost) { $targethost = ''; }
|
||||
|
||||
# handle insecure direct connection arguments
|
||||
my $directconnect = "";
|
||||
my $directlisten = "";
|
||||
my $directtimeout = 60;
|
||||
my $directmbuffer = 0;
|
||||
|
||||
if (length $args{'insecure-direct-connection'}) {
|
||||
if ($sourcehost ne '' && $targethost ne '') {
|
||||
print("CRITICAL: relaying between remote hosts is not supported with insecure direct connection!\n");
|
||||
pod2usage(2);
|
||||
exit 127;
|
||||
}
|
||||
|
||||
my @parts = split(',', $args{'insecure-direct-connection'});
|
||||
if (scalar @parts > 4) {
|
||||
print("CRITICAL: invalid insecure-direct-connection argument!\n");
|
||||
pod2usage(2);
|
||||
exit 127;
|
||||
} elsif (scalar @parts >= 2) {
|
||||
$directconnect = $parts[0];
|
||||
$directlisten = $parts[1];
|
||||
} else {
|
||||
$directconnect = $args{'insecure-direct-connection'};
|
||||
$directlisten = $args{'insecure-direct-connection'};
|
||||
}
|
||||
|
||||
if (scalar @parts == 3) {
|
||||
$directtimeout = $parts[2];
|
||||
}
|
||||
|
||||
if (scalar @parts == 4) {
|
||||
if ($parts[3] eq "mbuffer") {
|
||||
$directmbuffer = 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# figure out whether compression, mbuffering, pv
|
||||
# are available on source, target, local machines.
|
||||
# warn user of anything missing, then continue with sync.
|
||||
|
|
@ -366,6 +409,7 @@ sub syncdataset {
|
|||
}
|
||||
|
||||
my $newsyncsnap;
|
||||
my $matchingsnap;
|
||||
|
||||
# skip snapshot checking/creation in case of resumed receive
|
||||
if (!defined($receivetoken)) {
|
||||
|
|
@ -543,7 +587,7 @@ sub syncdataset {
|
|||
my $bookmark = 0;
|
||||
my $bookmarkcreation = 0;
|
||||
|
||||
my $matchingsnap = getmatchingsnapshot($sourcefs, $targetfs, \%snaps);
|
||||
$matchingsnap = getmatchingsnapshot($sourcefs, $targetfs, \%snaps);
|
||||
if (! $matchingsnap) {
|
||||
# no matching snapshots, check for bookmarks as fallback
|
||||
my %bookmarks = getbookmarks($sourcehost,$sourcefs,$sourceisroot);
|
||||
|
|
@ -715,7 +759,7 @@ sub syncdataset {
|
|||
return 0;
|
||||
}
|
||||
|
||||
if (!$quiet) { print "WARN: removing existing destination: $existing\n"; }
|
||||
writelog('WARN', "removing existing destination: $existing");
|
||||
my $rcommand = '';
|
||||
my $mysudocmd = '';
|
||||
my $existingescaped = escapeshellparam($existing);
|
||||
|
|
@ -753,6 +797,42 @@ sub syncdataset {
|
|||
|
||||
$replicationCount++;
|
||||
|
||||
# if "--use-hold" parameter is used set hold on newsync snapshot and remove hold on matching snapshot both on source and target
|
||||
# hold name: "syncoid" + identifier + hostname -> in case of replication to multiple targets separate holds can be set for each target by assinging different identifiers to each target. Only if all targets have been replicated all syncoid holds are removed from the matching snapshot and it can be removed
|
||||
if (defined $args{'use-hold'}) {
|
||||
my $holdcmd;
|
||||
my $holdreleasecmd;
|
||||
my $hostid = hostname();
|
||||
my $matchingsnapescaped = escapeshellparam($matchingsnap);
|
||||
my $holdname = "syncoid\_$identifier$hostid";
|
||||
if ($sourcehost ne '') {
|
||||
$holdcmd = "$sshcmd $sourcehost " . escapeshellparam("$sourcesudocmd $zfscmd hold $holdname $sourcefsescaped\@$newsyncsnapescaped");
|
||||
$holdreleasecmd = "$sshcmd $sourcehost " . escapeshellparam("$sourcesudocmd $zfscmd release $holdname $sourcefsescaped\@$matchingsnapescaped");
|
||||
} else {
|
||||
$holdcmd = "$sourcesudocmd $zfscmd hold $holdname $sourcefsescaped\@$newsyncsnapescaped";
|
||||
$holdreleasecmd = "$sourcesudocmd $zfscmd release $holdname $sourcefsescaped\@$matchingsnapescaped";
|
||||
}
|
||||
writelog('DEBUG', "Set new hold on source: $holdcmd");
|
||||
system($holdcmd) == 0 or warn "WARNING: $holdcmd failed: $?";
|
||||
# Do hold release only if matchingsnap exists
|
||||
if ($matchingsnap) {
|
||||
writelog('DEBUG', "Release old hold on source: $holdreleasecmd");
|
||||
system($holdreleasecmd) == 0 or warn "WARNING: $holdreleasecmd failed: $?";
|
||||
}
|
||||
if ($targethost ne '') {
|
||||
$holdcmd = "$sshcmd $targethost " . escapeshellparam("$targetsudocmd $zfscmd hold $holdname $targetfsescaped\@$newsyncsnapescaped");
|
||||
$holdreleasecmd = "$sshcmd $targethost " . escapeshellparam("$targetsudocmd $zfscmd release $holdname $targetfsescaped\@$matchingsnapescaped");
|
||||
} else {
|
||||
$holdcmd = "$targetsudocmd $zfscmd hold $holdname $targetfsescaped\@$newsyncsnapescaped"; $holdreleasecmd = "$targetsudocmd $zfscmd release $holdname $targetfsescaped\@$matchingsnapescaped";
|
||||
}
|
||||
writelog('DEBUG', "Set new hold on target: $holdcmd");
|
||||
system($holdcmd) == 0 or warn "WARNING: $holdcmd failed: $?";
|
||||
# Do hold release only if matchingsnap exists
|
||||
if ($matchingsnap) {
|
||||
writelog('DEBUG', "Release old hold on target: $holdreleasecmd");
|
||||
system($holdreleasecmd) == 0 or warn "WARNING: $holdreleasecmd failed: $?";
|
||||
}
|
||||
}
|
||||
if (defined $args{'no-sync-snap'}) {
|
||||
if (defined $args{'create-bookmark'}) {
|
||||
my $ret = createbookmark($sourcehost, $sourcefs, $newsyncsnap, $newsyncsnap);
|
||||
|
|
@ -778,6 +858,29 @@ sub syncdataset {
|
|||
}
|
||||
}
|
||||
|
||||
if (defined $args{'delete-target-snapshots'}) {
|
||||
# Find the snapshots that exist on the target, filter with
|
||||
# those that exist on the source. Remaining are the snapshots
|
||||
# that are only on the target. Then sort by creation date, as
|
||||
# to remove the oldest snapshots first.
|
||||
my @to_delete = sort { $snaps{'target'}{$a}{'creation'}<=>$snaps{'target'}{$b}{'creation'} } grep {!exists $snaps{'source'}{$_}} keys %{ $snaps{'target'} };
|
||||
while (@to_delete) {
|
||||
# Create batch of snapshots to remove
|
||||
my $snaps = join ',', splice(@to_delete, 0, 50);
|
||||
my $command;
|
||||
if ($targethost ne '') {
|
||||
$command = "$sshcmd $targethost " . escapeshellparam("$targetsudocmd $zfscmd destroy $targetfsescaped\@$snaps");
|
||||
} else {
|
||||
$command = "$targetsudocmd $zfscmd destroy $targetfsescaped\@$snaps";
|
||||
}
|
||||
writelog('DEBUG', "$command");
|
||||
my ($stdout, $stderr, $result) = capture { system $command; };
|
||||
if ($result != 0 && !$quiet) {
|
||||
warn "$command failed: $stderr";
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
} # end syncdataset()
|
||||
|
||||
# Return codes:
|
||||
|
|
@ -808,7 +911,15 @@ sub runsynccmd {
|
|||
# if no rollbacks are allowed, disable forced receive
|
||||
if (!defined $args{'no-rollback'}) { $recvoptions .= ' -F'; }
|
||||
|
||||
if (defined $args{'preserve-recordsize'}) {
|
||||
if (defined $args{'preserve-properties'}) {
|
||||
my %properties = getlocalzfsvalues($sourcehost,$sourcefs,$sourceisroot);
|
||||
|
||||
foreach my $key (keys %properties) {
|
||||
my $value = $properties{$key};
|
||||
writelog('DEBUG', "will set $key to $value ...");
|
||||
$recvoptions .= " -o $key=$value";
|
||||
}
|
||||
} elsif (defined $args{'preserve-recordsize'}) {
|
||||
my $type = getzfsvalue($sourcehost,$sourcefs,$sourceisroot,'type');
|
||||
if ($type eq "filesystem") {
|
||||
my $recordsize = getzfsvalue($sourcehost,$sourcefs,$sourceisroot,'recordsize');
|
||||
|
|
@ -1057,9 +1168,6 @@ sub checkcommands {
|
|||
return %avail;
|
||||
}
|
||||
|
||||
if (!defined $sourcehost) { $sourcehost = ''; }
|
||||
if (!defined $targethost) { $targethost = ''; }
|
||||
|
||||
if ($sourcehost ne '') { $sourcessh = "$sshcmd $sourcehost"; } else { $sourcessh = ''; }
|
||||
if ($targethost ne '') { $targetssh = "$sshcmd $targethost"; } else { $targetssh = ''; }
|
||||
|
||||
|
|
@ -1125,6 +1233,22 @@ sub checkcommands {
|
|||
$avail{'compress'} = 0;
|
||||
}
|
||||
|
||||
if (length $args{'insecure-direct-connection'}) {
|
||||
writelog('DEBUG', "checking availability of $socatcmd on source...");
|
||||
my $socatAvailable = `$sourcessh $checkcmd $socatcmd 2>/dev/null`;
|
||||
if ($socatAvailable eq '') {
|
||||
die "CRIT: $socatcmd is needed on source for insecure direct connection!\n";
|
||||
}
|
||||
|
||||
if (!$directmbuffer) {
|
||||
writelog('DEBUG', "checking availability of busybox (for nc) on target...");
|
||||
my $busyboxAvailable = `$targetssh $checkcmd busybox 2>/dev/null`;
|
||||
if ($busyboxAvailable eq '') {
|
||||
die "CRIT: busybox is needed on target for insecure direct connection!\n";
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
writelog('DEBUG', "checking availability of $mbuffercmd on source...");
|
||||
$avail{'sourcembuffer'} = `$sourcessh $checkcmd $mbuffercmd 2>/dev/null`;
|
||||
if ($avail{'sourcembuffer'} eq '') {
|
||||
|
|
@ -1137,6 +1261,9 @@ sub checkcommands {
|
|||
writelog('DEBUG', "checking availability of $mbuffercmd on target...");
|
||||
$avail{'targetmbuffer'} = `$targetssh $checkcmd $mbuffercmd 2>/dev/null`;
|
||||
if ($avail{'targetmbuffer'} eq '') {
|
||||
if ($directmbuffer) {
|
||||
die "CRIT: $mbuffercmd is needed on target for insecure direct connection!\n";
|
||||
}
|
||||
writelog('WARN', "$mbuffercmd not available on target $t - sync will continue without target buffering.");
|
||||
$avail{'targetmbuffer'} = 0;
|
||||
} else {
|
||||
|
|
@ -1288,6 +1415,55 @@ sub getzfsvalue {
|
|||
return $wantarray ? ($value, $error) : $value;
|
||||
}
|
||||
|
||||
sub getlocalzfsvalues {
|
||||
my ($rhost,$fs,$isroot) = @_;
|
||||
|
||||
my $fsescaped = escapeshellparam($fs);
|
||||
|
||||
if ($rhost ne '') {
|
||||
$rhost = "$sshcmd $rhost";
|
||||
# double escaping needed
|
||||
$fsescaped = escapeshellparam($fsescaped);
|
||||
}
|
||||
|
||||
writelog('DEBUG', "getting locally set values of properties on $fs...");
|
||||
my $mysudocmd;
|
||||
if ($isroot) { $mysudocmd = ''; } else { $mysudocmd = $sudocmd; }
|
||||
writelog('DEBUG', "$rhost $mysudocmd $zfscmd get all -s local -H $fsescaped");
|
||||
my ($values, $error, $exit) = capture {
|
||||
system("$rhost $mysudocmd $zfscmd get all -s local -H $fsescaped");
|
||||
};
|
||||
|
||||
my %properties=();
|
||||
|
||||
if ($exit != 0) {
|
||||
warn "WARNING: getlocalzfsvalues failed for $fs: $error";
|
||||
if ($exitcode < 1) { $exitcode = 1; }
|
||||
return %properties;
|
||||
}
|
||||
|
||||
my @blacklist = (
|
||||
"available", "compressratio", "createtxg", "creation", "clones",
|
||||
"defer_destroy", "encryptionroot", "filesystem_count", "keystatus", "guid",
|
||||
"logicalreferenced", "logicalused", "mounted", "objsetid", "origin",
|
||||
"receive_resume_token", "redact_snaps", "referenced", "refcompressratio", "snapshot_count",
|
||||
"type", "used", "usedbychildren", "usedbydataset", "usedbyrefreservation",
|
||||
"usedbysnapshots", "userrefs", "snapshots_changed", "volblocksize", "written",
|
||||
"version", "volsize", "casesensitivity", "normalization", "utf8only"
|
||||
);
|
||||
my %blacklisthash = map {$_ => 1} @blacklist;
|
||||
|
||||
foreach (split(/\n/,$values)) {
|
||||
my @parts = split(/\t/, $_);
|
||||
if (exists $blacklisthash{$parts[1]}) {
|
||||
next;
|
||||
}
|
||||
$properties{$parts[1]} = $parts[2];
|
||||
}
|
||||
|
||||
return %properties;
|
||||
}
|
||||
|
||||
sub readablebytes {
|
||||
my $bytes = shift;
|
||||
my $disp;
|
||||
|
|
@ -1367,10 +1543,19 @@ sub buildsynccmd {
|
|||
if ($avail{'localpv'} && !$quiet) { $synccmd .= " $pvcmd $pvoptions -s $pvsize |"; }
|
||||
if ($avail{'compress'}) { $synccmd .= " $compressargs{'cmd'} |"; }
|
||||
if ($avail{'sourcembuffer'}) { $synccmd .= " $mbuffercmd $args{'source-bwlimit'} $mbufferoptions |"; }
|
||||
if (length $directconnect) {
|
||||
$synccmd .= " $socatcmd - TCP:" . $directconnect . ",retry=$directtimeout,interval=1 |";
|
||||
}
|
||||
$synccmd .= " $sshcmd $targethost ";
|
||||
|
||||
my $remotecmd = "";
|
||||
if ($avail{'targetmbuffer'}) { $remotecmd .= " $mbuffercmd $args{'target-bwlimit'} $mbufferoptions |"; }
|
||||
if ($directmbuffer) {
|
||||
$remotecmd .= " $mbuffercmd $args{'target-bwlimit'} -W $directtimeout -I " . $directlisten . " $mbufferoptions |";
|
||||
} elsif (length $directlisten) {
|
||||
$remotecmd .= " busybox nc -l " . $directlisten . " -w $directtimeout |";
|
||||
}
|
||||
|
||||
if ($avail{'targetmbuffer'} && !$directmbuffer) { $remotecmd .= " $mbuffercmd $args{'target-bwlimit'} $mbufferoptions |"; }
|
||||
if ($avail{'compress'}) { $remotecmd .= " $compressargs{'decomcmd'} |"; }
|
||||
$remotecmd .= " $recvcmd";
|
||||
|
||||
|
|
@ -1382,10 +1567,19 @@ sub buildsynccmd {
|
|||
my $remotecmd = $sendcmd;
|
||||
if ($avail{'compress'}) { $remotecmd .= " | $compressargs{'cmd'}"; }
|
||||
if ($avail{'sourcembuffer'}) { $remotecmd .= " | $mbuffercmd $args{'source-bwlimit'} $mbufferoptions"; }
|
||||
if (length $directconnect) {
|
||||
$remotecmd .= " | $socatcmd - TCP:" . $directconnect . ",retry=$directtimeout,interval=1";
|
||||
}
|
||||
|
||||
$synccmd = "$sshcmd $sourcehost " . escapeshellparam($remotecmd);
|
||||
$synccmd .= " | ";
|
||||
if ($avail{'targetmbuffer'}) { $synccmd .= "$mbuffercmd $args{'target-bwlimit'} $mbufferoptions | "; }
|
||||
if ($directmbuffer) {
|
||||
$synccmd .= "$mbuffercmd $args{'target-bwlimit'} -W $directtimeout -I " . $directlisten . " $mbufferoptions | ";
|
||||
} elsif (length $directlisten) {
|
||||
$synccmd .= " busybox nc -l " . $directlisten . " -w $directtimeout | ";
|
||||
}
|
||||
|
||||
if ($avail{'targetmbuffer'} && !$directmbuffer) { $synccmd .= "$mbuffercmd $args{'target-bwlimit'} $mbufferoptions | "; }
|
||||
if ($avail{'compress'}) { $synccmd .= "$compressargs{'decomcmd'} | "; }
|
||||
if ($avail{'localpv'} && !$quiet) { $synccmd .= "$pvcmd $pvoptions -s $pvsize | "; }
|
||||
$synccmd .= "$recvcmd";
|
||||
|
|
@ -1803,6 +1997,7 @@ sub getbookmarks() {
|
|||
# as though each were an entirely separate get command.
|
||||
|
||||
my $lastguid;
|
||||
my %creationtimes=();
|
||||
|
||||
foreach my $line (@rawbookmarks) {
|
||||
# only import bookmark guids, creation from the specified filesystem
|
||||
|
|
@ -1819,7 +2014,24 @@ sub getbookmarks() {
|
|||
$creation =~ s/^.*\tcreation\t*(\d*).*/$1/;
|
||||
my $bookmark = $line;
|
||||
$bookmark =~ s/^.*\#(.*)\tcreation.*$/$1/;
|
||||
$bookmarks{$lastguid}{'creation'}=$creation . "000";
|
||||
|
||||
# the accuracy of the creation timestamp is only for a second, but
|
||||
# bookmarks in the same second are possible. The list command
|
||||
# has an ordered output so we append another three digit running number
|
||||
# to the creation timestamp and make sure those are ordered correctly
|
||||
# for bookmarks with the same creation timestamp
|
||||
my $counter = 0;
|
||||
my $creationsuffix;
|
||||
while ($counter < 999) {
|
||||
$creationsuffix = sprintf("%s%03d", $creation, $counter);
|
||||
if (!defined $creationtimes{$creationsuffix}) {
|
||||
$creationtimes{$creationsuffix} = 1;
|
||||
last;
|
||||
}
|
||||
$counter += 1;
|
||||
}
|
||||
|
||||
$bookmarks{$lastguid}{'creation'}=$creationsuffix;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -2121,8 +2333,11 @@ Options:
|
|||
--no-sync-snap Does not create new snapshot, only transfers existing
|
||||
--keep-sync-snap Don't destroy created sync snapshots
|
||||
--create-bookmark Creates a zfs bookmark for the newest snapshot on the source after replication succeeds (only works with --no-sync-snap)
|
||||
--use-hold Adds a hold to the newest snapshot on the source and target after replication succeeds and removes the hold after the next succesful replication. The hold name incldues the identifier if set. This allows for separate holds in case of multiple targets
|
||||
--preserve-recordsize Preserves the recordsize on initial sends to the target
|
||||
--preserve-properties Preserves locally set dataset properties similiar to the zfs send -p flag but this one will also work for encrypted datasets in non raw sends
|
||||
--no-rollback Does not rollback snapshots on target (it probably requires a readonly target)
|
||||
--delete-target-snapshots With this argument snapshots which are missing on the source will be destroyed on the target. Use this if you only want to handle snapshots on the source.
|
||||
--exclude=REGEX DEPRECATED. Equivalent to --exclude-datasets, but will be removed in a future release. Ignored if --exclude-datasets is also provided.
|
||||
--exclude-datasets=REGEX Exclude specific datasets which match the given regular expression. Can be specified multiple times
|
||||
--exclude-snaps=REGEX Exclude specific snapshots that match the given regular expression. Can be specified multiple times. If a snapshot matches both the exclude-snaps and include-snaps patterns, then it will be excluded.
|
||||
|
|
@ -2134,6 +2349,7 @@ Options:
|
|||
--sshport=PORT Connects to remote on a particular port
|
||||
--sshcipher|c=CIPHER Passes CIPHER to ssh to use a particular cipher set
|
||||
--sshoption|o=OPTION Passes OPTION to ssh for remote usage. Can be specified multiple times
|
||||
--insecure-direct-connection=IP:PORT[,IP:PORT] WARNING: DATA IS NOT ENCRYPTED. First address pair is for connecting to the target and the second for listening at the target
|
||||
|
||||
--help Prints this helptext
|
||||
--version Prints the version number
|
||||
|
|
|
|||
|
|
@ -7,10 +7,10 @@ set -e
|
|||
|
||||
. ../../common/lib.sh
|
||||
|
||||
POOL_IMAGE="/tmp/syncoid-test-8.zpool"
|
||||
MOUNT_TARGET="/tmp/syncoid-test-8.mount"
|
||||
POOL_IMAGE="/tmp/syncoid-test-10.zpool"
|
||||
MOUNT_TARGET="/tmp/syncoid-test-10.mount"
|
||||
POOL_SIZE="100M"
|
||||
POOL_NAME="syncoid-test-8"
|
||||
POOL_NAME="syncoid-test-10"
|
||||
|
||||
truncate -s "${POOL_SIZE}" "${POOL_IMAGE}"
|
||||
|
||||
|
|
@ -81,7 +81,7 @@ setup_snaps
|
|||
#####
|
||||
|
||||
../../../syncoid --debug --compress=none --no-sync-snap --exclude-snaps='hourly' "${POOL_NAME}"/src "${POOL_NAME}"/dst
|
||||
verify_checksum 'fb408c21b8540b3c1bd04781b6091d77ff9432defef3303c1a34321b45e8b6a9 -'
|
||||
verify_checksum '494b6860415607f1d670e4106a10e1316924ba6cd31b4ddacffe0ad6d30a6339 -'
|
||||
clean_snaps
|
||||
|
||||
#####
|
||||
|
|
@ -92,7 +92,7 @@ clean_snaps
|
|||
#####
|
||||
|
||||
../../../syncoid --debug --compress=none --no-sync-snap --exclude-snaps='hourly' --no-stream "${POOL_NAME}"/src "${POOL_NAME}"/dst
|
||||
verify_checksum 'c9ad1d3e07156847f957509fcd4805edc7d4c91fe955c605ac4335076367d19a -'
|
||||
verify_checksum '0a5072f42180d231cfdd678682972fbbb689140b7f3e996b3c348b7e78d67ea2 -'
|
||||
clean_snaps
|
||||
|
||||
#####
|
||||
|
|
@ -103,7 +103,7 @@ clean_snaps
|
|||
#####
|
||||
|
||||
../../../syncoid --debug --compress=none --no-sync-snap --include-snaps='hourly' "${POOL_NAME}"/src "${POOL_NAME}"/dst
|
||||
verify_checksum 'f2fb62a2b475bec85796dbf4f6c02af5b4ccaca01f9995ef3d0909787213cbde -'
|
||||
verify_checksum 'd32862be4c71c6cde846322a7d006fd5e8edbd3520d3c7b73953492946debb7f -'
|
||||
clean_snaps
|
||||
|
||||
#####
|
||||
|
|
@ -114,7 +114,7 @@ clean_snaps
|
|||
#####
|
||||
|
||||
../../../syncoid --debug --compress=none --no-sync-snap --include-snaps='hourly' --no-stream "${POOL_NAME}"/src "${POOL_NAME}"/dst
|
||||
verify_checksum '194e60e9d635783f7c7d64e2b0d9f0897c926e69a86ffa2858cf0ca874ffeeb4 -'
|
||||
verify_checksum '81ef1a8298006a7ed856430bb7e05e8b85bbff530ca9dd7831f1da782f8aa4c7 -'
|
||||
clean_snaps
|
||||
|
||||
#####
|
||||
|
|
@ -126,7 +126,7 @@ clean_snaps
|
|||
#####
|
||||
|
||||
../../../syncoid --debug --compress=none --no-sync-snap --include-snaps='hourly' --exclude-snaps='3' "${POOL_NAME}"/src "${POOL_NAME}"/dst
|
||||
verify_checksum '55267405e346e64d6f7eed29d62bc9bb9ea0e15c9515103a92ee47a7439a99a2 -'
|
||||
verify_checksum '5a9dd92b7d4b8760a1fcad03be843da4f43b915c64caffc1700c0d59a1581239 -'
|
||||
clean_snaps
|
||||
|
||||
#####
|
||||
|
|
@ -138,5 +138,5 @@ clean_snaps
|
|||
#####
|
||||
|
||||
../../../syncoid --debug --compress=none --no-stream --exclude-snaps='syncoid' "${POOL_NAME}"/src "${POOL_NAME}"/dst
|
||||
verify_checksum '47380e1711d08c46fb1691fa4bd65e5551084fd5b961baa2de7f91feff2cb4b8 -'
|
||||
verify_checksum '9394fdac44ec72764a4673202552599684c83530a2a724dae5b411aaea082b02 -'
|
||||
clean_snaps
|
||||
|
|
@ -45,6 +45,9 @@ wait
|
|||
sleep 1
|
||||
|
||||
../../../syncoid --debug --compress=none --no-resume "${POOL_NAME}"/src "${POOL_NAME}"/dst | grep "reset partial receive state of syncoid"
|
||||
|
||||
sleep 1
|
||||
|
||||
../../../syncoid --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst
|
||||
|
||||
exit $?
|
||||
|
|
|
|||
|
|
@ -47,6 +47,9 @@ sleep 1
|
|||
|
||||
zfs destroy "${POOL_NAME}"/src@big
|
||||
../../../syncoid --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst # | grep "reset partial receive state of syncoid"
|
||||
|
||||
sleep 1
|
||||
|
||||
../../../syncoid --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst
|
||||
|
||||
exit $?
|
||||
|
|
|
|||
|
|
@ -0,0 +1,66 @@
|
|||
#!/bin/bash
|
||||
|
||||
# test preserving locally set properties from the src dataset to the target one
|
||||
|
||||
set -x
|
||||
set -e
|
||||
|
||||
. ../../common/lib.sh
|
||||
|
||||
POOL_IMAGE="/tmp/syncoid-test-9.zpool"
|
||||
MOUNT_TARGET="/tmp/syncoid-test-9.mount"
|
||||
POOL_SIZE="1000M"
|
||||
POOL_NAME="syncoid-test-9"
|
||||
|
||||
truncate -s "${POOL_SIZE}" "${POOL_IMAGE}"
|
||||
|
||||
zpool create -m none -f "${POOL_NAME}" "${POOL_IMAGE}"
|
||||
|
||||
function cleanUp {
|
||||
zpool export "${POOL_NAME}"
|
||||
}
|
||||
|
||||
# export pool in any case
|
||||
trap cleanUp EXIT
|
||||
|
||||
zfs create -o recordsize=16k -o xattr=on -o mountpoint=none -o primarycache=none "${POOL_NAME}"/src
|
||||
zfs create -V 100M -o volblocksize=8k "${POOL_NAME}"/src/zvol8
|
||||
zfs create -V 100M -o volblocksize=16k -o primarycache=all "${POOL_NAME}"/src/zvol16
|
||||
zfs create -V 100M -o volblocksize=64k "${POOL_NAME}"/src/zvol64
|
||||
zfs create -o recordsize=16k -o primarycache=none "${POOL_NAME}"/src/16
|
||||
zfs create -o recordsize=32k -o acltype=posixacl "${POOL_NAME}"/src/32
|
||||
|
||||
../../../syncoid --preserve-properties --recursive --debug --compress=none "${POOL_NAME}"/src "${POOL_NAME}"/dst
|
||||
|
||||
|
||||
if [ "$(zfs get recordsize -H -o value -t filesystem "${POOL_NAME}"/dst)" != "16K" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$(zfs get mountpoint -H -o value -t filesystem "${POOL_NAME}"/dst)" != "none" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$(zfs get xattr -H -o value -t filesystem "${POOL_NAME}"/dst)" != "on" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$(zfs get primarycache -H -o value -t filesystem "${POOL_NAME}"/dst)" != "none" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$(zfs get recordsize -H -o value -t filesystem "${POOL_NAME}"/dst/16)" != "16K" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$(zfs get primarycache -H -o value -t filesystem "${POOL_NAME}"/dst/16)" != "none" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$(zfs get recordsize -H -o value -t filesystem "${POOL_NAME}"/dst/32)" != "32K" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$(zfs get acltype -H -o value -t filesystem "${POOL_NAME}"/dst/32)" != "posix" ]; then
|
||||
exit 1
|
||||
fi
|
||||
Loading…
Reference in New Issue