Protocol Documentation

Table of Contents

• klio_admin.proto

  • DeleteBackupRequest

  • DeleteBackupResponse

  • FailedBackup

  • FailedWAL

  • ListBackupsRequest

  • ListBackupsResult

  • QueueDiscardBackupsRequest

  • QueueDiscardResponse

  • QueueDiscardWALsRequest

  • QueueListFailedBackupsRequest

  • QueueListFailedBackupsResponse

  • QueueListFailedWALsRequest

  • QueueListFailedWALsResponse

  • QueueRetryBackupsRequest

  • QueueRetryResponse

  • QueueRetryWALsRequest

  • QueueStatusRequest

  • QueueStatusResponse

  • RefreshRequest

  • RefreshResult

  • Tier

  • Admin

• klio_wal.proto

  • CloseBackupRequest

  • CloseBackupResult

  • ClusterMetadata

  • GetMetadataRequest

  • GetRequest

  • GetResult

  • GetRetentionPolicyRequest

  • GetRetentionPolicyResult

  • PutRequest

  • PutResult

  • RequestWALStartRequest

  • RequestWALStartResult

  • ResetWALStreamRequest

  • ResetWALStreamResult

  • RetentionPolicy

  • SetRetentionPolicyRequest

  • SetRetentionPolicyResult

  • StartWALFile

  • TierRetentionPolicy

  • WALGap

  • WAL

• Scalar Value Types

Top

klio_admin.proto

DeleteBackupRequest

DeleteBackupRequest is the request to delete a backup from the server.

Field	Type	Label	Description
backup_name	string		The name of the backup to delete.
tiers	Tier	repeated	The storage tiers from which to delete the backup. At least one tier must be specified.
cluster_name	string		The name of the cluster that owns the backup.

DeleteBackupResponse

DeleteBackupResponse is the response to a backup deletion request.

FailedBackup

Field	Type	Label	Description
cluster_name	string
last_attempt_time	google.protobuf.Timestamp

FailedWAL

Field	Type	Label	Description
cluster_name	string
wal_name	string
sequence	uint64
last_attempt_time	google.protobuf.Timestamp

ListBackupsRequest

ListBackupsResult

Field	Type	Label	Description
backup_manifests	bytes		JSON-serialized array of backup manifests. Each manifest contains fields like: - id: string - cluster_name: string - timestamp: RFC3339 string - size_bytes: number See klioclient.BackupManifest for the canonical structure. We use JSON bytes here to avoid duplicating the internal type definition and conversion logic, as this is a local admin API.

QueueDiscardBackupsRequest

Field	Type	Label	Description
cluster_name	string	optional	Cluster whose failed tasks are discarded. When unset, the failed tasks of every cluster are discarded.

QueueDiscardResponse

Field	Type	Label	Description
discarded	uint64		Number of failed tasks removed from the dead-letter queue.

QueueDiscardWALsRequest

Field	Type	Label	Description
cluster_name	string	optional	Cluster whose failed tasks are discarded. When unset, the failed tasks of every cluster are discarded.
wal_names	string	repeated	WAL files whose failed tasks are discarded, which requires cluster_name. When empty, every failed WAL task of the cluster is discarded.

QueueListFailedBackupsRequest

Field	Type	Label	Description
cluster_name	string	optional

QueueListFailedBackupsResponse

Field	Type	Label	Description
backups	FailedBackup	repeated

QueueListFailedWALsRequest

Field	Type	Label	Description
cluster_name	string	optional

QueueListFailedWALsResponse

Field	Type	Label	Description
wals	FailedWAL	repeated

QueueRetryBackupsRequest

Field	Type	Label	Description
cluster_name	string	optional

QueueRetryResponse

QueueRetryWALsRequest

Field	Type	Label	Description
cluster_name	string	optional
wal_names	string	repeated

QueueStatusRequest

QueueStatusResponse

Field	Type	Label	Description
pending_backups	uint64		Number of backup synchronization tasks pending in the queue
pending_wals	uint64		Number of WAL relay tasks pending in the queue

RefreshRequest

RefreshResult

Tier

Tier represents a storage tier in the backup system.

Name	Number	Description
TIER_UNSPECIFIED	0	TIER_UNSPECIFIED is the default value and should not be used.
TIER_1	1	TIER_1 represents the local cache storage.
TIER_2	2	TIER_2 represents the object storage.

Admin

Method Name	Request Type	Response Type	Description
Refresh	RefreshRequest	RefreshResult	Invoked to refresh the policies and the cache of the Kopia server
ListBackups	ListBackupsRequest	ListBackupsResult	List every backup on the server
QueueListFailedBackups	QueueListFailedBackupsRequest	QueueListFailedBackupsResponse	List backups failed to be processed from the queue
QueueListFailedWALs	QueueListFailedWALsRequest	QueueListFailedWALsResponse	List WAL files failed to be processed from the queue
QueueRetryWALs	QueueRetryWALsRequest	QueueRetryResponse	Retry WAL files that failed to be processed from the queue
QueueDiscardWALs	QueueDiscardWALsRequest	QueueDiscardResponse	Discard the failed WAL tasks from the queue, not the WAL files themselves
QueueRetryBackups	QueueRetryBackupsRequest	QueueRetryResponse	Retry Backups that failed to be processed from the queue
QueueDiscardBackups	QueueDiscardBackupsRequest	QueueDiscardResponse	Discard the failed backup tasks from the queue, not the backups themselves
QueueStatus	QueueStatusRequest	QueueStatusResponse	Get the status of the task queue (pending backups and WALs)
DeleteBackup	DeleteBackupRequest	DeleteBackupResponse	Delete a backup from the server

Top

klio_wal.proto

CloseBackupRequest

This is sent to the WAL server every time a backup has been completed.

Field	Type	Label	Description
cluster_name	string		The name of the cluster.
backup_name	string		The name of the backup.
timeline	int32		The backup timeline
start_wal	string		the first WAL required to restore this backup.
end_wal	string		The last WAL required to restore this backup.
segment_size	uint64		The size of a WAL segment. Needed to generate the sequence of WAL files between the start and the end.
send_to_tier2	bool		Require this backup to be sent to tier2.
tier2_compression_policy	string		When present, set the tier2 compression policy to the specified JSON-serialized policy.

CloseBackupResult

This is sent by the WAL server in response to a CloseBackupRequest message.

Field	Type	Label	Description
tier2_schedule	bool		True when the backup has been scheduled to be synchronized to tier2 storage
missing_wal_files	string	repeated	List of WAL files needed by this backup but still not uploaded to tier1

ClusterMetadata

The following messages are written in the cluster metadata file

Field	Type	Label	Description
system_id	string		The system ID of the current cluster
gaps	WALGap	repeated	The gaps we are aware of in the collected WALs.

GetMetadataRequest

Field	Type	Label	Description
cluster_name	string

GetRequest

Field	Type	Label	Description
cluster_name	string
wal_name	string

GetResult

Field	Type	Label	Description
wal_block	bytes
segment_size	uint64

GetRetentionPolicyRequest

Field	Type	Label	Description
cluster_name	string

GetRetentionPolicyResult

Field	Type	Label	Description
retention_policy	RetentionPolicy

PutRequest

Field	Type	Label	Description
cluster_name	string
wal_name	string
wal_block	bytes
segment_size	uint64
wal_start_lsn	uint64
send_to_tier2	bool

PutResult

Field	Type	Label	Description
write_lsn	uint64
flush_lsn	uint64

RequestWALStartRequest

Field	Type	Label	Description
cluster_name	string		This is the cluster name
system_id	string		This is the system ID
current_wal_name	string		This is the current WAL name that is being written by PostgreSQL. If empty, the start WAL name will be found by looking at the stored WAL files.

RequestWALStartResult

Field	Type	Label	Description
wal_name	string		The WAL file where the client is expected to start streaming.

ResetWALStreamRequest

Field	Type	Label	Description
cluster_name	string		This is the cluster name
system_id	string		This is the system ID
current_wal_name	string		This is the current WAL name that is being written by PostgreSQL. If empty, the start WAL name will be found by looking at the stored WAL files.

ResetWALStreamResult

Field	Type	Label	Description
wal_name	string		The WAL file where the client is expected to start streaming.

RetentionPolicy

Field	Type	Label	Description
tier1_policy	TierRetentionPolicy
tier2_policy	TierRetentionPolicy

SetRetentionPolicyRequest

Field	Type	Label	Description
cluster_name	string
retention_policy	RetentionPolicy

SetRetentionPolicyResult

StartWALFile

The following messages are used to write a WAL file in the Klio WAL Storage area

Field	Type	Label	Description
klio_version	uint64
file_length	uint64

TierRetentionPolicy

Field	Type	Label	Description
latest	int32

WALGap

WALGap is a know gap in the WAL collection process. This is usually caused by an invocation of the reset-lsn Klio feature.

Field	Type	Label	Description
ts	google.protobuf.Timestamp		When this gap was detected and created.
start	string		When the gap started.
end	string		When the gap ends.

WAL

Method Name	Request Type	Response Type	Description
Put	PutRequest stream	PutResult stream
Get	GetRequest	GetResult stream
GetMetadata	GetMetadataRequest	ClusterMetadata
RequestWALStart	RequestWALStartRequest	RequestWALStartResult
ResetWALStream	ResetWALStreamRequest	ResetWALStreamResult
CloseBackup	CloseBackupRequest	CloseBackupResult
SetRetentionPolicy	SetRetentionPolicyRequest	SetRetentionPolicyResult
GetRetentionPolicy	GetRetentionPolicyRequest	GetRetentionPolicyResult

Scalar Value Types

.proto Type	Notes	C++	Java	Python	Go	C#	PHP	Ruby
double		double	double	float	float64	double	float	Float
float		float	float	float	float32	float	float	Float
int32	Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint32 instead.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
int64	Uses variable-length encoding. Inefficient for encoding negative numbers – if your field is likely to have negative values, use sint64 instead.	int64	long	int/long	int64	long	integer/string	Bignum
uint32	Uses variable-length encoding.	uint32	int	int/long	uint32	uint	integer	Bignum or Fixnum (as required)
uint64	Uses variable-length encoding.	uint64	long	int/long	uint64	ulong	integer/string	Bignum or Fixnum (as required)
sint32	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
sint64	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s.	int64	long	int/long	int64	long	integer/string	Bignum
fixed32	Always four bytes. More efficient than uint32 if values are often greater than 2^28.	uint32	int	int	uint32	uint	integer	Bignum or Fixnum (as required)
fixed64	Always eight bytes. More efficient than uint64 if values are often greater than 2^56.	uint64	long	int/long	uint64	ulong	integer/string	Bignum
sfixed32	Always four bytes.	int32	int	int	int32	int	integer	Bignum or Fixnum (as required)
sfixed64	Always eight bytes.	int64	long	int/long	int64	long	integer/string	Bignum
bool		bool	boolean	boolean	bool	bool	boolean	TrueClass/FalseClass
string	A string must always contain UTF-8 encoded or 7-bit ASCII text.	string	String	str/unicode	string	string	string	String (UTF-8)
bytes	May contain any arbitrary sequence of bytes.	string	ByteString	str	[]byte	ByteString	string	String (ASCII-8BIT)