CockroachDB uses the URL provided in a BACKUP, RESTORE, IMPORT, or EXPORT statement to construct a secure API call to the service you specify. The URL structure depends on the type of file storage you are using.
We strongly recommend using cloud/remote storage (Amazon S3, Google Cloud Platform, etc.).
URLs for the files you want to import must use the format shown below. For examples, see Example file URLs.
[scheme]://[host]/[path]?[parameters]
| Location | Scheme | Host | Parameters |
|---|---|---|---|
| Amazon | s3 |
Bucket name | AUTH 1 (optional; can be implicit or specified), AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN |
| Azure | azure |
N/A (see Example file URLs | AZURE_ACCOUNT_KEY, AZURE_ACCOUNT_NAME |
| Google Cloud 2 | gs |
Bucket name | AUTH (optional; can be default, implicit, or specified), CREDENTIALS |
| HTTP 3 | http |
Remote host | N/A |
| NFS/Local 4 | nodelocal |
nodeID or self 5 (see Example file URLs) |
N/A |
| S3-compatible services 6 | s3 |
Bucket name | AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, AWS_REGION 7 (optional), AWS_ENDPOINT |
The location parameters often contain special characters that need to be URI-encoded. Use Javascript's encodeURIComponent function or Go language's url.QueryEscape function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.
If your environment requires an HTTP or HTTPS proxy server for outgoing connections, you can set the standard HTTP_PROXY and HTTPS_PROXY environment variables when starting CockroachDB.
If you cannot run a full proxy, you can disable external HTTP(S) access (as well as custom HTTP(S) endpoints) when performing bulk operations (e.g., BACKUP, RESTORE, etc.) by using the --external-io-disable-http flag. You can also disable the use of implicit credentials when accessing external cloud storage services for various bulk operations by using the --external-io-disable-implicit-credentials flag.
1 If the
AUTHparameter is not provided, AWS connections default tospecifiedand the access keys must be provided in the URI parameters. If theAUTHparameter isimplicit, the access keys can be omitted and the credentials will be loaded from the environment.2 If the
AUTHparameter is not specified, thecloudstorage.gs.default.keycluster setting will be used if it is non-empty, otherwise theimplicitbehavior is used. If theAUTHparameter isimplicit, all GCS connections use Google's default authentication strategy. If theAUTHparameter isdefault, thecloudstorage.gs.default.keycluster setting must be set to the contents of a service account file which will be used during authentication. If theAUTHparameter isspecified, GCS connections are authenticated on a per-statement basis, which allows the JSON key object to be sent in theCREDENTIALSparameter. The JSON key object should be base64-encoded (using the standard encoding in RFC 4648).3 You can create your own HTTP server with Caddy or nginx. A custom root CA can be appended to the system's default CAs by setting the
cloudstorage.http.custom_cacluster setting, which will be used when verifying certificates from HTTPS URLs.4 The file system backup location on the NFS drive is relative to the path specified by the
--external-io-dirflag set while starting the node. If the flag is set todisabled, then imports from local directories and NFS drives are disabled.5 Using a
nodeIDis required and the data files will be in theexterndirectory of the specified node. In most cases (including single-node clusters), usingnodelocal://1/<path>is sufficient. Useselfif you do not want to specify anodeID, and the individual data files will be in theexterndirectories of arbitrary nodes; however, to work correctly, each node must have the--external-io-dirflag point to the same NFS mount or other network-backed, shared storage.6 A custom root CA can be appended to the system's default CAs by setting the
cloudstorage.http.custom_cacluster setting, which will be used when verifying certificates from an S3-compatible service.7 The
AWS_REGIONparameter is optional since it is not a required parameter for most S3-compatible services. Specify the parameter only if your S3-compatible service requires it.
Example file URLs
Example URLs for BACKUP, EXPORT, or changefeeds:
| Location | Example |
|---|---|
| Amazon S3 | s3://acme-co/employees?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456 |
| Azure | azure://employees?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co |
| Google Cloud | gs://acme-co |
| HTTP | http://localhost:8080/employees |
| NFS/Local | nodelocal://1/path/employees, nodelocal://self/nfsmount/backups/employees 5 |
URLs for changefeeds should be prepended with experimental-.
Currently, cloud storage sinks (for changefeeds) only work with JSON and emits newline-delimited JSON files.
Example URLs for RESTORE or IMPORT:
| Location | Example |
|---|---|
| Amazon S3 | s3://acme-co/employees.sql?AWS_ACCESS_KEY_ID=123&AWS_SECRET_ACCESS_KEY=456 |
| Azure | azure://employees.sql?AZURE_ACCOUNT_KEY=123&AZURE_ACCOUNT_NAME=acme-co |
| Google Cloud | gs://acme-co/employees.sql |
| HTTP | http://localhost:8080/employees.sql |
| NFS/Local | nodelocal://1/path/employees, nodelocal://self/nfsmount/backups/employees 5 |
Yes
No