2016-01-29 20:53:56 +01:00
|
|
|
# go-getter
|
|
|
|
|
|
|
|
[![Build Status](http://img.shields.io/travis/hashicorp/go-getter.svg?style=flat-square)][travis]
|
2016-03-20 23:01:11 +01:00
|
|
|
[![Build status](https://ci.appveyor.com/api/projects/status/ulq3qr43n62croyq/branch/master?svg=true)][appveyor]
|
2016-01-29 20:53:56 +01:00
|
|
|
[![Go Documentation](http://img.shields.io/badge/go-documentation-blue.svg?style=flat-square)][godocs]
|
|
|
|
|
|
|
|
[travis]: http://travis-ci.org/hashicorp/go-getter
|
|
|
|
[godocs]: http://godoc.org/github.com/hashicorp/go-getter
|
2016-03-20 23:01:11 +01:00
|
|
|
[appveyor]: https://ci.appveyor.com/project/hashicorp/go-getter/branch/master
|
2016-01-29 20:53:56 +01:00
|
|
|
|
|
|
|
go-getter is a library for Go (golang) for downloading files or directories
|
|
|
|
from various sources using a URL as the primary form of input.
|
|
|
|
|
|
|
|
The power of this library is being flexible in being able to download
|
|
|
|
from a number of different sources (file paths, Git, HTTP, Mercurial, etc.)
|
|
|
|
using a single string as input. This removes the burden of knowing how to
|
|
|
|
download from a variety of sources from the implementer.
|
|
|
|
|
|
|
|
The concept of a _detector_ automatically turns invalid URLs into proper
|
|
|
|
URLs. For example: "github.com/hashicorp/go-getter" would turn into a
|
|
|
|
Git URL. Or "./foo" would turn into a file URL. These are extensible.
|
|
|
|
|
|
|
|
This library is used by [Terraform](https://terraform.io) for
|
|
|
|
downloading modules, [Otto](https://ottoproject.io) for dependencies and
|
|
|
|
Appfile imports, and [Nomad](https://nomadproject.io) for downloading
|
|
|
|
binaries.
|
|
|
|
|
|
|
|
## Installation and Usage
|
|
|
|
|
|
|
|
Package documentation can be found on
|
|
|
|
[GoDoc](http://godoc.org/github.com/hashicorp/go-getter).
|
|
|
|
|
|
|
|
Installation can be done with a normal `go get`:
|
|
|
|
|
|
|
|
```
|
|
|
|
$ go get github.com/hashicorp/go-getter
|
|
|
|
```
|
|
|
|
|
2016-03-20 23:01:11 +01:00
|
|
|
go-getter also has a command you can use to test URL strings:
|
|
|
|
|
|
|
|
```
|
|
|
|
$ go install github.com/hashicorp/go-getter/cmd/go-getter
|
|
|
|
...
|
|
|
|
|
|
|
|
$ go-getter github.com/foo/bar ./foo
|
|
|
|
...
|
|
|
|
```
|
|
|
|
|
|
|
|
The command is useful for verifying URL structures.
|
|
|
|
|
2016-01-29 20:53:56 +01:00
|
|
|
## URL Format
|
|
|
|
|
2016-03-20 23:01:11 +01:00
|
|
|
go-getter uses a single string URL as input to download from a variety of
|
2016-01-29 20:53:56 +01:00
|
|
|
protocols. go-getter has various "tricks" with this URL to do certain things.
|
|
|
|
This section documents the URL format.
|
|
|
|
|
|
|
|
### Supported Protocols and Detectors
|
|
|
|
|
|
|
|
**Protocols** are used to download files/directories using a specific
|
|
|
|
mechanism. Example protocols are Git and HTTP.
|
|
|
|
|
|
|
|
**Detectors** are used to transform a valid or invalid URL into another
|
|
|
|
URL if it matches a certain pattern. Example: "github.com/user/repo" is
|
|
|
|
automatically transformed into a fully valid Git URL. This allows go-getter
|
|
|
|
to be very user friendly.
|
|
|
|
|
|
|
|
go-getter out of the box supports the following protocols. Additional protocols
|
|
|
|
can be augmented at runtime by implementing the `Getter` interface.
|
|
|
|
|
|
|
|
* Local files
|
|
|
|
* Git
|
|
|
|
* Mercurial
|
|
|
|
* HTTP
|
|
|
|
* Amazon S3
|
|
|
|
|
|
|
|
In addition to the above protocols, go-getter has what are called "detectors."
|
|
|
|
These take a URL and attempt to automatically choose the best protocol for
|
|
|
|
it, which might involve even changing the protocol. The following detection
|
|
|
|
is built-in by default:
|
|
|
|
|
|
|
|
* File paths such as "./foo" are automatically changed to absolute
|
|
|
|
file URLs.
|
|
|
|
* GitHub URLs, such as "github.com/mitchellh/vagrant" are automatically
|
|
|
|
changed to Git protocol over HTTP.
|
|
|
|
* BitBucket URLs, such as "bitbucket.org/mitchellh/vagrant" are automatically
|
|
|
|
changed to a Git or mercurial protocol using the BitBucket API.
|
|
|
|
|
|
|
|
### Forced Protocol
|
|
|
|
|
|
|
|
In some cases, the protocol to use is ambiguous depending on the source
|
|
|
|
URL. For example, "http://github.com/mitchellh/vagrant.git" could reference
|
|
|
|
an HTTP URL or a Git URL. Forced protocol syntax is used to disambiguate this
|
|
|
|
URL.
|
|
|
|
|
|
|
|
Forced protocol can be done by prefixing the URL with the protocol followed
|
|
|
|
by double colons. For example: `git::http://github.com/mitchellh/vagrant.git`
|
|
|
|
would download the given HTTP URL using the Git protocol.
|
|
|
|
|
|
|
|
Forced protocols will also override any detectors.
|
|
|
|
|
|
|
|
In the absense of a forced protocol, detectors may be run on the URL, transforming
|
|
|
|
the protocol anyways. The above example would've used the Git protocol either
|
|
|
|
way since the Git detector would've detected it was a GitHub URL.
|
|
|
|
|
|
|
|
### Checksumming
|
|
|
|
|
|
|
|
For file downloads of any protocol, go-getter can automatically verify
|
|
|
|
a checksum for you. Note that checksumming only works for downloading files,
|
|
|
|
not directories, but checksumming will work for any protocol.
|
|
|
|
|
|
|
|
To checksum a file, append a `checksum` query parameter to the URL.
|
|
|
|
The paramter value should be in the format of `type:value`, where
|
|
|
|
type is "md5", "sha1", "sha256", or "sha512". The "value" should be
|
|
|
|
the actual checksum value. go-getter will parse out this query parameter
|
|
|
|
automatically and use it to verify the checksum. An example URL
|
|
|
|
is shown below:
|
|
|
|
|
|
|
|
```
|
|
|
|
./foo.txt?checksum=md5:b7d96c89d09d9e204f5fedc4d5d55b21
|
|
|
|
```
|
|
|
|
|
|
|
|
The checksum query parameter is never sent to the backend protocol
|
|
|
|
implementation. It is used at a higher level by go-getter itself.
|
|
|
|
|
|
|
|
### Unarchiving
|
|
|
|
|
|
|
|
go-getter will automatically unarchive files into a file or directory
|
|
|
|
based on the extension of the file being requested (over any protocol).
|
|
|
|
This works for both file and directory downloads.
|
|
|
|
|
|
|
|
go-getter looks for an `archive` query parameter to specify the format of
|
|
|
|
the archive. If this isn't specified, go-getter will use the extension of
|
|
|
|
the path to see if it appears archived. Unarchiving can be explicitly
|
|
|
|
disabled by setting the `archive` query parameter to `false`.
|
|
|
|
|
|
|
|
The following archive formats are supported:
|
|
|
|
|
|
|
|
* `tar.gz` and `tgz`
|
|
|
|
* `tar.bz2` and `tbz2`
|
|
|
|
* `zip`
|
|
|
|
* `gz`
|
|
|
|
* `bz2`
|
|
|
|
|
|
|
|
For example, an example URL is shown below:
|
|
|
|
|
|
|
|
```
|
|
|
|
./foo.zip
|
|
|
|
```
|
|
|
|
|
|
|
|
This will automatically be inferred to be a ZIP file and will be extracted.
|
|
|
|
You can also be explicit about the archive type:
|
|
|
|
|
|
|
|
```
|
|
|
|
./some/other/path?archive=zip
|
|
|
|
```
|
|
|
|
|
|
|
|
And finally, you can disable archiving completely:
|
|
|
|
|
|
|
|
```
|
|
|
|
./some/path?archive=false
|
|
|
|
```
|
|
|
|
|
|
|
|
You can combine unarchiving with the other features of go-getter such
|
|
|
|
as checksumming. The special `archive` query parameter will be removed
|
|
|
|
from the URL before going to the final protocol downloader.
|