terraform/lang/funcs
Martin Atkins 70fed23be5 lang/funcs: File hashing functions stream data from disk
Previously our file hashing functions were backed by the same "read file
into memory" function we use for situations like "file" and "templatefile",
meaning that they'd read the entire file into memory first and then
calculate the hash from that buffer.

All of the hash implementations we use here can calculate hashes from a
sequence of smaller buffer writes though, so there's no actual need for
us to create a file-sized temporary buffer here.

This, then, is a small refactoring of our underlying function into two
parts, where one is responsible for deciding the actual filename to load
opening it, and the other is responsible for buffering the file into
memory. Our hashing functions can then use only the first function and
skip the second.

This then allows us to use io.Copy to stream from the file into the
hashing function in smaller chunks, possibly of a size chosen by the hash
function if it happens to implement io.ReaderFrom.

The new implementation is functionally equivalent to the old but should
use less temporary memory if the user passes a large file to one of the
hashing functions.
2021-05-12 09:28:31 -07:00
..
testdata lang/funcs: templatefile function 2018-12-21 08:06:14 -08:00
cidr.go add test cases and remove no-longer-needed validation 2020-07-08 13:53:54 -04:00
cidr_test.go add test cases and remove no-longer-needed validation 2020-07-08 13:53:54 -04:00
collection.go functions: Improve marks support for lookup 2021-05-07 12:55:40 -04:00
collection_test.go functions: Improve marks support for lookup 2021-05-07 12:55:40 -04:00
conversion.go lang/funcs: add (console-only) TypeFunction (#28501) 2021-04-23 10:29:50 -04:00
conversion_test.go lang/funcs: add (console-only) TypeFunction (#28501) 2021-04-23 10:29:50 -04:00
crypto.go lang/funcs: File hashing functions stream data from disk 2021-05-12 09:28:31 -07:00
crypto_test.go lang/funcs: Add support for OpenSSH RSA key format 2020-06-03 10:50:38 -04:00
datetime.go cleanup comments for nicer godocs 2018-10-16 18:49:20 -07:00
datetime_test.go cleanup comments for nicer godocs 2018-10-16 18:49:20 -07:00
defaults.go lang/funcs: Defaults handling of marked arguments 2021-04-21 16:09:09 -04:00
defaults_test.go lang/funcs: Defaults handling of marked arguments 2021-04-21 16:09:09 -04:00
encoding.go lang/funcs: Rename the base64 character encoding functions 2020-10-21 10:56:56 -07:00
encoding_test.go lang/funcs: Rename the base64 character encoding functions 2020-10-21 10:56:56 -07:00
filesystem.go lang/funcs: File hashing functions stream data from disk 2021-05-12 09:28:31 -07:00
filesystem_test.go lang/funcs: Filesystem functions hint about dynamically-generated files 2020-06-24 09:02:38 -07:00
number.go remove old funcs code 2020-03-03 15:23:58 -05:00
number_test.go remove old funcs code 2020-03-03 15:23:58 -05:00
sensitive.go lang/funcs: Make nonsensitive more permissive 2021-04-12 12:31:59 -04:00
sensitive_test.go lang/funcs: Make nonsensitive more permissive 2021-04-12 12:31:59 -04:00
string.go remove old funcs code 2020-03-03 15:23:58 -05:00
string_test.go remove old funcs code 2020-03-03 15:23:58 -05:00