6 releases
0.1.5 | Mar 22, 2023 |
---|---|
0.1.4 | Apr 18, 2021 |
0.1.3 | Mar 25, 2021 |
#78 in #s3
64KB
1.5K
SLoC
Description
Buffer logs to S3, batching them up by size and/or time period. This is
intended to be a replacement for rotatelogs
on (e.g.) Apache HTTPD servers
running in the cloud.
Usage
stream-logs-to-s3 [options] s3://bucket/path-template
Options
-d, --duration #<unit>
Maximum duration to buffer before flushing to S3; defaults to 1h. The duration is any string acceptable to the humantime crate, e.g., "1hour 12min 5s".-t, --tempdir directory
Temporary directory to use for buffering; defaults to$TMPDIR
(if set),/tmp
otherwise.-s, --size #<unit>>
Maximum size to buffer before flushing to S3; defaults to 1MiB. The size is any string acceptable to the byte_unit crate, e.g., "123KiB".-i, --input <filename>
Read input from the specified file (should be a FIFO) instead of stdin; this is usually for testing.-z, --gzip
Compress output using gzip.-h, --help
Show this usage information
Environment variables
stream-logs-to-s3
uses the standard AWS SDK / Rusoto
methods of specifying AWS credentials.
AWS_DEFAULT_REGION
/AWS_REGION
Specify the region to make calls to S3 in. Defaults tous-east-1
.AWS_PROFILE
If specified, read AWS credentials from this section in the~/.aws/credentials
file.AWS_ACCESS_KEY_ID
/AWS_SECRET_ACCESS_KEY
/AWS_SESSION_TOKEN
If specified, the AWS credentials to use.
If credentials are not specified, they are read from the EC2 or ECS metadata endpoint.
Path template
The path template can include the following variables. Timestamps are generated in the UTC timezone.
{host_id}
— The EC2 instance id, ECS task id, hostname, or IP address.{year}
— The current year.{month}
— The current month as a 2-digit string.{day}
— The current day as a 2-digit string.{hour}
— The current hour as a 2-digit string.{minute}
— The current minute as a 2-digit string.{second}
— The current second as a 2-digit string.{unique}
— A unique identifier to ensure filename uniqueness.
To include a raw {
or }
in the output, double it: {{
/ }}
.
License
This program is dual licensed under the MIT and Apache-2.0 licenses.
Copyright © 2021 Ionosphere, LLC.
Dependencies
~21–37MB
~586K SLoC