This document assumes you've already set up an Amazon Web Services (AWS) account, created a master key in the Key Management Service (KMS), and have done the basic work to set up the MariaDB AWS KMS plugin. These steps are all described in Amazon Web Services (AWS) Key Management Service (KMS) Encryption Plugin Setup Guide.
Ultimately, keeping all the credentials required to read the key on a single host means that a user who has gained access to the host has enough information to read the encrypted files in the datadir, read the encrypted keys from the datadir, interact with AWS KMS to decrypt the encrypted keys, and then used the decrypted keys to decrypt the encrypted data.
Theoretically, a superuser can read the memory of the MariaDB server process to read the decrypted keys or restart MariaDB with password authentication disabled in order to dump data, or add new users to MariaDB in order to allow a user to connect and dump the data. Resolving these issues is beyond the scope of this document. A user who gains root access to your operating system or root access to your MariaDB server will have the ability to decrypt your data. Plan accordingly.
Putting the AWS credentials in a file inside the MariaDB home directory is not ideal. By default, any user with the FILE privilege can read any files that the MariaDB server has permission to read, which would include the credentials file. To protect against this, you should set secure_file_priv to restrict the location the server will allow a user to read from when executing LOAD DATA INFILE or the LOAD_FILE() function.
But putting them in other locations requires passing additional data to the server, which in the case of CentOS 7 requires customizing the systemd startup procedure. This is most easily done by creating a "drop-in" file in /etc/systemd/system/mariadb.service.d/. The file should end in ".conf" but can otherwise be named whatever you like. After making any changes to systemd files, execute systemctl daemon-reload and then start (or restart) the service as usual.
You can place the credentials file in a location of your choosing and then refer to that file by setting the AWS_CREDENTIAL_PROFILES_FILE environment variable in the drop-in file:
[Service] Environment=AWS_CREDENTIAL_PROFILES_FILE=/etc/aws-kms-credentials
The credentials file will need to be readable by the "mysql" user, but it does not need to be readable by any other user.
AWS credentials can also be put directly into a "drop-in" systemd file that will be read when starting the MariaDB service:
# cat /etc/systemd/system/mariadb.service.d/aws-kms.conf [Service] Environment=AWS_ACCESS_KEY_ID=AKIAIRSG2XYZATCJLZ4A Environment=AWS_SECRET_ACCESS_KEY=ux91LZIxCp4ZXabcdefgIViQNtTan42QAmJqJVqV
However, any OS user can read this information from systemd, which could be considered a security risk. Another solution is to put the credentials in a separate file that is only readable by root and then refer to that file using an EnvironmentFile directive in a drop-in systemd file.
# cat /etc/systemd/system/mariadb.service.d/aws-kms.env AWS_ACCESS_KEY_ID=AKIAIRSG2XYZATCJLZ4A AWS_SECRET_ACCESS_KEY=ux91LZIxCp4ZXabcdefgIViQNtTan42QAmJqJVqV # chown root /etc/systemd/system/mariadb.service.d/aws-kms.env # chmod 600 /etc/systemd/system/mariadb.service.d/aws-kms.env # cat /etc/systemd/system/mariadb.service.d/aws-kms.conf [Service] EnvironmentFile=/etc/systemd/system/mariadb.service.d/aws-kms.env
That has the advantage the the credentials can only be read directly by root. systemd adds those variables to the environment of the MariaDB server when starting it, and MariaDB can use the credentials to interact with AWS. Note, though, that any process running as the "mysql" user can still read the credentials from the proc filesystem on Linux.
$ whoami mysql $ cat /proc/$(pgrep mysqld)/environ | tr '\0' '\n' | grep AWS AWS_ACCESS_KEY_ID=AKIAIRSG2XYZATCJLZ4A AWS_SECRET_ACCESS_KEY=ux91LZIxCp4ZXabcdefgIViQNtTan42QAmJqJVqV
AWS KMS allows flexible, user-editable key policy. This offers fine-grained control over which users can operate on keys. The possibilities range from simply restricting which IP addresses are allowed to perform operations on the key, to requiring a MFA (Multi-Factor Authentication) device to use the key, to enforcing separation of responsibilities by creating an additional user with limited privileges to enable and disable the key. All 3 of these options will be outlined in this section.
For more details about customizing the Key Policy for your master keys, please consult the AWS Key Management Service Key Policy documentation.
A simple, common-sense restriction to put in place is to restrict the range of IP addresses that are allowed to use your master key. This way, even if someone obtains API credentials, they'll be unable to use them to decrypt your encryptions keys from a different host.
To restrict API access from only a specific IP address or range of IP addresses, you'll need to manually edit the key policy.
"Sid": "Allow use of the key". "Sid" line: "Condition": {
"IpAddress": {
"aws:SourceIp": [
"10.1.2.3/32"
]
}
},
... replacing 10.1.2.3/32 above with an IP address or range of IP addresses in CIDR format. For example, a single address would be 192.168.12.34/32, while a range of addresses might be 192.168.0.0/24. Access to the API will now be restricted to requests coming from the IP address or range of IP addresses specified in the policy.
One approach is to modify the key policy for the master key so that MFA (Multi-Factor Authentication) is required in order to use the key. This is achieved with a wrapper that handles prompting the user for an MFA token, acquires temporary, limited-privilege credentials from the AWS Security Token Service (STS), and puts those credentials into the environment of the MariaDB server process. The credentials can expire after as little as 15 minutes.
To require an MFA token for users of the key, you'll need to manually edit the key policy.
"Sid": "Allow use of the key". "Sid" line: "Condition": {
"Bool": {
"aws:MultiFactorAuthPresent": "True"
}
},
Now, add an MFA device for your user. You'll need to have a hardware MFA device or an application such as Google Authenticator installed on your smartphone.
Now, set up the wrapper program.
/etc/systemd/system/mariadb.service.d/:[Service] EnvironmentFile=/etc/systemd/system/mariadb.service.d/aws-kms.env ExecStart= ExecStart=/usr/local/bin/iam-kms-wrapper --config=/etc/my.cnf.d/iam-kms-wrapper.config /usr/sbin/mysqld $MYSQLD_OPTS
systemctl daemon-reload. /etc/my.cnf.d/iam-kms-wrapper.config with these contents, using the ARN for your MFA device as the value for kms_mfa_id:[kms] kms_mfa_id = arn:aws:iam::551888187628:mfa/MDBEnc kms_mfa_socket = /tmp/kms_mfa_socket
When you start the MariaDB service now, the wrapper will temporarily create a socket file at the location given by the kms_mfa_socket option. The wrapper will read the MFA code from the socket and will use it to authenticate to KMS. To give the MFA code, simply write the digits to the socket file using echo: echo 111676 > /tmp/kms_mfa_socket.
The systemctl command will block until MariaDB starts, so you will need to write the code to the socket file via a separate terminal.
Note that the temporary credentials put into the environment of the MariaDB process will expire after a period of time defined by the request to the AWS Security Token Service (STS). In the example below, they will expire after 900 seconds. After that time, MariaDB may be unable to generate new encrypted data keys, which means that, for example, an attempt to create a table with a previously-unused key ID would fail.
Here's an example wrapper program written in go. Build this into an executable named iam-kms-wrapper and use it as instructed above. This could of course be written in any language for which an appropriate AWS SDK exists, but go has the benefit of compiling to a static binary, which means you do not have to worry about interpreter versions or installing complex dependencies on the host that runs your MariaDB server.
package main
import (
"syscall"
"os"
"log"
"flag"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/service/sts"
"github.com/robfig/config"
)
func main() {
config_file_p := flag.String("config", "", "location of the config file")
flag.Parse()
if flag.NArg() < 1 {
log.Fatal("Command to wrap must be given as first command-line argument")
}
cmd := flag.Arg(0)
args := flag.Args()[0:]
conf, err := config.ReadDefault(*config_file_p)
if err != nil {
log.Fatal(err)
}
kms_mfa_id, err := conf.String("kms","kms_mfa_id")
mfa_socket_file, err := conf.String("kms","kms_mfa_socket")
sess := session.New()
svc := sts.New(sess)
syscall.Umask(0044)
log.Printf("Reading MFA token from %s\n",mfa_socket_file)
if err := syscall.Mknod(mfa_socket_file, syscall.S_IFIFO|uint32(os.FileMode(0622)), 0); err != nil {
log.Fatal(err)
}
file, err := os.Open(mfa_socket_file)
if err != nil {
log.Fatal(err)
}
token := make([]byte, 6)
if _, err := file.Read(token); err != nil {
log.Fatal(err)
}
file.Close()
if err := syscall.Unlink(mfa_socket_file); err != nil {
log.Fatal(err)
}
mfa_token := string(token)
token_params := &sts.GetSessionTokenInput{
DurationSeconds: aws.Int64(900),
SerialNumber: aws.String(kms_mfa_id),
TokenCode: aws.String(mfa_token),
}
resp, err := svc.GetSessionToken(token_params)
if err != nil {
if awsErr, ok := err.(awserr.Error); ok {
// Prints out full error message, including original error if there was one.
log.Fatal("Error:", awsErr.Error())
} else {
log.Fatal("Error:", err.Error())
}
}
creds := resp.Credentials
os.Setenv("AWS_ACCESS_KEY_ID",*creds.AccessKeyId)
os.Setenv("AWS_SECRET_ACCESS_KEY",*creds.SecretAccessKey)
os.Setenv("AWS_SESSION_TOKEN",*creds.SessionToken)
execErr := syscall.Exec(cmd, args, os.Environ())
if execErr != nil {
panic(execErr)
}
}
Another possibility is to use the API to disable access to the master key and enable it only when a trusted administrator knows that the MariaDB service needs to be started. A specialized tool on a separate host could be used to enable the key for a very short period of time while the service starts and then quickly disable the key.
To do this, you can create an extra IAM User that can only use the kms:EnableKey and kms:DisableKey API endpoints for your key. This user will not be able to encrypt or decrypt any data using the key.
First, create a new user.
credentials file with this structure:[MDBEncAdmin] aws_access_key_id=AKIAJMPPNO7EBKABCDEF aws_secret_access_key=pVdGwbuK5/jG64aBK1oEJOXRlkdM0aAylgabCDef
Now, give the new user permission to perform API operations on your key.
Statement array with this structure: {
"Sid": "Allow Enable and Disable of the key",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::551888181234:user/MDBEncAdmin"
},
"Action": [
"kms:EnableKey",
"kms:DisableKey"
]
},
...so that your Key Policy looks like this:{
"Version": "2012-10-17",
"Id": "key-consolepolicy-2",
"Statement": [
{
"Sid": "Allow Enable and Disable of the key",
"Effect": "Allow",
"Principal": {
...
You've now added a new IAM user and you've given that user privileges to enable and disable your key. This user will be able to perform those operations using the AWS CLI or via a script of your own design using the AWS API. For example, using the AWS CLI:
$ cat ~/.aws/credentials [MDBEncAdmin] aws_access_key_id=AKIAJMPPNO7EBKABCDEF aws_secret_access_key=pVdGwbuK5/jG64aBK1oEJOXRlkdM0aAylgabCDef $ AWS_PROFILE=MDBEncAdmin aws --region us-east-1 kms disable-key --key-id arn:aws:kms:us-east-1:551888181234:key/abcdf8f6-084b-4cff-99ca-abcdef6c7907c
In order for MariaDB to start, this new user will have to enable the master key, then the DBA can start MariaDB, and this user can once again disable the master key after the service has started. Note that in this workflow, MariaDB will be unable to create new encryption keys, such as would be done when a user creates a table that refers to a non-existent key ID. The AWS KMS plugin will encounter an error if it tries to generate a new encryption key while the master key is disabled. In that scenario, the key administrator would have to enable the key before the operation could succeed. Here's what you should expect to see in the journal if MariaDB tries to interact with a disabled master key:
[ERROR] AWS KMS plugin : GenerateDataKeyWithoutPlaintext failed : DisabledException - Unable to parse ExceptionName: DisabledException Message: arn:aws:kms:us-east-1:551888181234:key/abcdf8f6-084b-4cff-99ca-abcdef6c7907c is disabled.
It's also possible to add MFA to this technique so that the user that enables & disables the master key has to authenticate using an MFA device. Adapt the instructions in the MFA section above to add MFA to the policy section for the user with EnableKey and DisableKeys privileges, add an MFA device for that user, use Security Token Service (STS) to get temporary security credentials, and then use those credentials to make the API calls. Here's an example Python script that follows that workflow:
#!/usr/bin/env python
import boto3
import sys
# Command-line argument processing should be more robust than this...
action= sys.argv[1]
mfa_token= sys.argv[2]
# These should perhaps go into a config file instead of here
mfa_serial= 'arn:aws:iam::551888181234:mfa/MDBEncAdmin'
key_id= 'arn:aws:kms:us-east-1:551888181234:key/abcdf8f6-084b-4cff-99ca-abcdef6c7907c'
# Make the connection to the Security Token Service to get temporary credentials
token_client= boto3.client('sts')
token_response= token_client.get_session_token(
DurationSeconds= 900,
SerialNumber= mfa_serial,
TokenCode= mfa_token
)
cred= token_response['Credentials']
# Start new session using temporary, MFA-authenticated credentials
kms_session= boto3.session.Session(
aws_access_key_id= cred['AccessKeyId'],
aws_secret_access_key= cred['SecretAccessKey'],
aws_session_token= cred['SessionToken'],
region_name= key_id.split(':')[3]
)
# Start KMS client and execute operation against key
kms_client= kms_session.client('kms')
if action == 'enable' or action == 'e':
action_f= kms_client.enable_key
elif action == 'disable' or action == 'd':
action_f= kms_client.disable_key
else:
raise Exception('Action must be either "disable" or "enable"')
action_f(KeyId=key_id)
$ AWS_PROFILE=MDBEncAdmin python kms-manage-key disable 575290 $ AWS_PROFILE=MDBEncAdmin python kms-manage-key enable 799870
Amazon's CloudTrail service creates JSON-formatted text log files for every API interaction. Enabling CloudTrail requires S3, which incurs additional fees.
First, enable CloudTrail and add a trail.
If you navigate to the S3 bucket you created, you should find log files that contain JSON-formatted descriptions of your API interactions.
Amazon's CloudWatch service allows you to create alarms and event rules that monitor log information.
First, send your CloudTrail logs to CloudWatch.
Now, set up an SNS topic to facilitate email notifications.
Now, configure CloudWatch and create an Event Rule.
You should now get emails any time someone executes API calls on the KMS service in the region where you've created the CloudWatch Event rule. That means you should get email any time the key is enabled or disabled, and any time the AWS KMS plugin generates new keys or decrypts the keys stored on disk on the MariaDB server.
You may also wish to create an event rule (or an additional event) that matches only when an unauthorized user tries to access the key. You might accomplish that by manually editing the Event selector of the rule to look something like this:
{
"detail-type": [
"AWS API Call via CloudTrail"
],
"detail": {
"eventSource": [
"kms.amazonaws.com"
],
"errorCode": [
"AccessDenied",
"UnauthorizedOperation"
]
}
}
The emails are formatted as JSON. Further customization of the CloudWatch email workflow is beyond the scope of this document.
There are many other workflows available using CloudWatch, including workflows with alarms and dashboards. Those are beyond the scope of this document.
© 2019 MariaDB
Licensed under the Creative Commons Attribution 3.0 Unported License and the GNU Free Documentation License.
https://mariadb.com/kb/en/aws-key-management-encryption-plugin-advanced-usage/