December 15th, 2015
ElasticBeanstalk Tips and Tricks
This page includes a list of tips and tricks for new Elastic Beanstalk users. They are not eb_deployer specific, but mostly has a straightforward implementation in eb_deployer.
- Set autoscale group health check to get self-healing work
- SSL certificates in 3 steps
- Set “inactive_settings” for saving cost on blue green deployment
- Setup Instance Profile For Your EC2 Instances
- How to manage application configuration
Set autoscale group health check to get self-healing work
Autoscale group is awesome not only because it provides elastic scaling but also it is a self-healing mechanism. Bad instances can be detected, terminated and new instances can be provisioned to replace them. But this is not by default enabled for ElasticBeanstalk environments. To enable that you need setup both ELB health check and autoscale group health check.
To enable ELB health check, in option settings section:
- namespace: aws:elasticbeanstalk:application
option_name: Application Healthcheck URL
value: "/"To enable autoscale group health check, in your .ebextensions folder put in a config file:
Resources:
AWSEBAutoScalingGroup:
Type: "AWS::AutoScaling::AutoScalingGroup"
Properties:
HealthCheckType: "ELB"
HealthCheckGracePeriod: "600"The second step (autoscale group health check) is often get skipped, which will result in a weird situation: when ELB marks an instance bad and removes it from load balancing, autoscale group still thinks the instance is good so there is no new instance provisioned to replace it. You can end up with running out instances and whole service goes down.
Read more at: http://www.onegeek.com.au/articles/lessons-learned-configuring-a-ha-multi-docker-container-elastic-beanstalk
SSL certificates in 3 steps
Today, anything you deployed with serious usage should be under https. It is very quick to configure SSL with eb_deployer given you have uploaded the certificates follow AWS guide.
- Use aws cli to figure out certificates ARNS:
% aws iam list-server-certificates
{
"ServerCertificateMetadataList": [
{
"ServerCertificateId": "XXXXXXXXXX",
"ServerCertificateName": "my-production-ssl-certificate",
"Expiration": "2015-12-18T23:59:59Z",
"Path": "/",
"Arn": "arn:aws:iam::xxxxxxxx:server-certificate/my-production-ssl-certificate",
"UploadDate": "2015-06-18T18:37:03Z"
},
{
"ServerCertificateId": "XXXXXXXXX",
"ServerCertificateName": "my-staging-ssl-certificate",
"Expiration": "2016-07-06T12:00:00Z",
"Path": "/",
"Arn": "arn:aws:iam::xxxxxxx:server-certificate/my-staging-ssl-certificate",
"UploadDate": "2015-06-29T18:53:24Z"
},
}]
}
- Configure aws:elb:loadbalancer#SSLCertificateId option for each environment. Note the SSLCertificateId value should actually be certificate ARN in previous commands output
common:
option_settings:
- namespace: aws:elb:loadbalancer
option_name: LoadBalancerHTTPSPort
value: "443"
- namespace: aws:elasticbeanstalk:application
option_name: Application Healthcheck URL
value: "/"
environments:
staging:
option_settings:
- namespace: aws:elb:loadbalancer
option_name: SSLCertificateId
value: "arn:aws:iam::xxxxxxx:server-certificate/my-staging-ssl-certificate"
production:
option_settings:
- namespace: aws:elb:loadbalancer
option_name: SSLCertificateId
value: "arn:aws:iam::xxxxxxx:server-certificate/my-production-ssl-certificate"
- Deploy and give it a try.
Set “inactive_settings” for saving cost on blue green deployment
There are mostly 2 flavors of blue green implementation on AWS: switch routing on ELB (e.g. asguard) or on DNS record (e.g. our eb_deployer). Both has pros and cons. DNS switching approach is safer for on going requests and less intrusive for application’s own infrastructure, but has the con that it is never theoretically safe to terminate inactive environment because of DNS caching.
But isn’t keep a full environment not serving requests a big waste? eb_deployer solves this problem by allow you set special settings to environments that only applied when they become inactive.
Here is an example using “inactive-settings option” to change the autoscale group min size to 1:
option_settings:
- namespace: aws:autoscaling:asg
option_name: MinSize
value: "5"
inactive_settings:
- namespace: aws:autoscaling:asg
option_name: MinSize
value: "1"With this setting an environment will scale down to 1 instance gracefully along with traffic dropping after become inactive.
Similarly if you are confident traffic to inactive will die out in a specific period of time, you can use following settings to gradually kill idle instances to 0 to save cost.
option_settings:
# providing least redundancy
- namespace: aws:autoscaling:asg
option_name: MinSize
value: "2"
# make sure cooldown is reset back to default when environment become active again
- namespace: aws:autoscaling:asg
option_name: Cooldown
value: "360"
inactive_settings:
# reduce instance count to 0 to save cost
- namespace: aws:autoscaling:asg
option_name: MinSize
value: "0"
# make sure cooldown is big enough to cope with DNS cache
- namespace: aws:autoscaling:asg
option_name: Cooldown
value: "900"The above configuration will wait at least 15 minutes (900 seconds) to kill the last instance in inactive, and in most case it is safe. Also the scale down process is off from the deployment process, so the deployment time will not increase even you have long Cooldown buffer.
Setup Instance Profile For Your EC2 Instances
Your application will likely to use other AWS services. It is very easy to make a mistake directly put your AWS keys into the app for making those API calls work. Instead you should setup an IAM instance profile and make your instances bootstrap with it. This way you can exactly define what the machine can do and avoid the risk that someone uses your leaked AWS keys to steal your customer data or farm Bitcoins.
You can use AWS cli tool to setup instance profiles and manage it outside of your Elastic Beanstalk Environments. But with eb_deployer you have a more self-contained way doing this:
Create a CloudFormation template to define your resources stack, e.g. config/my-resources.json:
{
"Outputs": {
"InstanceProfile": {
"Description": "defines what ec2 instance can do with aws resources",
"Value": { "Ref": "InstanceProfile" }
}
},
"Resources": {
"Role": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Statement": [{
"Effect": "Allow",
"Principal": {
"Service": ["ec2.amazonaws.com"]
},
"Action": ["sts:AssumeRole"]
}]
},
"Path": "/",
"Policies": [ {
"PolicyName": "S3Access",
"PolicyDocument": {
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*",
"s3:PutObject"
],
"Resource": "*"
}
]
}
}, {
"PolicyName": "SQSAccess",
"PolicyDocument": {
"Statement": [ {
"Effect": "Allow",
"Action": [
"sqs:ChangeMessageVisibility",
"sqs:DeleteMessage",
"sqs:ReceiveMessage",
"sqs:SendMessage"
],
"Resource": "*"
}]
}
}]
}
},
"InstanceProfile": {
"Type": "AWS::IAM::InstanceProfile",
"Properties": {
"Path": "/",
"Roles": [ { "Ref": "Role" } ]
}
}
}
}Add a “resources” section into your eb_deployer.yml
resources:
template: config/my-resources.json
capabilities:
- CAPABILITY_IAM
outputs:
InstanceProfile:
namespace: aws:autoscaling:launchconfiguration
option_name: IamInstanceProfileIn the above example we defined an instance profile with policies enable specific accesses to S3 and SQS. Then map the instance profile name (output of the template) to Elastic Beanstalk option settings. Now every instance provisioned by Elastic Beanstalk will have the specified access, without needs of our AWS keys.
How to manage application configuration
Environment variables is the most straightforward way to configure simple application on Elastic Beanstalk. What needed is just a list of option settings, such as following:
- namespace: aws:elasticbeanstalk:application:environment
option_name: MY_DATABASE
value: mydatabase.myinterneldomain.comThen your application can read configuration values via environment variables at runtime. (Or via system properties if it is a java application).
But what if the application needs thirty configuration options? In this case downloading configuration files from somewhere else makes a lot of sense.
For example you can store your configuration files into a S3 bucket, and download them to proper places during deployment. Here is a sample config file in .ebextensions doing that:
"/usr/share/tomcat7/lib/my-app.properties":
mode: "000777"
owner: ec2-user
group: ec2-user
source: https://my-app-config.s3.amazonaws.com/my-app.properties
Resources:
AWSEBAutoScalingGroup:
Metadata:
AWS::CloudFormation::Authentication:
S3Access:
type: S3
roleName: aws-elasticbeanstalk-ec2-role
buckets: my-app-configsThe Resources part in previous example is for granting S3 bucket access. You can use IAM instance profile to setup the access if fine-grained control is needed. Take a look at previous tips: Setup Instance Profile For Your EC2 Instances
This approach also applies to other type of data your application need – such as licenses, database seed data.
Thanks @cpilsworth for suggesting this tip. And the tip itself is from this stackexchange answer by @diffa.
Author
