One of the additional security mechanisms that can be applied to the production EC2 instances is not to avoid any manual or applications errors that may accidentally stop the EC2 instances.
This post aims to show how to restart an EC2 instance as soon as it goes to Stopped state by using Lambda functions, Event Bridge, and IAM roles.
Launch an EC2 Instance
To restart the instance, Lambda will require an execution role with allowed permissions. Following permissions code will allow the Lambda function to log permissions into cloudwatch logs, and allows permissions to start and stop EC2 instances.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"ec2:Start*",
"ec2:Stop*"
],
"Resource": "*"
}
]
}
Setup Lambda Execution Role
Go to IAM management console > Create role > AWS Service > Lambda > Next Permissions
Next, Create a policy > JSON > enter above policy code. > Next Tags > Next Review
Name: Lambda_EC2_restart_Policy ( or as per your organization standards) > Create Policy
Go back to IAM Role Creation Tab > Refresh the policy list > Search > Lambda_EC2_restart_Policy > Select > Next Tags >Next Review
Role Name : Lambda_EC2_Start_Stop_Role
This role can be attached to the Lambda function and will have permission to start and stop a EC2 instance.
Create Lambda Function to restart EC2 instance
Go to Lambda Management Console > Create Function > Author from scratch
Function Name> ProductionEC2Restart (or as per organization documentation standards)
Runtime> Python 3.9
Architecture> x86_64
Change default execution role > Use an existing role > Lambda_EC2_Start_Stop_Role (one created earlier) > Create Function
On the Code Section paste the following python code > Deploy
# Python code to restart EC2 instances
# Receive the stop state event of the stopped instances in json dump
# Select all the instanceids
# Start the instances
import boto3
import os
import json
region = 'us-east-1'
ec2 = boto3.client('ec2', region_name=region)
def lambda_handler(event, context):
print("Received event: " + json.dumps(event))
instances=[ event['detail']['instance-id'] ]
ec2.start_instances(InstanceIds=instances)
print ('Production mission critical instance is stopped - starting up instance: '+str(instances))
Create a event rule in Event Bridge
Go to Amazon Event Bridge management console > Create Rule
Name: Production-EC2-Monitor ( as per your organization standards)
Select Event Pattern > Pre-defined pattern by service > Service Provider > AWS > Service Name > EC2 > Event type > EC2 Instance State-change Notification > Specific states > stopped > Specific Instance > Copy instance ID of the instance and paste it here.
Select Target > Target > Lambda Function > ProductionEC2Restart (the function created earlier) > Create
Event Bridge event is monitoring the instance and if the instance is stopped, it will trigger lambda function to restart the instance.
Test the implementation by stopping the instance manually and monitor the logs in Cloudwatch.
CloudWatch to monitor Lambda functions logs
Go to the CloudWatch management console > Cloudwatch > Log groups > ProductionEC2Restart (lambda function name)
The CloudWatch log group will be the same as the name of the Lambda function which will contain all the logs generated by the lambda function.
Cloudwatch logs show, the instance state change event notification that the instance stopped was received and the Lambda function restarted the instance. The instance is then successfully restarted.
This is one of the security features that can be implemented to protect the EC2 instances from accidental events.
Remember to clean the AWS account to avoid any charges. (Delete Event Bridge rule, delete lambda function, delete IAM role and policy and terminate the EC2 instance)