How Can We Help?
AWS – CloudFormation
CloudFormation is a crucial exam topic.
CloudFormation is a way of automating the creation and modification of infrastructure in AWS. Infrastructure as code – is declarative code.
Makes it easy to re-deploy an environment configuration either in same az, region or other az or regions, and at other times.
Provides for separation of concern:
vpc stacks
app stacks
network stacks
There are existing CF templates available to use
CF templates can be manually created using CloudFormation Designer
you can also use the Console to input parameters
or automatically using YAML files and CLI commands to deploy.
A stack file always refers to a version of the CloudFormation template syntax. The last version is from 2010.
The CF building blocks are:
Template components:
Resources – these are MANDATORY – the AWS resources declared in your template
Parameters – the dynamic inputs for your template
Mappings – static variables for your template
Outputs – references to what has been created
Conditionals – List of conditions to perform resource creation
Metadata
Template helpers include references and functions
The exam does NOT require you to actually write CloudFormation templates – but it expects you to know how to read templates.
YAML and JSON can be used for CF – but JSON is not pleasant – YAML is much easier and appears in the exam. YAML uses key-value pairs.
Resources
are the core of the CF Template – they are mandatory. AWS provides over 224 CF resources
They represent the components that can be created by CF. They are declared and can also reference each other
Resource types identifiers are of the form:
AWS::aws-product-name::data-type-name
Where to find them: there is a link in docs.aws.amazon.com
listed under
AWS Resource Types Reference
Almost every AWS service is represented in the CF Resources List. for those which aren’t, you can use AWS Lambda Custom Resources.
Resource definitions in the YAML form always contain a Type field and a Properties field
The contents of the Properties field will be specific to the actual resource they define.
You cannot create resources “dynamically”, everything in the CF template has to be declared, it is not possible to perform code generation in the template.
CloudFormation automatically takes care of resolving dependencies between resources defined in the stack.
This means that if you define a subnet and a VPC, for example, CloudFormation creates the VPC before the subnet, since a subnet always refers to a specific VPC. And when you delete a stack, CF will automatically delete the subnet before it deletes the VPC.
Parameters
They provide a powerful way for you to enter inputs into your CF template.
Esp useful if you want to re-use your templates in other departments or functions of your company
and where some inputs can not be determined precisely ahead of time
and help to prevent errors occurring due to type setting in the template.
So you should use a parameter definition when you think the resource config value/s are likely to change in the future.
Types include string, number, comma-delimited list
there are min or max lengths, constraints, allowed patterns/values
how to reference… you use a function called Ref
Fn::Ref
you use it in the form !Ref <paramenter>
eg
VpcId: !Ref MyVPC
They can be deployed anywhere in a template.
The !Ref is a YAML convention
and it can reference other elements in the template.
Pseudo Parameters
these can be used any time and are enabled by default
Mappings
are fixed variables that are hard-coded in the template.
They are often used to differentiate between different regions, AZs, environments, AMIs, etc.
Use when you are sure they will not change. They allow for safer templates. But if the values are more user specific, use parameters instead.
Fn::FindInMap
this returns a named value from a specific key…
this is used to access mapping values
!FindInMap [ MapName, TopLevelKey, SecondLevelKey ]
this is a way to read the value of the mapping.
Outputs
these are optional output values that can be imported into other stacks, provided you export them first.
You can view the outputs via Console or CLI.
Cross Stack Reference – know for the exam!
It is a way to do cross-stack collaboration to read variables from one stack into another.
in the other second template you use Fn::ImportValue function
eg
!ImportValue SSHSecurityGroup (or whatever)
Important; note that you cannot delete a CF stack if it has outputs referenced by another stack – you must remove those first (if possible).
Conditions
they are used to control creation of resources or outputs according to a set condition, usually if
Environment (dev /test /prod )
or any other parameter value…
then create or don’t create a resource
each condition can also reference another condition, parameter or mapping
How to define a condition?
Conditions:
CreateProdResources: !Equals [ !ref EnvType, prod ]
that means if the env type is equal to prod, then do something
the logical ID you choose yourself.
the logical functions can be…
Fn::
And
Equals
If
Not
Or
How to use a condition:
apply it to resources, outputs etc
a resource may only be created if a condition applies
Intrinsic Functions – you must know these for the exam!
Ref
Fn::GetAtt
Fn::FindInMap
Fn::ImportValule
Fn::Join
Fn::Sub
plus the condition functions:
ie
Fn:If, Fn::Not, Fn::Equals etc etc
Fn::Ref
this will reference a given parameter, resource,
remember – the shorthand in YAML for this is !Ref <item>
Fn::GetAtt
attributes are attached to any resource you create, to find out the attributes relating specifically to each resource, check the documentation.
You can get use the GetAtt to access whichever attribute you want to reference
eg to find out the aZ of an EC2 instance:
AvailabilityZone:
!GetAtt EC2Instance.Availability zone
Fn::FindInMap
used to access the mapping values
see above – already covered
Fn::ImportValue
to import values that are exported in other templates
Fn::Join
you can use this to join values with a delimiter eg to create a list a,b,c
or A:B:C
or whatever
Fn::Sub
or !Sub
– substitution of variables with a string
Finally, condition functions
we saw those above..
!GetAtt
How to add your user data to CloudFormation
A user data script can also be passed to CF. This is done via the function:
Fn::Base64 | <your user script contents>
(not the script file, the actual contents copypasted)
important note: you MUST include the pipe symbol with the function when passing the script contents, else the function will not work!
the user data script log is maintained at /var/log/cloud-init-output.log
cfn-init
important:
AWS::CloudFormation::Init MUST be in the metadata of a resource
this makes EC2 configs more readable
the EC2 will query CF to get the init data.
The logs are written to /var/log/cfn-init.log
NOTE DONT CONFUSE cloud-init and cfn-init!
cfn-init vs cloud-init
There are two similarly named tools that it is important not to mix up: cfn-init and cloud-init.
Both of these tools provide a means of customizing EC2 instances, but they are in fact different tools. So what’s the difference between them?
cloud-init: is a Ubuntu project that provides a way of customizing Cloud instances.
cfn-init is an official AWS-supported tool for customizing EC2 instances.
cfn-init
The cfn-init tool is an AWS utility for customizing EC2 instances.It has 2 main documentation pages: AWS::CloudFormation::Init and cfn-init.
cfn-init supports various config management tasks such as creating packages, files, users, groups, commands, and services.
You can find these tasks grouped together under “Configsets” which form part of a CloudFormation template, defined using YAML or JSON.
The cfn-init script is pre-installed on AmazonLinux2 distros.
The cfn-init script obtains the Configset definition from the CloudFormation stack template.
cfn-init is a set of helper scripts that work with the CloudFormation stack.
cfn-init does two things. Firstly, it reads the instructions from CF on how the EC2 instance should be initialized stack and executes it.
Secondly, it informs CloudFormation when completed or if an error occurred. CloudFormation waits until the cfn-init process is completed, and it can roll back in the event of errors.
cfn-init doesn’t require credentials, so you don’t need to use the –access-key, –secret-key, –role, or –credential-file options.
cfn-init uses the user data to initialize and start the init process.
It performs 3 tasks:
1. installs the aws-cfn-bootstrap script
2. runs cfn-init
3. then signals to CloudFormation the result of step 2
This means you don’t have to change the user data when you want to install more services. The arguments for the functions are passed from CloudFormation directly via (–stack ${AWS::StackName}, –region ${AWS::Region}), or else through referencing other items in the template (–resource Instance, which is the resource name, or by using the –configsets setup).
For step 3 to complete, CloudFormation has to expect an initialization signal. this is done via CreationPolicy. Without this, CloudFormation will wait until the instance resource is created and up and running, and only then will mark it as completed.
Configs
The basic building block for cfn.init is a config. This defines (and in this order):
packages – these define which package is to be installed via a package manager (eg apt or yum).
sources – these download compressed files and then extract them to a directory.
files – this creates files with defined content.
commands – this runs arbitrary commands, eg building an app or copying files.
services – these instruct the system service manager – generally systemd – to start a service and to keep it up and running
ConfigSets
ConfigSets are a configuration that defines which configs are to be run and in which order. Each config has to complete running before cfn-init will start the next one.
Ordering
It’s important to know in which order cfn-init runs the various component parts of the configuration, as it does not necessarily follow the order in which the elements are placed in the template file.
First, the configSet defines the order for the configs. Anything else in the list gets run later on.
Secondly, within a config the elements will run according to a fixed ordering defined by AWS.
As an example of this, it will always install packages before running any commands. Generally you should maintain the template order in the same order as when cfn-init will run it.
Finally, there is also ordering inside a config step. In this case note that commands will be run in alphabetical ordering. For this reason, it is advisable to number the commands with 01, 02, 03, 04, nd so on to ensure the correct running order is followed.
/opt/aws/cfn-init and /opt/aws/cfn-signal
Some best practices for using cfn-init:
Comment out the CreationPolicy from the template before you write the cfn-init scripts. This is because otherwise CloudFormation will roll back the whole stack, and with that destroy the instance, if you make a mistake. But be sure to uncomment it when you’re finished.
You can run the cfn-init command if you SSH into the instance too, in order to run a quick deploy test.
Remember when debugging you can check the logs at /var/log/cfn-init.log and /var/log/cfn-init-cmd.log.
Aim to maintain the template in the correct ordering of the elements, keeping the configs listed according to the configSet, and the parts of the config listed according to their own implicit ordering. And be sure to prefix the commands with 01, 02, 03, and so on, so you can follow what is happening in which order.
Refer to the AWS-provided CF template at https://s3.eu-west-1.amazonaws.com/cloudformation-templates-eu-west-1/WordPress_Single_Instance.template.
Define a WAIT CONDITION – this blocks the template until it receives a signal from cfn-signal
Attach a CREATIONPOLICY
Exam question:
what to do if the wait condition hasn’t received the required number of signals back from an EC2 instance?
1. Ensure the AMI image you used has the CF helper scripts installed. (you can also download them to the instance) – check this first!
2. Verify that cfn-init and cfn-signal have run ok on the instance, check the logs at /var/log/cloud-init.log or cfn-init.log
retrieve the logs by logging into the instance, but!
make sure you DISABLE ROLLBACK ON FAILURE else CloudFormation deletes the EC2 after your stack fails to create!
3. Ensure the EC2 has a connection to the internet – via a NAT device if on a private subnet, or via IGW if on a public subnet
Nested Stacks
these are stacks that are repeated patterns or common components for other stacks that you can reuse to avoid having to rewrite code each time.
eg a load balancer config that is often used, or a security group config that is often deployed.
exam question: remember you never alter or update or configure the nested stack, only the PARENT stack.
To update a stack, use ChangeSets
But – exam q: ChangeSets do NOT tell you if the update will work!
To declare a nested stack:
first, in your nestedstacks yaml file, declare the Resource type AWS::CloudFormation::Stack – this is essential to create the stack as a “nested stack” type. You give this resource a name eg
Resources:
myStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL:
https://s3.amazonaws.com/<the file name of the template (can be yaml or json>
When you click on create stack – you have to confirm that you might be creating iam resources with custom names, and that it might require the CAPABILITY_AUTO_EXPAND
– this means CF will expand the nested stack into other stacks if you invoke it.
confirm these two and click on Create stack…
CloudFormation Drift
This allows you to protect your CF infra from manual config changes
CF Drift tracks the changes made manually so you can check them
Retaining Data: Deletion Policy for any resource
eg DeletionPolicy =
= Retain – you specify which resource to preserve or backup
= Snapshot – for EBS, Elasticache, RDS, Redshift
= delete – this is the default – to delete s3 bucket the bucket must first be empty! – remember this for exam!
you can set max concurrent actions and failure tolerance level as well
Termination Protection for CloudFormation
Very Important when using CF:
Termination Protection – this protects your CF stack from being inadvertently deleted
How to control who can change termination protection on stacks.
To enable or disable termination protection on stacks, a user requires permission to perform the cloudformation:UpdateTerminationProtection action.
Here is an example of how to allow users to enable or disable termination protection on stacks:
{
“Version”:”2012-10-17″,
“Statement”:[
{
“Effect”:”Allow”,
“Action”:[“cloudformation:UpdateTerminationProtection”
],
“Resource”:”*”
}]
}
ASG Creation Policy and
ASG Update Policy – need to know these for the exam!
“depends on” – waits for another action to complete first
StackPolicy – enables you to enable/deny certain actions from being made.
StackSets
CloudFormation StackSets are used to create update or delete stacks across multiple accounts and regions via single operation..
Very useful for maintaining CF stacks across multiple regions and accounts.
The admin account must first create the stackset
a trusted account can then create update or delete the stackset
when updating – all associated stack instances are updated across all accounts and regions
You can set a maximum concurrent actions on targets (either #number or a %)
Important: because StackSets operate across multiple accounts, you must first create the necessary IAM permissions before you can use them.
Determine which AWS account is the administrator account – because the Stacksets are created using this account. The target account will then be the account in which you create the individual stacks which belong to the Stackset.
Decide how you want to set the permissions for the Stacksets.. easiest is where you give ALL users and groups permission to create and update all Stacksets. But you may want to restrict this, by specifying specific users and groups, and the resources they can include in their Stacksets and which Stackset operations they are permitted to perform.
Then create the necessary IAM services roles in your admin and target accounts to define these chosen permissions.
There is a CF pre-defined template that can create a set of IAM permissions for this task to make it easier.
CloudFormation will then go ahead and create these permissions in IAM for you when you run it.
How CloudFormation Rollbacks Work
If the stack creation fails:
then the default is for everything to be rolled back, ie deleted. Check the logs.
remember you have the option to disable this rollback and then troubleshoot exactly where it went wrong.
if the stack update fails:
then the stack automatically rolls back to the previous known working stage, you can check in the logs and error messages for what went wrong.
Some CloudFormation CLI Command Examples
Two AWS CLI commands to display information about your CloudFormation stacks:
aws cloudformation list-stacks and
aws cloudformation describe-stacks
Creating a Stack
aws cloudformation create-stack –stack-name example –template-body file://templates/single-instance.yml –parameters file://parameters/single-instance.json
To create the change set, example:
aws cloudformation create-change-set –stack-name example –template-body file://templates/instance-and-route53.yml –parameters file://parameters/instance-and-route53.json –change-set-name changeset-1
Listing the Stack Resources
After running the aws cloudformation create-stack command, you can list the stack’s resources by running the aws cloudformation list-stack-resources command.
This displays a summary of each resource in the stack specified with the –stack-name parameter – including a summary of the stack and the creation or deletion status.
Updating a Stack
for example:
aws cloudformation update-stack –stack-name example –template-body file://templates/instance-and-route53.yml –parameters file://parameters/instance-and-route53.json
Check the status of the stack with the AWS CloudFormation console via the Events tab.
aws cloudformation execute-change-set –stack-name example –change-set-name changeset-1
Alternatively you can execute the Change Set from the CloudFormation console. However, the CLI execute-change-set method shows you the Change Set was applied and so gives you a clear audit trail.
Deleting Stacks
this can be done from the dashboard as well as from the CLI, example:
aws cloudformation delete-stack –stack-name <name of stack>
Managing events with CloudFormation and EventBridge
CloudFormation can send events to EventBridge whenever a create, update, delete, or drift-detection action takes place on your stack.
Unlike for other destinations, you don’t need to select which event types you want to track.
You can use EventBridge rules to route events to your defined targets.