Tags Archives: aws

AWS ElastiCache for Redis/Memcached

ElastiCache is an in-memory DB cache. 

 

ElastiCache supports two open-source in-memory caching engines: Memcached and Redis

 

applications query ElastiCache (EC), called a “cache hit”

 

– if not available it seeks from RDS and then stores in EC

 

this relieves load on RDS

 

cache must have an invalidation strategy set in order to ensure the most relevant data is cached -not so easy in practice

 

is used for managed Redis or 1memcached

 

is an in-memory db with v high performance and low latency,

 

reduces load for read-intensive workloads

 

helps the application operate stateless

 

AWS takes care of maintenance, upgrades, patching, backups etc.

 

BUT you have to make major application code changes to query EC!

 

EC can also be used for DB user session store

 

which avoids users having to reauthenticate and relogin to the DB

 

 

Difference between Redis and Memcached

 

REDIS

 

is similar to RDS:

 

allows for multi-az with auto-failover

 

use read replicas

 

data durability and high availability rather like RDS

 

backup and restore features

 

 

MEMCACHED

 

uses multi-node for partitioning of data, known as sharding

 

NO high availability and no replication

 

it is not a processing cache, so not persistent

 

no backup and no restore

 

has multi-threaded architecture

 

 

pure cache with no high availability or data protection for failure – a simpler option

 

Deployment Options

 

Amazon ElastiCache can use on-demand cache nodes or reserved cache nodes.

 

On-demand nodes provide cache capacity by the hour, resources are assigned when a cache node is provisioned.

 

You can terminate an on-demand node at any time. Billing is monthly for the actual hours used.

 

Reserved nodes use 1-year or 3-year contracts.

 

Hourly cost of reserved nodes is much lower than hourly cost for on-demand nodes.

 

 

 

REDIS Use Case:

 

need to know for exam!

 

Gaming Leaderboards, as these require intensive processing-

 

REDIS uses “sorted-sets”, this ensures accurate real-time data is generated for the leaderboard – important!

 

 

EC Security – important for exam

 

The EC caches do not use IAM policies, except for AWS API-level security

 

REDIS AUTH

 

you use Redis Auth to authenticate – sets a password/token when creating the redis cluster

 

on top of security groups

 

supports inflight ssl

 

 

Memcached

 

uses SASL authentication (more advanced)

 

3 kinds of patterns for EC – need for exam

 

lazy loading – all read data is put in the cache, but can become stale, but future data is only loaded when it cant find the data present in the cache, it reads from db and then copies to cache

write-through – adds or updates cahce data when written to DB – there is no stale data

 

Session Store: stores temp session data in cache using a TTL

 

 

 

DB TCP Ports

 

PostgreSQL: 5432

 

MySQL: 3306

 

Oracle RDS: 1521

 

MSSQL Server: 1433

 

MariaDB: 3306 (same as MySQL)

 

Aurora: 5432 (if PostgreSQL compatible) or 3306 (if MySQL compatible)

 

For the exam make sure to be able to differentiate an “Important Port” vs an “RDS database Port”.

 

Continue Reading

AWS Aurora

AWS Aurora is a high-performance highly available database engine for AWS.

 

Proprietary AWS tech, not open source

 

supports Postgres and MySQL db

 

is AWS cloud optimized and claims 5x performance improvement over MySQL on RDS

 

and 3x improvement on Postgres on RDS

 

Storage grows automatically in blocks of 10GB to 128TB

 

can have up to 15 replicas vs 5 mysql replicas

 

replication also much faster and failover is almost instantaneous,
failover very fast within 30 secs, and it is HA high availability native

 

self-healing with peer-to-peer replication

 

and supports cross-region replication

 

stores 6 copies of your data across 3 AZs

 

4 copies of 6 needed for writes
3 out of 6 needed for reads

 

provides a writer endpoint that points to the master db.

you can have asg auto-scaling on top (but max 15 replicas)

 

 

patching, updating etc is done by AWS

 

also provides BackTrack service – you can restore a point in time without any extra backup routine

 

important for the exam!

 

READER ENDPOINT connects automatically to ALL the read replicas, so this provides for connection level load balancing for the read replicas

 

storage is highly striped across 100s of volumes

 

security similar to RDS

 

encryption at rest using KMS

 

 

automated backups, snapshots and replicas are also encrypted

 

encryption in flight uses SSL

 

you can authenticate using IAM, same as with RDS

 

but you are responsible for protecting your instance with security groups

 

 

also important – for exam!

 

you cannot SSH into Aurora

 

 

 

 

Aurora Read Replica Auto Scaling

 

You create a writer endpoint for DB writes, while reads go to a single separate reader endpoint, which connects to multiple aurora DBs.

 

As auto scaling takes place and more read DBs are added, these are connected to the reader endpoint

 

However, you can also create a separate custom endpoint for specific other traffic purposes, eg a read analytics software which needs to connect and which generates intensive traffic load

 

or you might want to have a set of read replicas which have different instance type to the others, again, you can use custom endpoints for this, this creates an additional endpoint.

 

Aurora Serverless

 

automated db instance and auto-scaling

 

no need for capacity planning

 

use case:

 

infrequent, unpredictable workloads

 

billing is pay per second, can be more cost-effective

 

client talks to a proxy-fleet and in the backend Aurora creates the necessary instances.

 

 

Important for EXAM!

Aurora Multi-Master

 

every node is Read-Write

 

useful for immediate failover for the WRITE node – offers high availability for the writer node.

 

Global Aurora

 

you can have

 

Cross Region Read Replicas

 

useful for Disaster Recovery (DR)

 

easy to set up

 

Global Database (recommended)

 

you have one primary region for read-write

 

and up to 5 secondary read only regions with v low replication lag
plus up to 16 read replicas per secondary region

 

provides for very low latency

 

when you promote another region for DR the RTO recovery time overhead is less than 1 minute.

 

Aurora Machine Learning

 

you can add ML predictions to apps via sql

supports

 

AWS
– sagemaker
– comprehend – for sentiment analysis

 

can be used for fraud detection, ad targeting, sentiment analysis, product recommendations

 

Continue Reading

ASG Auto Scaling Groups

Auto Scaling Groups or ASGs provide a way to scale in and out with your instances and infra.

 

Scale out: add instances for increased workload

 

Scale in: remove instances for decreased workload

 

you can set a minimum, desired, and maximum capacity no of instances

 

automatically connect new instances to a load balancer

 

recreate an instance if a previous one is terminated eg if unhealthy

 

ASG is FREE! – but you pay for the underlying instances

 

ASGs also work with load balancers

 

You create a Launch Template (used to be called Launch Configuration – now deprecated)

 

in this you set:

 

AMI and instance type

 

EC2 user data script, if used
EBS volumes
security groups
ssh key pair
IAM roles for the instances
network/subnet info
lb info

 

plus also set
min,max,initial capacity
scaling policies

 

CloudWatch Alarms also integrate with ASGs.

 

you set a metric to be monitored by CW and this then triggers the ASG activity eg scale in policy or scale out policy.

 

 

 

ASG Termination Policy

by default, ASG looks for the AZ which has the most number of instances, and then deletes the one within that AZ which has the oldest launch configuration.

ASG always seeks to balance number of instances across AZs by default.

 

 

Lifecycle Hooks

 

when an instance is launched you can first determine that it goes first into a pending state, you then move it into pending proceed.. then it goes into in service state.

 

if no pending state, then it goes straight to in-service state.

 

 

also for teminating, you can set a terminating wait state, this is so you have time to carry out some other actions first.

 

 

Launch Configs – these are legacy deprecated, you have to recreate each time.
Launch Templates – new, are versionable, recommended by AWS

 

only use Launch Templates from now on!

 

 

 

 

 

 

Continue Reading

AWS Load Balancers

NOTE: health checks for EC2 instances are crucial when using  load balancers, because you do not want to send traffic to an EC2 instance or other service if it is not working properly.

 

You set up your security group for the load balancer, your endpoints eg EC2 instances should only accept traffic from the load balancer security group and not from the external internet. This is an enhanced security mechanism.

 

 

 

Types of Load Balancer in AWS

 

 

ELB Elastic Load Balancer

CLB Classic Load Balancer (deprecated)

ALB Application Load Balancer

NLB Network Load Balancer

GWLB Gateway Load Balancer

 

 

 

 

ELB Elastic Load Balancer

 

is a managed load balancer,

aws guarantees it will work, takes care of upgrades and availability

costs more than setting up your own load balancer, but is more convenient and less overhead for you

is integrated with many aws services

 

 

 

CLB Classic Load Balancer

 

 

is deprecated, don’t use for new installs

 

operates on tcp layer 4 and http/https layer 7

 

health checks are based on above

 

fixed hostname

 

ALB Application Load Balancer

 

works at layer 7 http

 

balances to multiple http servers machines ie target groups

 

also can be multiple applications on SAME machine eg via containers

 

supports websocket as well

 

and redirects from http to https

 

can route acc to target url path eg example.com/users and example.com/posts

 

also based on hostname eg

 

one.example.com and two.example.com

 

also query string or headers in the url

 

good for micro services and container-based apps eg docker and amazon ecs

 

also have port mapping feature

 

comparison with old classic lb: you would need additional clbs to do the same with one alb if you want different routing

 

 

 

NLB Network Load Balancers

 

operates at layer 4

 

forwards TCP/UDP traffic to instances

 

high volume traffic, millions of requests per sec

low latency 100ms vs 400ms for ALB

 

NLB has one static ip per AZ, supports Elastic IP

 

Useful for having 2 incoming points for traffic to your network

 

use case:

 

when you need extreme performance or tcp udp traffic

 

Note: NLB is NOT in the free-tier pricing!

 

 

GWLB  Gateway Load Balancer 

 

esp used for firewalls, intrusion detection, prevention system (IDS/IDPS), deep packet inspection systems etc

 

can also be used to manage a fleet of 3rd party network virtual appliances running on aws

 

operates at layer 3 network layer ip packets

 

has 2 functions:

 

1. transparent network gateway – a single point of entry/exit for traffic

 

2. load balancer to distribute traffic to your virtual appliances

 

exam tip:
GENEVE protocol port 6081 is the gateway load balancer

 

EC2s must be private addresses for GWLB

 

 

Sticky Sessions or Session Affinity

 

this means the same client is always connected to the same instance behind a load balancer to complete a transaction

 

this works for CLBs and ALBs

 

uses a cookie with an expiry date.

 

this is to ensure a user does not lose his session data

 

but – it can cause an imbalance within the balanced load cluster

 

 

types:

 

application-based cookie

– custom cookie, is generated by the target, can include any attribute

 

– application cookie – generated by load balancer, cookie name is AWSALBAPP

 

but some names are reserved: AWSALB, AWSALBAPP AWSALBTG

 

duration-based cookie

 

– generated by load balancer
cookie name is AWSALB for ALB and AWSELB for CLB

 

 

 

 

Cross-Zone Load Balancing

 

a point to note about cross-zone load balancing…

 

if this feature is  ON, then it will ensure each INSTANCE gets the equal amount of share of traffic as all other instances.

 

but if this feature is OFF , then it will vary between the instances depending on how many instances in each AZ, if this is unequal eg one AZ has fewer EC2s than others, then it will be unequally divided among the actual EC2s although equally shared out at the AZ 1lb level.

 

Be aware:

 

CZ-LB is enabled by default for ALB – and cannot be disable –  but for NLB it is disabled by default – but you pay extra if you want to enable it. 

 

but for CLB: it is disabled, but you can enable, and it is free to enable

 

 

 

SSL/TLS and AWS Load Balancers

 

encrypts via “in-flight” in-transit encryption

 

SSL: secure sockets layer

TLS: transport layer security, the newer ssl version

 

public SSL certificates are issued by certificate authorities (CAs)

 

eg Globalsign, Digicert, GoDaddy etc

 

have an expiry date, must be renewed

 

Load Balancer uses an X.509 SSL certificates, can be managed via ACM – the AWS certificate manager

 

you can create your own certificate

 

clients can also use SNI server name indication – client must declare which hostname it wants in the SSL handshake. Server then finds the correct SSL certificate or else returns the default one.

 

 

 

SNI Server Name Indication for SSL

 

solves problem of loading multiple SSL certificates onto one webserver to serve multiple websites.

 

only works with alb and nlb and cloudfront, not with clb

 

 

Elastic load balancer elb only supports

 

CLB – only 1 SSL certificate

 

must use multiple clbs for more than one certificate

 

ALB and NLB 

 

supports multiple SSL certificates and uses sni to make it work

 

 

Connection Draining and load balancers

 

CLB call it connection draining
ALB and NLB: call it deregistration delay

 

it allows some time for instances to complete in-flight SSL requests while instance is unhealthy or de-registering

 

it stops lb sending requests to the instance during this period

 

you can set a period of between 1 and 3600 secs, default is 300 secs, or disable, by setting to 0.

 

set a low value if requests are short

 

if there tend to be longer requests, eg for uploads, downloads etc… then set a higher value.

 

Continue Reading

AWS Snow Family

AWS Snow Family

 

for data migration in and out of AWS:

 

Snowcone

 

Snowball Edge

 

Snowmobile

 

for Edge computing data migration:

 

Snowcone

Snowball Edge

 

These are offline devices that perform data migration for migration that would take more than a week by network transfer.

 

Snowcone and Snowball Edge are sent by post using a physical route rather than digitally.

 

Snowball Edge, used for TB or PB data transfer

 

pay per data transfer job

 

 

2 versions

 

Snowball Edge Storage Optimized:

 

80TB max of HDD capacity for block volume and S3 compatible object storage

 

Snowball Edge Compute Optimized:

 

42TB of HDD capacity for block volume and S3 compatible object storage

 

 

use case for Edge:

 

large data cloud migration, Disaster recovery prep, disaster recovery action

 

For Snowcone:

 

a much smaller device, can withstand harsh environments, very light and secure

 

used for smaller size transfers, up to 8TB storage

 

10 times less than snowball edge.

 

you must provide own battery and cables

 

can be sent to AWS in post or via internet – and use AWS DataSync to send data.

 

 

Snowmobile is a truck – very high-scale transfers.

 

request snowball devices on the AWS website

 

install a snowball client on servers

 

connect the snowball to your servers, then copy the file,

 

then ship the device back

 

many EBs of data 1EB = 1PB

 

very high security, staffed,

 

best for more than 10PB data transfer

 

Edge Computing:

 

somewhere creating data or needing data but has no internet access or limited access…

 

snowball edge or snowcone can be embedded into your edge site

so it gives you a way to transfer data despite having no strong internet connection

 

The Snowcone and Edge can run EC2 instances and Lambda functions!

good for preprocessing data, transcoding media screens, machine learning at the edge

 

Snowcone 2 CPUs 4GB RAM, wired/wi-fi, powered by usb-c or optional battery

 

Snowcone Edge:

many more vCPUs up to 52vCPUs and 208 GiB RAM
optional GPU
object storage clustering available

 

can get discount pricing 1 or 3 years

 

 

Also AWS OpsHub – management software for snow family

 

This enables you to manage the device from your client pc or laptop, you install on your client

 

 

 

 

Continue Reading

AWS CloudFront

CloudFront is the AWS CDN Content Delivery Service

 

CF offers:

 

ddos protection against webserver attacks
web application firewalls WAF and Shield
improved web content read service through edge region caching (currently 216 edge locations globally, number steadily increasing) ie content caching

 

can use both https for external traffic and also forward https traffic internally – latter not usually possible due to TLS certificate limitations

 

can be used for:

 

s3 buckets CF origin access identity or CF OAI

 

– this is an IAM role used for CloudFront connections to S3.

 

 

two ways to upload content to S3:

 

first,

you can  use CF to upload files to S3 – ie ingress traffic

 

secondly,
other option is to use Custom Origin http

 

can use ALB
can use EC2 Instance
can use S3 website – must enable the bucket as a static web instance
any http backend eg on premises webserver

 

two different architectures are possible acc to whether you have an ALB or not with CF.

 

if EC2 is the origin, (origin means your website content original site) then the security group for the EC2 must allow public access

 

ie the edge location sites to access the EC2 instance

 

if an ALB is attached, then the ALB security group MUST be public – but the EC2 instance can in this case be private access, ie need not give public access. in this case your ALB is your “origin”. 

 

 

these are two different architectures – be sure to understand for the exam!

 

 

 

the edge location always serves cached content originally from your S3 site ie your “origin”.

 

 

so, web users send request to your CDN CF IP at the edge location – if the content is not already present on the CF Edge Location then it forwards the request to your S3 bucket and sends the result back to the edge location of your CF ip… 

 

 

it traverses the security group, so the ip of the ec2 instances ie the origin, must be public and allow public ip of the edge location

 

 

 

CF Geo Restriction

 

You can whitelist or blacklist users from specific countries, eg for copyright etc reasons.

 

What is the difference between CF and S3 Cross Region Replication?

 

CF:

 

a global edge network

caches for a short TTL
good for static content

 

S3 Cross Region Replication:

 

must be set up for each region you want to replicate

 

files updated in near real-time

 

read only

 

good for dynamic content that needs to be available at low-latency in a few regions

 

 

Continue Reading

AWS EBS & EFS Elastic Block Storage & Elastic File System

EBS is a network drive you can attach to ONLY ONE INSTANCE at any one time

 

 

important: EBS is not a network file system!

is bound to an AZ

 

think of it like a network USB stick

 

30GB per month free of type SSD or magnetic GP2 or GP3 volume

 

EBS can have some latency on the AWS network

 

can be detached from one EC2 and attached to another, eg for failovers, big advantage!

 

but cannot be moved to other AZs, unless you do a snapshot and then move.

 

have to provision in advance and the level of IOPS you want

 

you pay according to this after the first 30GB.

 

 

BUT you can have 2 EBS volumes attached to an EC2 – that is not limited.

 

however they are bound to an AZ

 

they can be attached on demand, do not have to be actively attached.

 

 

EBS Delete on Termination attribute – is enabled by default for EBS root volume

 

but not for other EBS volumes as it is disabled for the latter by default

 

you can change this…
advantage: to preserve root volume when EC2 instance is terminated.

 

You can transfer an EBS volume to another region or az by means of using snapshots.

 

You can move snapshots to archive which is much cheaper.

 

Snapshots are usually gone once deleted, but if you have recycle bin enabled then you can retrieve them for a limited time period according to retention rule you set.

 

 

EBS Volume Types

 

6 types:

 

gp2 /gp3 SSD general purpose balances price/performance

 

io1/io2 SSD high performance for mission critical low latency/high throughput workloads

 

st1 HDD low cost HDD for frequently accessed throughput intensive workloads

 

sc1 HDD lowest cost HDD for less frequently accessed workloads

 

EBS volumes are by size, throughput, iops

 

NOTE: only gp2/gp3 and io1/op2 can be used as boot volumes

 

 

General Purpose SSD

 

cost-effective storage

 

system boot vols, virtual desktops, dev and test env

 

1GiB – 16TiB

 

gp3

 

3000 iops and 125 miB/s

 

can go up to 16k iops and 1000 MiB/s

 

gp2

 

small vols can do burst iops to 3000

 

size of volume and iops are linked max iops is 16k

 

3 iops per GB means at 5,334 GB we have max iops

 

 

Provisioned iops ssd

 

for critical biz apps with sustained iops performance

or apps that need more than 16k iops

 

good for db workloads

 

io1/io2 4GiB – 16TiB

 

max piops 64k for nitro ec2 and 32k for others

 

can increase piops indep of storage size

 

io2 gives more durability and more iops per gib

 

io2 block express 4gib – 64tib

 

very low latency
max piops 256k

 

 

with hdd:

 

cannot be a boot vol

125 mib to 16tib

 

throughput optimized hdd st1:

 

big data, data warehousing and log processing

 

max througput is 500 mib/s max iops 500

 

cold hdd sc1

 

for infreq accessed data
where low cost is important
max throughput is 250 mib/s max iops 250

 

 

 

you *dont* need to know these details for the exam, but be aware of the main difference in the variations

 

 

 

EBS Multi-Attach – for io1/io2 family

 

attaches same ebs volume to multiple instances in same az at same time

 

each instance has full r/w permissions to the vol

 

use case:

 

for high app availability in clustered linux apps eg teradata

 

apps must be able to manage concurrent write ops

 

only thus for very specific workloads,

 

you must use a cluster aware file system ie NOT ext4 xfs etc.

 

 

EFS Elastic File System

 

is a managed NFS-type system that can be mounted on multiple EC2s in multiple AZs.

 

highly available, scalable, but expensive, you pay per usage per gigabyte

 

use cases:

 

content management,

 

web and data sharing, wordpress

 

uses nfs4.1

 

uses security groups to control access

 

important!

compatible only with linuxbased ami’s and not windows!

 

can enable enctryption at rest using kms

 

is a posix linux system with standard file api

 

scales automatically by itself!

 

exam question:

 

efs performance and storage classes:

 

efs scaling:

 

can support 1000s of concurrent nfs clients 10gb throughput

 

can grow to petabyle size automatically

 

performance mode

 

– set at efs creation time
general purpose default – latency sensive use cases such as webserver, cms etc

 

to maximum i/o — gives higher latency, highly paralleluse for this io1 : this is best for big data applications, eg media processing etc

 

throughput mode
— bursting 1tb = 50mib/s and up to 100 mib/s

 

provisioned: set your throughput regardless of storage size eg 1 1gib/s per 1tb storage

 

 

EFS Storage Classes

 

you can set up storage tiers for lifecycle management

 

eg move to another tier after N days…

 

 

 

– standard tier – used for frequently accessed files
– infrequent access tier (IA) -efs-ia: costs to retrieve files

 

but lower price to store, enable efs-ia by means of a lifecycle policy

 

 

Availability and Durability of EFS

 

2 options:

 

standard: can set up EFS to be multi AZ
one-zone: you use one AZ only, backups are default enabled, compatible with IA

 

90% cost saving

important:
exam will ask you which tier /storage class you should use for which use case, and you need to be aware of the cost implications!

 

 

 

Differences Between EBS & EFS

 

must know for exam!

 

EBS

 

can only be attached to only one instance at a time
are locked into an AZ

 

gp2. io increases if disk size increase
io1: can increase the io independently.

 

to migrate an EBS across AZs

 

– take a snapshot
– restore the snapshot to the other desired AZ

 

note that ebs backups use up io and so you should not run them when your app has heavy traffic overhead

 

also, root ebs volumes get terminated by default if or when the ec2 instance gets terminated! very important to be aware of this
– but you can disable this

 

 

EFS by comparison:

 

can mount on 100
s of instances across AZs!

 

it is multi-AZ, multi-client/instance

 

can be used to share data

 

only available for linux posix, not windows!

 

 

efs more expensive than EBS but can use efs-ia to reduce costs

 

 

so: efs is more for multi instances

 

ebs is more for one instance

 

 

and

 

Instance Store

 

instance store: is an ephemeral local instance drive just for an instance – you lose it with the instance when the instance is deleted.

 

 

 

 

Continue Reading

AWS Route 53

Route 53 is the AWS DNS service.

 

Highly available
scalable
fully-managed
authoritative DNS – you the customer can update the dns records
is also a domain registrar

 

AWS provides 100% SLA guarantee availability

 

You define how you route traffic to a specific domain.

 

domain name
record type eg A or AAAA
value ie ip number
routing policy
TTl – time to live for the record caching time

 

different record types

 

A, AAA, CNAME, NS – essential to know

 

A – maps hostname to ipv4 address
AAAA – maps hostname to ipv6 address
CNAME – maps hostname to another hostname
you cannot create CNAMES for top record of domain eg example.com but you can for eg www.example.com

 

NS – the name servers for the hosted zone

 

Hosted Zones

 

are a container for dns records

 

 

public hosted zones

for internet available ips. Any client can request

 

private hosted zones

for not publicly available ips, within VPCs which can only be accessed within the subnet
this enables you to make ips resolvable within the private network ie internally, not publicly via internet.

 

 

otherwise they work the same way – public hosted and private hosted.

 

you pay 50c per month per hosted zone

 

 

from your CLI you can then check your domain registration and ip records with
nslookup or dig

 

do apt install bind-utils -y to install them if not installed yet on the machine.

 

dig <your domain name and machine>

 

nslookup <your domain name and machine>

 

 

 

TTL Time To Live

 

 

set in seconds

 

TTL: client will cache a lookup for the set TTL time period, this is to relieve DNS server from too much request and response traffic

 

high TTL:

 

less traffic, lower costs
but possibly not up-to-date records

 

low TTL:

 

more traffic, more costly
records more likely to be fully accurate

 

 

CNAME vs Alias

 

you need to use Alias for mapping to an alias hostname, not CNAME!

 

sometimes you may want to map a hostname to another hostname

 

CNAME does this, but only for non-root domains ie eg www.example.com not example.cmm
The CNAME maps to the root domain name only!

 

Alias: this works for both root and non-root domains
always either A for ipv4 or AAAA for ipv6

 

you cannot set the TTL for Alias, this is set by Route 53 automatically

 

you can use as aliases things like

 

elastic load balancers ELB
CloudFront
API Gateways
Elastic Beanstalk
S3 websites
VPC interface endpoints
Global accelerator
Route 53 record IN THE SAME HOSTED ZONE

 

Important: You *cannot* set an ALIAS for an EC2 DNS name!

 

 

 

Route 53 routing policies

 

simple
weighted
failover
latency-based
geolocation
multi-value
geoproximity

 

you set the routing policy in the Route 53 Dashboard for the dns record

 

 

simple policy

 

you can specify multiple records in the same record but then a random one is chosen by the client

 

can’t be associated with health checks

 

 

weighted policy

 

you set a % of requests to go to each resource you specify

 

eg to different EC2 instances

 

to do this you assign each record a relative weight

 

the weights don’t need to add up to 100

 

but the DNS records involved must have same name and type

CAN be associated with Health Checks

use cases: load balancing between regions, testing new application versions

 

NOTE if you assign a weight of 0 to a record then the resource will not receive any traffic!

 

Also, if ALL records have a weight of 0 then all records will be equal! ie balanced responses

 

 

 

Latency-based

 

you want to redirect to the resource with the least latency, ie closest to us in terms of SPEED of internet

latency based on traffic between users and AWS Regions

 

so depends on traffic speed, not necessarily same as geographical closeness

 

Can use Health Checks

 

 

 

 

Health Checks

 

HTTP Health Checks are only for PUBLIC resources

 

If one region is down, then we can use a Health Check in Route 53.

 

These provide for automated DNS failover

 

the check can be against:

an endpoint eg app server

an other health check eg calculated health checks

 

cloudwatch alarms eg for dynamodb, rds

 

To pass an HTTP health check the endpoint must respond with 2xx or 3xx status codes

 

you can combine up to 256 health checks into a single health check using OR, AND, or NOT 

 

and define how many must pass 

 

How to perform health checks for private hosted zones

use a CloudWatch Metric and Alarm then create a Health Check that monitors the alarm!

 

 

 

 

Failover Policy (Active-Passive Failover)

 

you associate your DNS record with a health check – essential for this

 

but you can only have one primary and one secondary record

 

so, you need to first set up your health check for each machine,

 

then you reference them in your dns records

 

policy: set to failover, and the type: primary or secondary

 

and then associate the respective health check you have defined.

 

then do the same for the secondary record and instance.

 

 

 

Geolocation

 

this is where user is physically based

 

use cases:

 

website localization
restrict content distribution
simple load balancing method

 

Geoproximity

 

enables you to specify “bias values” for specific geo regions

 

1- 99: more traffic to the resource
-1 to -99: less traffic to the resource

 

can be for AWS resources, specifying aws-region or non-AWS resources , specified by latitude/longitude

 

exam tip:

 

this can be useful when you need to shift traffic from one region to another

 

 

 

Multi-Value Policy

 

multi-value or multi-value answer is similar to an ELB but it is a client-side load balancer in effect.

 

used to route traffic to multiple resources but the client chooses which to use

 

can associate with Health Checks – up to 8 checks for each multi-value query

 

NOT a substitute though for an ELB!

 

 

 

Route 63 Traffic Policies

 

You can use these to define your DNS policy.

 

 

These make it easier to set policies.

 

Continue Reading

AWS RDS

 RDS is the AWS Relational Database Service

 

 

a managed DB service for DB, uses SQL, is a “serverless service”, which runs:

 

Postgres
MySQL
MariaDB
Oracle
MS SQL Server
AWS Aurora (AWS proprietary DB)

 

 

You have to know this for the exam!

 

Advantages of RDS vs a db engine on an EC2 instance

 

rds is managed by AWS
automated provisioning, os and db engine upgrades and patching
continuous backups and point-in-time restore
monitoring dashboards
read replicas for improved read performance
multi-AZ for disaster recovery (DR)
maintenance windows for upgrades
scaling capacity, both horizontal and vertical
storage backed by EBS

 

BUT . you can’t SSH into the RDS instances – we don’t have access to the engine or underlying OS. RDS is purely a managed DB solution.

 

RDS Backups

 

automatically enabled
automated
daily full backup
transaction logs backed up every 5 mins
can restore to any point in time
7 days retention, can be increased to 35 days

 

 

DB Snapshots

 

rds db snapshots have to be manually triggered by user

retention for as long as you want

 

 

 

RDS Storage Auto Scaling

 

dynamic increase in storage is automatic – so this avoids manual scaling, but you have to set a Max Storage Threshold ie limit for db storage – exam question!

 

set rds to automatically modify storage if:

 

free storage is less than 10% of allocated storage

 

low storage lasts at least 5 mins

6 hrs have passed since last modification

 

useful for apps with unpredictable workloads

 

supports all rds DB engines – MariaDB, MySQL, PostgreSQL, SQL server and Oracle

 

 

Read Replicas with RDS

 

RDS read replicas provide for faster reads, but not for writes

 

you create a replica of the DB – up to 5 MAX – can be in the same AZ, cross AZ or cross-region – not needed for exam!

 

Asynchronous replication takes place – but there is a time lag involved.

 

hence:  “eventually consistent async replication”

 

exam question:
a replica can be promoted to become a full separate read-write DB.

 

application clients must update connection string to connect to the read replica/s.

 

a very typical exam question use case example:

 

eg prod DB but you need to run intensive read analysis eg reporting…

 

this would slow down the prod DB, so you can create a read replica to run the read process from there.

 

note: only SELECT type statements can be run ie reads, NOT insert, update, delete etc

 

 

Cost:

 

normally a charge for data transfer from one AZ zo another

 

but AWS does not charge for RDS read replica data traffic within the same REGION

 

ie is for free

 

BUT cross-region is chargeable…

 

Using RDS Multi-AZ for Disaster Recovery (DR)

 

This is a common use case for multi-az RDS systems.

 

This uses sync replication – this means that when the application writes to the master db instance, it is also written to the standby DB instance in the other AZ at the same time.

 

one dns name, auto app failover to the standover, this increases availability, meaning there is no manual intervention in apps necessary for the changeover. 

 

Note that it is not used for scaling, just for failover from master db in the event of a failure of the master db.

 

Note that a lot of questions in exam focus on RDS!

 

the read replicates can also be set up as multi-az for DR – this is often an exam question! In this case they switch from async replication to sync replication.

 

 

How to Move RDS from Single-AZ to Multi-AZ

 

There is no need to stop DB – just click on modify for the db

 

how it happens:

 

a snapshot is done automatically from master db and this is then restored from snapshot to a new db in the new az

 

 

a sync is then done. You then have a multi-AZ RDS DR setup.

 

 

 

my rds test database:

 

Endpoint & port
Endpoint: database-1.c2duk8wm0bjt.us-east-1.rds.amazonaws.com
Port  3306

 

 

More on RDS 

 

its a managed db on the postgresql/myswl/Oracle/SQL level

 

you must however an ec2 instance and ebs vol type and sufficient size

 

it supports read replicas and multi-AZ
security is via iam and security groups, kms, and ssl in transit
backup, snapshot and point in time restores all possible

 

managed and scheduled maintanance

 

monitoring available via cloudwatch

 

 

use cases include:

 

storing relational datasets rdbms/oltp performing sql queries, transactional inserts, update, delete is possible

 

rds for solutions architect, considerations include these “5 pillars”:

 

operations
security
reliability
performance
cost

 

 

operations_ small downtimes when failover happens, when maintenance happens, when scaling read replicas, ec2 instances, and restoring from ebs, this requires manual intervention, and when application changes

 

 

security: aws is responsible for os security, but we are responsible for setting up kms, security groups, iam policies, authorizing users in db and using ssl

 

 

reliability: the multi-az feature makes rds v reliable, good for failover in failure situations

 

performance: dependent on ec2 instance type, ebs vol type, can add read replicas, storage autoscaling is possible, and manual scaling of instances is also possible

 

costs: is pay per hour based on provisioned number and type of ec2 instances and ebs usage

 

 

Continue Reading

AWS EC2 Networking Basics

Elastic IPs

 

Important to remember!
When you stop and then start an EC2 instance, it can change its public IP!

 

So if you need a fixed public IP for an instance, you must use an Elastic IP – or else register your own public IP and domain name

 

You own the Elastic IP as long as you don’t delete it, you can attach it to one instance.

 

but you can remap the ip address to another instance in your account in the event of an instance or software failure.

 

But – you can only have max 5 Elastic IPs in your account.

 

Best practice: DON’T use Elastic IP but instead use your own registered public ip with a registered domain name. You can do this via AWS Route 53.

 

 

Even better, you can use a load balancer to connect to the private ip addresses of your instances.

 

Always remember – your EC2-allocated public ip will change if you shutdown and restart the instance!

 

The *private* iP however always remains the same.

 

You allocate an Elastic IP to an EC2 instance from your EC2 Dashboard.

 

 

 

EC2 Placement Groups (PGs)

 

 

these give you control over where your EC2 instances are placed.

 

This gives you a way to specify where you want your EC2s to be physically located.

 

Placement groups can be cluster, spread or partition-based

 

 

cluster:

 

all in same hw rack
in same AZ

 

pro: network speeds high between instances
con: high risk of total failure if the rack fails – because all EC2 instances will fail together

 

use cases: best for big data that must be quickly processed

 

spread:

 

this minimizes risk compared to cluster PG

 

instances across different AZs
and across different hw racks

 

pro: less risk of failure

 

con: you are allowed max 7 instances per AZ per PG permitted

 

use cases: best for high availability, critical applications that must be isolated from failure from each other

 

 

partition:

 

each partition = one rack

 

you can have multiple partitions – 7 maximum in each AZ in same region

 

100s of EC2s per partition possible

 

– failure can affect other EC2s on same rack, but not other partitions

 

can use metadata service to get info about other EC2s on the partition

 

use cases include apache-kafka, hdfs, cassandra

 

 

 

ENI Elastic Network Interfaces

 

This is the AWS virtual network interface

 

it can have

 

one private primary IPv4 address

 

one or more secondary IPv4 addresses

 

one Elastic IP per private IPv4

 

one public IPv4

 

one or more security groups

 

a MAC address

 

you can move ENIs to other instances of EC2 on failover!

 

but note – ENIs are bound to a specific AZ

 

 

 

 

 

 

Continue Reading

AWS Pricing & Billing

EC2 Pricing & Billing Options

 

 

Overview of AWS Pricing Options

 

On-Demand – for short workloads, pay by second

 

Reserved – 1 or 3 years – for long workloads

 

Convertible Reserved – long workloads with flexible instances

 

Savings Plans – 1 or 3 years – commits to a fixed usage, for long workloads

 

Spot – short workloads, can be cancelled at any time by AWS according to general load of other customers, very cheap

 

Dedicated Host – book entire physical machine and control instance placement

 

Dedicated Instance – no other customers share hardware

 

Capacity Reservation – reserve capacity in a specific AZ (Availability Zone) for later use

 

 

Pricing Options in Detail

 

On-Demand

 

has highest cost, no long-term commitment, best for short-term loads

 

Reserved Instances

 

72% max discount vs on-demand, you reserve specific instance attributes

 

specify reservation period eg 1 year

 

payment – upfront or not upfront, discount for upfront

 

scope – in specific region or AZ

 

recommended for stead state loads eg database apps

 

you can buy and sell them in the reserved instance AWS marketplace

 

 

Convertible Reserved Instance

 

allows you to change the EC2 instance spec

 

 

 

Savings Plans

 

discount acc to long term usage as for Reserved up to 72%

 

commit to certain type of usage, eg $10 per hr for 1 or 3 yrs

 

Usage beyond this is billed at on-demand price

 

but you are locked to a specific instance type and region

 

however you can change instance size and OS, and tenancy – host, dedicated default – can change this

 

Spot instances

 

up to 90% discount

 

the most cost-efficient

 

good for resilient to failure loads, eg batch jobs, data analysis, flexible jobs

NOT suited to databases!

 

 

Dedicated Hosts

 

you get a physical server dedicated to your own usage

 

advantageous for compliance requirements and server/hw-bound licenses for software eg per socket/core/CPU etc

 

the most expensive option in AWS

 

can be on demand pay per second

or reserved – 1 or 3 yrs with no-upfront, partial upfront, no upfront

 

 

Dedicated Instances

 

run on hw dedicated to you, but you may share the hw with others in the SAME ACCOUNT

 

but you don’t have access to the actual hardware, unlike dedicated host – important difference

 

no control over instance placement

 

suitable for short-term uninterrupted workloads that need to run in a specific AZ

 

 

AWS Pricing Calculator

 

The AWS Pricing Calculator is a web-based planning tool for creating budget estimates for your AWS projects before building them.

 

AWS Pricing Calculator is available free of charge via a web-based console at https://calculator.aws/#/

 

You do NOT need an AWS account in order to use the Pricing Calculator.

 

It provides an estimate of your AWS fees and charges, but the estimate doesn’t include any applicable local sales taxes.

 

 

 

 

Continue Reading

AWS – Connecting to AWS

AWS Remote Connectivity

 

you have following options to connect to your AWS EC2 instances:

 

SSH to an instance. Default port is 22

 

AWS Instance Connect is a web-based console you can also use

 

 

 

To connect using SSH

 

 

download your .pem access key.

 

remove the space within the filename if there is a space.

 

Then place in a directory on your local client machine and from that directory,

 

get your public ip from your ec2 instance that you want to ssh into.

 

and also check the security of your instance to check that port 22 from anywhere exists as a rule and allows access.

 

then do ssh ec2user@<yourEC2publicIP>

 

you will see an authentication error initially, because you have to specify your access key… so,

 

we need to reference our .pem file into our command…

 

so cd to the directory where your .pem file is located, then do:

 

ssh -i <your.pem file> ec2user@<yourEC2publicIP>

 

then you will see the error that you have to change your permissions of your key file:

 

so on your local client do

 

chmod 0400 *.pem

 

then you should be able to login ok.

 

 

Resolving SSH Connectivity Problems

 

if you still have a connection problem…

 

if it is “connection refused”, then this means the instance is reachable, but no SSH utility is running on the instance

 

Solutions:

 

Try to restart the instance

 

If it doesn’t work, terminate the instance and create a new one. Make sure you’re using Amazon Linux 2

 

If your security group is properly configured as above, and you still have connection timeout issues, then that means a corporate firewall or a personal firewall is blocking the connection.

 

 

In this case use EC2 Instance Connect as described below.

 

 

Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

 

This means either two things:

 

You are using the wrong security key or not using a security key. Please look at your EC2 instance configuration to make sure you have assigned the correct key to it.

 

You are using the wrong user. Make sure you have started an Amazon Linux 2 EC2 instance, and make sure you’re using the user ec2-user.

 

Finally,

 

if you were able to connect yesterday, but not today…

 

This is probably because you have stopped your EC2 instance and then started it again today. When you do so, the public IP of your EC2 instance will change. So you need to get the new public ip of the instance and try connecting to that.

 

 

EC2 Instance Connect

 

login to your AWS Web Management Desktop and navigate to the EC2 Dashboard, and select the instance you want to connect to, and then click on CONNECT.

 

This will open a CLI shell console session on your EC2 instance via the web-browser window.

 

 

 

 

 

Use IAM Roles instead of AWS Credentials (access keys) to login to your EC2 Instances

 

DO NOT EVER ENTER YOUR ID ACCESS KEY OR SECRET KEY CREDENTIALS into an EC2 instance.

 

 

You CAN enter your access key id and secret access key into your aws credentials within your ssh session, using the “aws credentials” cli command – but you are strongly advised NOT to do this!

 

if you do this, then anyone who has access to the file on your laptop or pc can access your instance… and they can retrieve your access keys from your instance!

 

 

Therefore, always use IAM Roles instead to grant users access to EC2 instances!

 

 

To do this, we attach the role we created earlier to our ec2 instance, via the ec2 dashboard:

 

in your EC2 Dashboard, select the EC2 instance  then select Actions – Security – Modify iam role 

 

then add the role you want, then choose the iam role you want to add… in this case our demo role.

 

 

you can check with:

 

[ec2-user@ip-172-31-87-242 ~]$
[ec2-user@ip-172-31-87-242 ~]$ aws iam list-users
{
“Users”: [
{
“UserName”: “kevin”,
“PasswordLastUsed”: “2022-08-08T09:21:00Z”,
“CreateDate”: “2022-08-08T09:17:45Z”,
“UserId”: “AIDAQT3XOKFNXLOJMYNW6”,
“Path”: “/”,
“Arn”: “arn:aws:iam::042663301467:user/kevin”
}
]
}
[ec2-user@ip-172-31-87-242 ~]$

 

 

 

Continue Reading

AWS Security Groups

An EC2 security group plays the role of a firewall. By default, a security group will deny all incoming traffic while permitting all outgoing traffic

 

You define group behavior by setting policy rules that will either block or allow specified traffic types.

 

You can update your security group rules and/or apply them to multiple instances.

 

Security groups control traffic at the instance level.

 

Security Groups or SGs are an essential part of the task of securing your incoming and outgoing network traffic within a VPC.

 

However, SGs also have limitations and shouldn’t be thought of as the only line of defence.

 

AWS provides a range of additional networking security tools, such as Network ACLs, AWS Network Firewall, DNS Firewall, AWS WAF (Web Access Firewall), AWS Shield, as well as monitoring and compliance tools like AWS Security Hub, GuardDuty, and Network Access Analyzer.

 

Quick Overview of the main features of AWS Security Groups

 

Security groups are access control lists (ACLs) that allow network traffic inbound and outbound from an Elastic Network Interface (ENI). SGs serve as a basic firewall system for all AWS resources to which they are attached.

 

Security groups implicitly DENY traffic, they only have “allow” rules, and not “deny” rules.

 

Thus the absence of an “allow” rule will implicitly DENY access

 

Security group rules are “stateful” – meaning that if a server can communicate outbound to a service, the return traffic to that server will also be automatically permitted.

 

This behaviour contrasts with Network ACLs which are “stateless”.

 

Security groups are VPC-specific and thus also region-specific – which means they can only be used within the VPC where they are created.

 

Sole exception to this is where there is a peering connection to another VPC, in which case they can be referred to in the peered VPC.

 

Security groups can be applied to multiple instances within a VPC, and can be valid across subnets within that VPC.

 

When a VPC is first created, a “default” security group is automatically created by AWS for it and within it.

 

This default SG has no inbound rules and just a single outbound rule, which allows all traffic to any destination (0.0.0.0/0).

 

If a new resource is launched within the VPC without association to an SG then it will automatically be assigned to this “default” SG.

 

By default, a new security group attached to a group of instances does not allow these instances to communicate with each other. For this to be possible you need to create new inbound and outbound rules, and then define the source and destination as the security group itself.

 

Note that SGs are allocated to the actual Elastic Network Interface device (ENI) which is attached to an EC2 instance, as opposed to the EC2 / RDS instance itself.

 

You can assign up to five SGs to each Elastic Network Interface. Thus an EC2 instance with multiple ENIs could actually therefore have more than five security groups assigned to it, though this is not best practice.

 

Best Practices for Deploying Security Groups

Avoid using the AWS “default” security group.

 

The “default” security group should not be used for active AWS resources. The reason is that new AWS resources could be inadvertently assigned to it and so be permitted undesired access to your resources. Since you cannot delete the default SG, you should instead delete all inbound and outbound rules in the “default” SG instead.

 

Then create new SGs for your AWS resources. Then ensure any new AWS resources are assigned to the correct SGs you created.

 

Keep the number of your created SGs to a minimum

 

The AWS EC2 Launch Wizard will encourage you to create a new security group for each EC2 instance you create. However, the problem with that is that it can lead to the creation of too many security groups, which then become hard to manage and track.

 

Instead of relying on the Launch Wizard, you should decide on a strategy for creating security groups based on your actual application and service access needs and then assign them in accordance with your requirements.

 

Restrict, restrict, restrict

 

This means you should always strive to ensure all new security groups apply the principle of least privilege. this means for example:

 

Don’t open inbound traffic for the whole VPC CIDR-IP or subnet range unless absolutely necessary.

 

Also avoid allowing all IP addresses (0.0.0.0/0) unless absolutely necessary.

 

Restrict outbound traffic where possible e.g. do not open all ports to the internet, for example, only open HTTP/HTTPS to allow internet webaccess.

 

If allowing a specific service, only open the ports and protocols required for that service. For example, for DNS, you open port 53 for TCP and UDP, and only to the external DNS provider – for example for Google DNS that would be 8.8.8.8/32.

 

Open relevant ports and protocols to other security groups, rather than for IP addresses/subnets. This is because AWS recommends using dynamic IP addresses.

 

Develop a security group strategy

 

Set a strategy for creating and using security groups based on your application and service needs. Then use AWS tools to enforce this strategy.

 

Some examples of strategy options:

 

Decide the best way to set up your security groups.

 

For example: deploy one security group per service type, such as “mysql-db”, “web”, “ssh-access”,“active-directory”. Then assign the required inbound and outbound ports for that service.

 

Define one security group per application type, such as “web-servers”, “file-servers”, or “db-servers”. Then assign the ports for that application or service.

 

Define one security group per operating system, such as “linux” or “windows”, then assign the basic ports required for that OS.

Define one or two default SGs to cover access requirements common to all or most of the servers in the VPC. This minimizes the total number of security groups required.

 

Example:
if all instances need outbound using HTTP / HTTPS to access the web
if all instances need an inbound port for eg a monitoring system.

Try to use a naming strategy that provides clarity to help avoid confusion if you manage multiple VPCs. If each VPC has a security group called for example “web-servers”, then it can quickly become difficult to keep track of which is which.

 

 

Create and Maintain a Separate Security Group Just for Managing SSH Access

 

This is a very good idea as SSH is complicated and it is important to ensure it is securely managed.

 

 

Time Out versus Connection Refused

Important Exam Question!

 

The Difference Between Time Out Errors and Connection Refused Errors

 

Time out errors are always to do with security group permissions or firewall issues.

 

Connection refused errors are to do with the application level, not the security group level

 

 

You can access your EC2 instance using AWS EC2 Instance Connect as a workaround while you fix the problem.

 

 

AWS Firewall Manager can help create security group policies and associate them with accounts and resources. It can also be used to monitor and manage the policies for all linked accounts.

 

AWS Security Hub works with CloudTrail and CloudWatch to monitor and trigger alarms based on security best practice alerts.

 

One indication to watch out for is the rate of change within SG rules – such as ports being opened and closed again within a very short time period.

 

AWS Config can be used to ensure compliance with defined best practices. Config Rules can be created to check and alert for non-compliance, and then perform automated remediation steps.

 

For example, checking for unrestricted security group rules and ensuring the “default” AWS SG has no inbound or outbound rules set.

 

 

Security Groups Simplified

 

Security Groups are used to control access (SSH, HTTP, RDP, etc.) with EC2. They act as a virtual firewall for your instances to control inbound and outbound traffic.

 

When you launch an instance in a VPC, you can assign up to five security groups to the instance. 

 

Note that security groups act at the instance level, not the subnet level.

 

 

Security groups filters IP & Port according to rules. 

 

This is the basic firewalling system of AWS, this can be modified…

 

Security Group INBOUND allows inbound port 22 traffic from your computer

 

Security Group OUTBOUND allows outbound ANY PORT to ANY IP

 

In other words,

 

by default in AWS:

All INBOUND traffic is blocked by default
All OUTBOUND traffic is authorized by default

 

 

Security groups can be attached to MULTIPLE EC2 instances, not just assigned to a single EC2.

 

 

An instance can also belong to multiple security groups.

 

They are set for a specific region or VPC combination. So they are NOT cross-regional.

 

Security groups operate OUTSIDE the EC2 not inside it – so if the traffic is already blocked then the EC2 won’t see it. So it is not an “app” running on your EC2 but a service running outside of it.

 

Best practice is to create a separate security group just for SSH access.

 

TIP for error-debugging:

 

if your application gives a “time out” error, then it is not accessible and this means it is most likely a security group issue.

 

if your application gives a “connection refused” error, then it is an application-internal error or the application is not running.

 

 

Security Groups Key Details

 

Security groups control inbound and outbound traffic for your instances (they act as a Firewall for EC2 Instances) while NACLs control inbound and outbound traffic for your subnets (they act as a Firewall for Subnets). Security Groups usually control the list of ports that are allowed to be used by your EC2 instances and the NACLs control which network or list of IP addresses can connect to your whole VPC.

 

Every time you make a change to a security group, that change occurs immediately.

 

Whenever you create an inbound rule, an outbound rule is created immediately. This is because Security Groups are stateful. This means that when you create an ingress rule for a security group, a corresponding egress rule is created to match it. This is in contrast with NACLs which are stateless and require manual intervention for creating both inbound and outbound rules.

 

Security Group rules are based on ALLOWs and there is no concept of DENY when in comes to Security Groups. This means you cannot explicitly deny or blacklist specific ports via Security Groups, you can only implicitly deny them by excluding them in your ALLOWs list.

 

Because of the above detail, everything is blocked by default. You must intentionally allow access to certain ports.

 

Security groups are specific to a single VPC, so you can’t share a Security Group between multiple VPCs. However, you can copy a Security Group to create a new Security Group with the same rules in another VPC for the same AWS Account.

 

Security Groups are regional and CAN span AZs, but can’t be cross-regional.

 

Outbound rules exist if you need to connect your server to a different service such as an API endpoint or a DB backend. You need to enable the ALLOW rule for the correct port though so that traffic can leave EC2 and enter the other AWS service.

 

You can attach multiple security groups to one EC2 instance and you can have multiple EC2 instances under the umbrella of one security group.

 

You can specify the source of your security group (basically who is allowed to bypass the virtual firewall) to be a single /32 IP address, an IP range, or even a separate security group.

 

You cannot block specific IP addresses with Security Groups (use NACLs instead).

 

You can increase your Security Group limit by submitting a request to AWS

 

 

 

 

 

 

 

 

 

 

 

Continue Reading

AWS EC2 Elastic Cloud Compute

EC2 or Elastic Cloud Compute is the most popular AWS service.

 

EC2 consists of

 

Renting virtual machines (EC2s)

 

Storing data on virtual drives (EBS or Elastic Block Storage)

 

Distributing load across machines (ELB or Elastic Load Balancer)

 

Scaling EC2 services by using an auto-scaling group or ASG

 

 

EC2 Provisioning Considerations

 

You select an EC2 Instance Type, which specifies:

 

CPU power and cores

 

OS to deploy

 

RAM

 

Storage space

 

– network attached eg EBS & EFS

 

– hardware attached ie EC2 Instance Store

 

Networking – speed of interface, public IP, subnet/s

 

 

Firewalling rules – security group

 

Bootstrap script for first bootup – “EC2 User Data” script – can do almost anything – eg update software, download files, configure or copy config or data files etc

 

 

NOTE the user data script runs with sudo root user.

 

 

Choosing an AWS Region

 

You will usually want to choose a region closest to your customers or users.

 

For price comparisons for AWS regions see  https://instances.vantage.sh/ for precise info about ec2 options

 

and the AWS https://calculator.aws/#/

 

and https://www.instance-pricing.com/provider=aws-ec2/cheapest/ for cheapest regions

 

ap-south-1 Mumbai is cheap

 

Virginia, Ohio, Oregon and Mumbai are often the cheapest,  London and Frankfurt more expensive. 

 

N. Virginia is typically the cheapest region.

 

If you want to have the cheapest prices, best selection of products and highest number of Availability Zones, then N. Virginia tends to be best. This region is followed closely by Ireland, Oregon and Ohio.

 

Choosing any region outside of N. Virginia, Ireland, Oregon or Ohio means you could end up with only 2 AZs, probably pay more and may have to wait several weeks – even months – before new services are implemented in those regions.

 

 

EC2 Instance Type Naming Convention

 

example:

 

m5.2large

 

m = instance class
5 = hardware generation – improves over time
2xlarge = size within the instance class

 

price will obviously vary according to instance type.

 

 

 

Quick Overview of Instance Types 

 

General purpose:  A1, T3, T3a, T2, M6g, M5, M5a, M5n, M4

 

Compute optimized:  C5, C5n, C4

 

Memory optimized:  R5, R5a, R5n, X1e, X1, High Memory, z1d

 

Accelerated computing:  P3, P2, Inf1, G4, G3, F1

 

Storage optimized:  I3, I3en, D2, H1

 

 

Amazon EC2 Instance Types In More Detail 

 

M and T families: General purpose

 

The M and T families are the main EC2 instance types.

 

M family provides a good mix of CPU, RAM, plus disk size/performance,. M serves as a good choice for applications that have consistent performance requirements.

 

Generally, you can start with an M instance and then monitor performance. If the instance turns out to be inadequate for your performance needs, then you can switch over later to another family.

 

The T family is a low-cost alternative to the M family. It accommodates general-purpose workloads but is also burstable.

 

T instances are intended to operate at a lower level of basic performance but they also provide a form of built-in elasticity in being able to automatically burst performance as required.

 

They are especially useful for lower-throughput applications such as admin applications, low-traffic websites, or testing and development projects.

 

 

C family: Compute-optimized

 

C instances are designed for applications that require large amounts of computing power, using high ratios of vCPUs to RAM, but also offering the lowest cost per vCPU.

 

For example: on-demand batch processing activities, video encoding tasks, and high-performance science and engineering-based applications as well as front-end fleets that run high-traffic websites.

 

 

X, R, z1d, and High Memory families: Memory-optimized

 

The memory-optimized instance families offer the most RAM of all EC2 instance types and are primarily designed for running large in-memory databases. They range from 3TB to 24TB RAM

 

The X1, X2, R4, R5, R6, and z1d instances are especially designed for memory-intensive applications. These families also offer the lowest cost per GB of RAM, which makes them an ideal choice for applications that are heavy users of RAM.

 

The R families are well-suited to big-data processing in real-time, data mining, and Hadoop or Spark clusters.

 

X1 and X2 instances are ideal for enterprise-sized in-memory applications, such as SAP HANA which require greater RAM than the R family.

 

z1d instances offer high single-thread performance with a sustained all-core frequency of up to 4.0 GHz. This makes them the fastest of all AWS EC2 instances, providing both high-compute performances combined with high memory.

 

z1d is especially suited to tasks such as  electronic design automation or EDA, gaming, or relational database workloads which have high per-core licensing costs.

 

 

H, D, and I families: Storage-optimized

 

H, D, and I families provide high performance combined with local storage. This is in contrast to most other instance families, such as the compute-optimized and general-purpose instance types, which rely solely on attached EBS volumes rather than their own local storage.

 

They offer a wide range of storage sizes, either with HDDs or SSDs. H1 offers up to 16TB of hard drive storage.

 

H family is well-suited to workloads that use MapReduce or streaming operations such as Apache Kafka.

 

D3 provides up to 48TB of hard drive storage. Use cases for D3 include massively parallel processing data warehousing, Hadoop, and distributed file systems.

 

I3 instances include Non-Volatile Memory Express (NVMe) SSD-based instance storage. The I family provides low latency with high sequential read throughputs and very high random I/O performance, which makes I3 ideal for hosting in-memory databases,  NoSQL databases, data warehousing, running Elasticsearch, as well as analytics workloads.

 

P and G families: Accelerated computing

 

P and G instance families offer a combination of high performance and cost efficiency and are ideal for graphics-processing-intensive  and machine-learning applications.

 

P instances are suitable for general-purpose GPU applications such as video editing.

 

G instances are specifically designed for running GPU-heavy applications, such as automated speech recognition or language translation processes.

 

 

 

Securing Your EC2 Instance

 

 

You are responsible for configuring appropriate and effective access controls to protect your EC2 instances from unauthorized use.

 

Broadly speaking, AWS provides four tools to help you with this task:

 

security groups

 

Identity and Access Management (IAM) roles

 

network address translation (NAT) instances

 

key pairs.

 

 

 

EC2 Hibernate

 

 

You set hibernation in the EC2 Dashboard for the instance. 

 

Then to activate hibernation click on “Instance State” -> hibernate 

 

 

We have following options for EC2 Instances

 

Stop: data on EBS is kept intact for next start

 

Terminate: data on EBS is destroyed

 

First Start: OS boots, EC2 user data script runs

 

Subsequent Starts: OS boots

 

This however takes time…

 

With Hibernate however,

 

RAM in-memory state is preserved and boot is faster – OS state is frozen

 

this is written to a file in EBS volume -must have sufficient space and root

EBS volume must be encrypted. RAM is dumped to EBS.

 

Use cases:

 

for long running processes you dont want to interrupt or stop

 

saving the RAM state

 

services that take long time to initialize and start

 

limitations:

 

RAM must be under 150GB

 

not available for bare metal instances

 

must be Linux or Windows os

 

AND – cannot be hibernated for more than 60 days!

 

 

 

 

EC2 Nitro 

 

This is a new virtualization tech for EC2 for the future.

 

offers
better networking options
high performance computing
ipv6 support
higher speed EBS volumes supported
better security

 

types using Nitro: C5+, D G, M, instance types.
plus bare metal al.metal, c5.metal etc

 

vCPU

 

multiple threads can run on 1 CPU. multithreading

 

each thread is called vCPU in AWS

 

eg m5.2xlarge

 

is 4 CPU
2 threads per CPU
which makes 8 vCPU

 

the vCPU value does not specifiy the thread and CPU combination!

 

but in some cases you may want to decrease the number of vCPUs on your instance

 

eg to create more RAM and lower CPUs for software licensing costs

 

to change number of threads per core: you can disable multithreading which will mean 1 thread per CPU, which is helpful for hpc computing workloads

 

 

EC2 Capacity Reservations

 

ensure you have sufficient capacity available when needed-

 

it is a short term reservation

 

immediate reservation, billing starts immediately on reservation

 

you specify instance type, which AZ – only 1 AZ per reservation,  if you want eg 3 AZs then you need to specify 3 reservations.

 

you can combine with reserved instances and savings plans to cut cost.

 

 

AMI Amazon Machine Image

 

 

AMI is a customization of an EC2 instance

 

you add your own software, config, os, etc

 

faster boot/config because all the software is prepackaged

 

can be built for a specific region and can be copied across regions

 

you can launch EC2 instances from

 

a public AMI -provided by AWS

 

and we can also create our own AMIs but you have to make and maintain them yourself

 

or you can purchase AMIs in the AWS Marketplace made by others

 

you can also sell AMIs yourself in the marketplace

 

The AMI Process from an EC2 perspective

basically

 

we start an EC2 instance and customize it

 

then stop the instance – to ensure data integrity

 

we create an AMI of the instance – and this also creates EBS snapshots

 

we launch instances from our AMI

 

 

so, in detail:

 

 

create an EC2 instance, select type, network etc, add storage, some advanced details, including user data script.

 

then you create the AMI image using the EC2 dashboard, this takes a little time then it will be listed under ami images.

 

you can then use the image to create additional new EC2s – ie it serves as a full template instance.

 

 

 

 

 

 

 

 

 

 

Continue Reading

AWS – Setting Up A Billing Budget

In order to grant permissions to your admin user to use billing, you must first login as your AWS Root Account 

 

In AWS Management Console -> My Account: 

 

Then click on IAM User and Role Access to Billing Information

 

edit -> activate IAM Access and click update

 

then you can go to your admin user account in the AWS Management Console (logout of root) and you should now have access permission to Billing and Cost Management Dashboard.

 

 

You can call up your bills, and see exactly what charges are being incurred for which service and for which time period.

 

You can also display the Free Tier Services 

 

Next, set up a cost budget. Choose a recurring fixed budget and set a value eg 10 USD for the course.

Set at least one threshold alert to send an email when this is reached, eg at 80%

 

and also a forecast alert, eg 60% if it looks like this will be reached, you will be sent an email.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Continue Reading

AWS – Accessing AWS

First, you need to install AWS on your desktop system.

 

To install aws cli using apt-get package on Ubuntu:

 

apt-get update
apt-get install awscli

 

check with:

root@len:/home/kevin# aws –version
aws-cli/1.18.69 Python/3.8.10 Linux/5.15.0-43-generic botocore/1.16.19
root@len:/home/kevin#

 

The cli-command “aws”   is the tool command that is used to manage your AWS resources from the command-line.

 

kevin@len:~$ aws
usage: aws [options] <command> <subcommand> [<subcommand> …] [parameters]
To see help text, you can run:

aws help
aws <command> help
aws <command> <subcommand> help
aws: error: the following arguments are required: command
kevin@len:~$

 

 

For detailed help see: 

 

NAME
aws –

DESCRIPTION
The AWS Command Line Interface is a unified tool to manage your AWS
services.

SYNOPSIS
aws [options] <command> <subcommand> [parameters]

Use aws command help for information on a specific command. Use aws
help topics to view a list of available help topics. The synopsis for
each command shows its parameters and their usage. Optional parameters
are shown in square brackets.

 

 

 

The Three Ways of Accessing AWS

 

 

AWS Management Console (password + MFA)

 

AWS CLI Command Line Interface (via access keys)

 

  • this can be via AWS Cloudshell from the AWS Management Console, or directly using SSH

 

AWS SDK Software Developer Kits – (via access keys)

 

Access keys are generated on the AWS Console

 

Access Key ID is your username

 

Secret Access Key is your password – don’t share this!

 

 

 

if you do:

 

root@len:/home/kevin# aws iam list-users
Unable to locate credentials. You can configure credentials by running “aws configure”.
root@len:/home/kevin#

 

you can see you have no credentials with AWS. So you first have to create them.

 

Obtain your access-key id and secret access key from your IAM Management Console for the required user.

 

then copy paste these keys into:

 

root@len:/home/kevin# aws configure
AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]:
Default output format [None]:
root@len:/home/kevin#

 

 

 
Use access keys to make programmatic calls to AWS from the AWS CLI, Tools for PowerShell, AWS SDKs, or direct AWS API calls. You can have a maximum of two access keys (active or inactive) at a time.

 

For your protection, you should never share your secret keys with anyone. As a best practice, use frequent key rotation.
If you lose or forget your secret key, you cannot retrieve it. Instead, create a new access key and make the old key inactive. 

 

You can also use the cloudshell on the aws system directly, via the aws website dashboard. You can perform pretty well the same actions on cloudshell as you can with the aws cli on your own machine.

 

AWS has different services such as EC2, S3, etc. To get help on configuring specific AWS services using AWS CLI, you can read the respective man pages as follows:

$ aws <service-name> help

 

Continue Reading