Tags Archives: aws

Elasticache

Elasticache is an AWS managed data caching service mainly for databases and applications.

 

ElastiCache uses one of two open-source in-memory cache engines for its functionality:

 

Memcached and Redis.

 

 

 Elasticache is used to reduce traffic overhead for RDS and some other applications. It is extremely fast as db is held in ram memory.

 

Your cache must have an invalidation strategy defined to ensure only the most currently used data is stored in the cache.

 

It can also be used to store user sessions for an application for cases where users may be redirected later to different instances of the application, saving having to re-do the user login session.

 

But it does require code configurations for apps to be able to query the cache.

 

 

ElastiCache includes a feature for master/slave replication and multi-AZ, can be used for achieving cross-AZ redundancy and thus high-availability through the use of Redis replication groups.

 

 

Memcached

 

Memcached is an ASCII text file memory object caching system. ElastiCache is protocol compliant with Memcached, thus all the tools used with existing Memcached environments can also be used with ElastiCache. This is the simplest caching model and can also be used when deploying large nodes with multiple cores and threads.

 

Redis

Redis is an open-source in-memory key-value store that supports information structures such as lists and sorted sets.

 

Redis can power multiple databases, as well as maintain the persistence of your key store and works with complex data types — including bitmaps, sorted sets, lists, sets, hashes, and strings.

 

If Cluster-Mode is disabled, then there is only one shard. The shard comprises the primary node together with the read replicas. Read replicas store a copy of the data from the cluster’s primary node.

 

Elasticache allows for up to 250 shards for a Redis cluster if Cluster-Mode is enabled. Each shard has a primary node and up to 5 read replicas.

 

When reading or writing data to the cluster, the client determines which shard to use based on the keyspace. This avoids any potential single point of failure.

 

 

Implementing ElastiCache

 

There are three main implementation steps:

 

Creating an ElastiCache cluster
Connecting to the ElastiCache cluster from an EC2 instance
Managing the ElastiCache environment from the AWS console

 

 

Creating an ElastiCache cluster

 

This involves choosing and configuring the caching engine to be used. This will be either Redis or Memcached. For each caching engine, configuration parameters differ.

 

Next, we need to choose the location ie in AWS cloud or On-Premise.

 

AWS Cloud – This uses the AWS cloud for your ElastiCache instances

 

On-Premises – In this case, you can create your ElastiCache instances using AWS Outpost.

 

AWS Outposts is a fully managed service that extends AWS infrastructure, services, APIs, and tools to your own on-site infrastructure.

 

 

ElastiCache REDIS Replication –  Cluster Mode Disabled

 

There are two possible configuration modes for running ElastiCache and REDIS:

 

Cluster Mode Disabled, and Cluster Mode Enabled:

 

In this configuration you run ONE PRIMARY NODE of ElastiCache with up to 5 Read Replicas

 

Note that uses asynchronous replication to maintain the Read Replicas, so there is a lag.

 

The primary node is always used for read/write. The other nodes are read-only.

 

There is just ONE SHARD and all shards hold all the data.

 

Multi-AZ is enabled by default for failovers.

 

 

ElastiCache REDIS Replication –  Cluster Mode Enabled 

 

With Cluster Mode Enabled the data is  partitioned across MULTIPLE SHARDS

 

Data is divided across all your shards. This helps especially with scaling write transactions.

 

Each shard consists of a primary node and up to 5 read replica nodes.

Also has multiple AZ availability

 

Provides up to 500 nodes per cluster with a single master node.

or 250 nodes with 1 master and 1 replica.

 

 

 

Scaling REDIS with ElastiCache

 

For “Cluster Mode Disabled”:

 

Horizontal scaling – you scale out or in by adding or removing read replicas

 

Vertical scaling – you alter the type of the underlying nodes 

 

Important for exam!

 

This is done by means of ElastiCache creating a NEW node group with the new type specification for the nodes, then performing a replication to the new node group, and then finally updating the DNS records so that they point to the new node group and not any longer to the old original node group before scaling.

 

For “Cluster Mode Enabled”:

 

this can be done in two different ways – online, and offline:

 

Online: no interruption to service no downtime, but can be some performance degredation during the scaling.

 

Offline: service is down, but additional configurations are supported

 

So, when doing horizontal REDIS scaling, you can do online and office rescaling, and you can do resharding or shard rebalancing for this:

 

Resharding: “resharding” – this means scaling in or out by adding or removing shards.

 

Shard rebalancing:  involves equally redistributing the keyspaces among the shards as balanced as possible.

 

Vertical Scaling: you are changing to a larger or smaller node type, this is done online only, relatively straightforward.

 

 

 

REDIS Metrics to Monitor

 

Evictions: this is the number of non-expired items the cache has removed in order to make space for new writes ie the memory was full.

 

In this case choose an eviction policy to evict expired items eg least recently used items, LRU  or scale up to a larger node type with more memory, or else scale out by adding more nodes

 

CPUUtilization: this monitors CPU usage for the entire host, if too high, then scale up to a larger node type with more memory

 

SwapUsage: this should not be allowed to exceed 50Mb, if it does then verify you have configured enough reserved memory

 

CurrConnections: no of current connections – see if a specific app is causing this

 

DatabaseMemoryUsagePercentage:

 

NetworkBytesIn/Out & NetworkPAcketsIn/Out

 

ReplicationBytes: vol of data being replicated

 

ReplicationLag: how far behind the replica is from the primary node

 

 

 

 

 

 

 

 

 

Some ElastiCache use cases

 

know these for the exam!

 

Updating and managing leaderboards in the gaming industry

 

Conducting real-time analytics on live e-commerce sites

 

Monitoring status of customers’ accounts on subscription-based sites

 

Processing and relaying messages on instant messaging platforms

 

Online media streaming

 

Performing geospatial processes

 

 

 

Pros and Cons of Using ElastiCache

 

Pros of ElastiCache

 

Fully-managed – ElastiCache is a fully-managed cloud-based solution.

 

AWS takes care of backups, failure recovery, monitoring, configuration, setup, software updating and patches, and hardware provisioning.

 

Improved application performance – ElastiCache provides in-memory RAM data storage that substantially reduces database query times.

 

Easily scalable – you can scale up and down with minimal overhead

 

Highly available – ElastiCache achieves high availability through automatic failover detection and use of standby read replicas.

 

Cons of ElastiCache

 

Limited and complex integration – ElastiCache doesn’t provide many easy options for integration. And you can only connect Elasticache to databases and applications hosted by AWS.

High learning curve – the Elasticache user interface is not intuitive and the system requires a high learning overhead to properly understand.

 

High price – You pay only for what you use but the costs of using Elasticache can swiftly rise according to usage.

 

 

Comparison of ElastiCache With Redis, CloudFront, And DynamoDB

 

ElastiCache is very different to all these services.

 

 

AWS ElastiCache versus Redis

 

 

ElastiCache is an in-memory cache in the cloud. With very fast retrieval of data from managed in-memory caches, Elasticache improves overall response times, and saves relying wholly on slow disk-based databases for processing queries.

 

Redis stands for Remote Dictionary Server — a fast, in-memory, open-source, key-value data store that is usually implemented as a queue, message broker, cache, and database.

 

ElastiCache is developed on open-source Redis to be compatible with the Redis APIs, as well as operating seamlessly with Redis clients.

 

This means that you can run your self-managed Redis applications and store the data in an open Redis format, without having to change the code.

 

ElastiCache versus CloudFront

 

While ElastiCache and CloudFront are both AWS caching solutions, their approaches and framework differ greatly.

 

ElastiCache enhances the performance of web applications by retrieving information from fully-managed in-memory data stores at high speed.

 

To do this it utilizes Memcached and Redis, and is able in this way to substantially reduce the time applications need to read data from disk-based databases.

 

Amazon CloudFront is primarily a Content Delivery Network (CDN) for faster delivery of web-based data through deploying endpoint caches that are positioned closer to the traffic source. This saves too much web traffic from further-flung geolocations from having to source content entirely from the original hosting server.

 

ElastiCache versus DynamoDB

 

DynamoDB is a NoSQL fully-managed AWS database service that holds its data on solid state drives (SSDs). These SSDs are then cloned across three availability zones to increase reliability and availability. In this way, it saves the overhead of building, maintaining, and scaling costly distributed database clusters.

 

ElastiCache is the AWS “Caching-as-a-Service”, while DynamoDB serves as the AWS “Database as a Service”.

 

 

Pricing of ElastiCache

 

To use ElastiCache you have to make a reservation- Pricing for this is based on the caching engine you choose, plus the type of cache nodes.

 

If you are using multiple nodes (ie replicas) in your cluster, then ElastiCache will require you to reserve a node for each of your cluster nodes.

 

 

Difference Between Redis and Memcached

 

 

REDIS: similar to RDS

multi AZ with auto failover
read replicas used to scale reads and provide HA.

 

Data durability

 

provides backup and restore

REDIS:

Primary use case: In-memory database & cache   Cache
Data model: In-memory key-value 
Data structures: Strings, lists, sets, sorted sets, hashes, hyperlog
High availability & failover: Yes 

 

Memcached by contrast:

 

Memcached
Primary use case: Cache
Data model: In-memory key-value
Data structures: Strings, objects
High availability & failover: No

 

 

is multi-node data partitioning ie sharding

 

no HA

 

non-persistent data

 

no backup and restore

multi-threaded architecture

 

 

Main Points To Remember About REDIS and Memcached

 

REDIS is for high-availability – memcached has no AZ-failover, only sharding.

 

Also REDIS provides backup & restore – memcached does not.

 

Memcached has a multi-threaded architecture, unlike REDIS.

 

 

 

Redis Metrics to Monitor

 

Evictions: this is the number of non-expired items the cache has removed in order to make space for new writes ie the memory was full.

 

In this case choose an eviction policy to evict expired items eg least recently used items, LRU

 

scale up to a larger node type with more memory, or else scale out by adding more nodes

 

CPUUtilization: this monitors CPU usage for the entire host, if too high, then scale up to a larger node type with more memory

 

SwapUsage: this should not be allowed to exceed 50Mb, if it does then verify you have configured enough reserved memory

CurrConnections: no of current connections – see if a specific app is causing this

 

DatabaseMemoryUsagePercentage:

 

NetworkBytesIn/Out & NetworkPAcketsIn/Out

 

ReplicationBytes: vol of data being replicated

 

ReplicationLag: how far behind the replica is from the primary node

 

 

 

Memcached Scaling

 

Memcached clusters can have 1-40 nodes soft limit

 

Horizontal scaling: you add or remove nodes from the cluster and use “Auto-discovery” to allow you app to identify the new nodes or new node config.

 

Vertical scaling:  scale up or down to larger or smaller node types

 

to scale up: you create a new cluster with the new node type

 

then update your app to use the new cluster endpoints

 

then delete the old cluster

 

Important to note that memcached clusters/nodes start out empty, so your data will be re-cached from scratch once again.

 

there is no backup mechanism for memcached.

 

 

Memcached Auto Discovery

 

automatically detects all the nodes

 

all the cache nodes in the cluster maintain a list of metadata about all the nodes

 

note: this is seamless from the client perspective

 

Memcached Metrics to Monitor

Evictions: the number of non-expired items the cache evicted to allow space for new writes (when memory is overfilled). The solution: use a new eviction policy to evict expired items, and/or scale up to larger node type with more RAM or else scale out by adding more nodes

 

CPUUtilization: solution: scale up to larger node type or else scale out by adding more nodes

 

SwapUsage: should not exceed 50MG

 

CurrConnections: the number of concurrent and active connections

 

FreeableMemory: amount of free memory on the host

 

 

 

 

 

Continue Reading

AWS DB Parameters

Database or DB parameters specify how your database is configured. For example, database parameters can specify the amount of resources, such as memory, to allocate to a database.

 

You manage your database configuration by associating your DB instances and Multi-AZ DB clusters with parameter groups. Amazon RDS defines parameter groups with default settings.

 

A DB Parameter Group is a collection of engine configuration values that you set for your RDS database instance.

 

It contains the definition of what you want each of these over 400 parameters to be set to.

 

By default, RDS uses a default parameter group specified by AWS. It is not actually necessary to use a different parameter group.

 

Each default parameter group is unique to the database engine you select, the EC2 compute class, and the storage allocated to the instance.

 

You cannot change a default parameter group, so if you want to make modifications then you will have to create your own parameter group.

 

RDS database engine configuration is managed through the use of parameters in a DB parameter group.

 

DB parameter groups serve as an effective container for holding engine configuration values that are applied to your DB instances.

 

A default DB parameter group is created if you make a database instance without specifying a custom DB parameter group. This default group contains database engine defaults and Amazon RDS system defaults based on the engine, compute class, and allocated storage of the instance.

 

 

When you create a new RDS database, you should ensure you have a new custom DB parameter group to use with it. If not then you might have to perform an RDS instance restart later on to replace the default DB parameter group, even though the database parameter you want to alter is dynamic and modifiable.

 

This is the best approach that gives you flexibility further down the road to change your configuration later on.

 

Creating your own parameter group can be done via the console or the CLI. Self-created parameter groups take as their basis the default parameter group for that particular instance and selected db engine.

 

After creation, you can then modify the parameters via the console or CLI to suit your needs as you change.

 

Parameters can either be static or dynamic.

 

Static means that changes won’t take effect without an instance restart.

 

Dynamic means a parameter change can take effect without an instance restart.

 

Dynamic parameters are either session scope or global scope.

 

Global scope dynamic parameters mean that changes will impact the entire server and all sessions.

 

Session scope dynamic parameters however are only effective for the session where they were set.

 

Note however that some parameter variables can have both global and session scope.

 

In these cases, the global value is used as the default for the session scope and any global change to a parameter that also has a session scope will only affect new sessions.

 

Another important aspect to bear in mind when creating a DB parameter group:

 

 

You should wait at least 5 minutes before creating your first DB instance that uses that DB parameter group as the default parameter group. This allows Amazon RDS to fully complete the create action before the parameter group is used as the default for a new DB instance.

 

 

For exam!

 

Important to know this DB Parameter above all for the exam:

 

for PostgreSQL and SQLServer:

 

rds.force_ssl=1

 

this forces ssl connections to be used

 

 

 

BUT – for MySQL /MariaDB you must instead use a grant select command:

 

GRANT SELECT ON mydatabase.* TO ‘myuser’@%’IDENTIFED BY ‘…’ REQUIRE SSL;

 

 

 

Continue Reading

AWS – Migration of On-Premises Infrastructure to AWS Cloud

The Migration Process can be split into three parts:

 

 

Before AWS Migration

 

During AWS Migration

 

After AWS Migration

 

 

 

 

AWS Migration: 5 Cloud Migration Steps

 

 

These are the 5 principal AWS Migration steps you need to consider:

 

Planning and Assessment
Migration Tools
AWS Cloud Storage Options
Migration Strategies
Application Migration Options

 

Planning and Assessment

 

The planning and assessment phase is divided into:

 

Financial Assessment
Security & Compliance Assessment
Technical and Functional assessment

 

Financial Assessment

 

Before deciding on-premises to cloud migration, you need to estimate the cost of moving data to the AWS cloud. A careful and detailed analysis is required to weigh the financial considerations of on-premises center versus employing a cloud-based infrastructure.

 

Security and Compliance Assessment

 

Overall risk tolerance
Main concerns around availability, durability, and confidentiality of your data.
Security threats
Options available to retrieve all data back from the cloud

 

Classify your data according to these concerns. This will help you decide which datasets to move to the cloud and which ones to keep in-house.

 

 

Technical and Functional Assessment

 

Assess which applications are more suited to the cloud strategically and architecturally.

 

Points to consider:

 

Which applications or data should best be moved to the cloud first?
Which data can we transfer later?
Which applications should remain on-premises?
Can we reuse our existing resource management/configuration tools?
What do we do about support contracts for hardware, software, and networking?

 

For small-scale data migrations

 

Unmanaged Cloud Data Migration Tools

 

For simple, low-cost methods for transferring smaller volumes of data:

 

Glacier command line interface- On-premises data → Glacier vaults
S3 command line interface- Write commands → Data moves directly into S3 buckets
Rsync- Open source tool combined with 3rd party file system tools. Copy data directly → S3 buckets

 

 

For large-scale data migrations

 

AWS Managed Cloud Data Migration tools

For moving larger volumes of data:

how much data to migrate? Which AWS data migration tool is best suited

Migrate petabytes of data in batches to the cloud AWS Import/Export Snowball
Migrate exabytes of data in batches to the cloud AWS Snowmobile
Connect directly to an AWS regional data center AWS Direct Connect
Migrate recurring jobs, plus incremental changes over long distances Amazon S3 Transfer Acceleration

 

 

Some Practical Strategies for AWS Migration

 

Forklift Migration Strategy

 

This is more suitable for self-contained, tightly-connected or stateless applications. Its a “pick up everything and move it in one go to the cloud” method.

Is best suited to smaller environments.

 

Hybrid Mixed-Migration Strategy

 

This involves moving some parts of an application to the cloud while leaving other parts of the application on-premises.

 

It is best suited to migrating larger systems which run multiple applications. However, it can be more time-consuming to complete the migration in this way.

 

 

Configuring and Creating AMI Images

 

AMIs provide the information needed to launch an EC2 instance.

 

 

Online data transfer from on-premises to AWS

 

Here are the online data transfer options.

 

 

AWS Virtual Private Network

 

There are two options for using AWS VPN:

 

 

AWS Site-to-Site VPN
AWS Client VPN

 

AWS VPN is encrypted, easy to configure and cost-effective for small data volumes. However, it is a shared connection, so not as fast or reliable as other options.

 

 

AWS Virtual Private Network (AWS VPN) establishes secure private connection your network to AWS.

 

 

 

AWS VPN is encrypted, easy to configure and cost-effective for small data volumes. However, it is a shared connection, so not as fast or reliable as other options.

 

 

 

AWS Database Migration Service

 

 

The AWS Database Migration Service as the name suggests handles database migration to AWS. The big advantage of DMS is that the database remains fully operational and usable during the migration.

 

AWS S3 Transfer Acceleration

 

To migrate large quantities of data over longer distances to AWS S3, AWS S3 Transfer Acceleration enables you to do this 50-500% faster yet still using the public internet.

 

Data is routed to S3 via optimized network paths using Amazon CloudFront Edge Locations situated across the globe. This maximizes available bandwidth. You select this service on the S3 Dashboard console, selecting one of two TA options. The transfer acceleration is then activated without any need for special client applications or additional network protocols.

 

AWS DataSync

 

AWS DataSync enables users to automate the migration of on-premises storage to S3 or Amazon EFS and can transfer up to 10 times faster than some open source migration services. It deploys an on-premises software agent which connects to your on-premises storage system via NFS (Network File System) and SMB (Server Message Block) protocols.

 

DataSync also takes care of much of the transfer overhead such as Running instances, encryption, managing scripts, network optimization, and validating data all while transferring data up to 10 times faster than many open source migration services.

 

It can be used to copy data via AWS Direct Connect or public internet to AWS, and is suitable for both one-time data migration, and recurring workflows, as well as for automated backup and recovery actions.

 

 

AWS Direct Connect

 

AWS Direct Connect is a dedicated connection from your on-premises to AWS.

 

As with AWS VPN, Direct Connect provides an encrypted connection between your on-premises environment and AWS.

 

However, Direct Connect does not use the public internet and instead runs via a private connection it establishes which will be either via a 1 GB or 10 GB fiber-optic Ethernet cable used to connect your router to an AWS Direct Connect router. On other words, the Direct Connect solution is part software and part hardware.

 

Because of this dedicated connection, Direct Connect is significantly more costly than using just public internet-and-VPN solutions.

 

But if you need to transfer or stream very large amounts of data back and forth to the AWS Cloud, then a Direct Connect line may be the best solution. However for smaller transfer one-off migrations it is not so suited.

 

 

AWS Storage Gateway

 

Storage Gateway enables users to connect and extend their on-premises applications to AWS storage.

 

Storage Gateway provides cloud-backed file shares and provides a low-latency cache for on-premises applications to access data in AWS.

 

This service has three alternative gateways available:

 

File Gateway: data is stored in S3 using Amazon S3 File Gateway or using fully-managed file shares through Amazon FSx File Gateway.

 

Tape Gateway: this is a virtual tape library (VTL) which integrates with existing backup software for long-term storage on S3 Glacier and S3 Glacier Deep Archive.

 

Volume Gateway: this stores data locally, backing up block volumes with EBS snapshots

 

 

AWS data transfer pricing

 

AWS wants to encourage potential customers to use its platform, so generally speaking it doesn’t charge for migrating data to AWS.

 

However note that there are often charges levied for transferring back out again from AWS.

 

Generally, the charges for data migration depend on the resources and infrastructure used in facilitating the transfer. This will depend on the method you choose, your region/s used, the instances and other resources you use, and how fast the connection is.

 

As from April 2022, inter-Availability Zone (AZ) data transfers within the same AWS Region for AWS PrivateLink, AWS Transit Gateway, and AWS Client VPN are now free of charge.

 

The best way to calculate your exact data transfer costs is to use the AWS Pricing Calculator and the AWS Cost Explorer.

 

 

AWS VPN pricing

 

AWS VPN costs are calculated according to how many hours the connection is active:

 

$0.05 per Site-to-Site VPN connection per hour and per AWS Client VPN connection per hour for connections to US

 

 

AWS Database Migration Service pricing

 

If you’re using AWS Database Migration Service to transfer existing databases to Amazon Aurora, Redshift, or DynamoDB, then you can enjoy free usage for six months.

 

After that time, you only pay for the compute resources, ie instances that you use to port databases to AWS, plus any additional log storage space required.

 

Each DMS database migration instance will include sufficient storage for swap space, replication logs, and data caching to cover the majority of cases.

 

On-demand EC2 instances are priced by hourly usage, depending on how powerful the instance is, and whether you are choosing single or multiple availability zones for your instances.

 

Instance pricing is from $0.018 per hour, up to $21.65 per hour for multi-AZ instances with fastest processor performance and lowest network latency.

 

 

AWS S3 Transfer Acceleration pricing

 

Pricing for AWS S3 Transfer Acceleration service is based on the volume of data you are migrating to S3, rather than how long you are using the connection.

 

examples:

 

Data accelerated via Edge Locations in the United States, Europe, and Japan: $0.04 per GB

 

Data accelerated via all other AWS Edge Locations: $0.08 per GB

 

Transfer Acceleration constantly monitors its own speed, and if speeds are not faster than a standard transfer via public internet then you will not be charged for the service.

 

AWS DataSync pricing

 

For AWS DataSync, you are charged according to the amount of data you transfer via the service. This is currently priced at $0.0125 per gigabyte (GB) of data transferred.

 

AWS Direct Connect pricing

 

Direct Connect is priced by the hour. There are two cost options according to the capacity of your Dedicated Connection:

 

1G: $0.30/hour

 

10G: $2.25/hour

 

If you wish to transfer data out using Direct Connect, then there are additional charges to pay for this facility.

 

AWS Storage Gateway pricing

 

Charges for AWS Storage Gateway are based on the type and amount of storage you use, as well as the requests you make and the volume of data you are transferring out.

 

Data Transfer out from AWS Storage Gateway service to on-premises gateway device is charged between $0.05-$0.09 per GB.

 

Data Transfer in via your gateway device to Amazon EC2 costs $0.02 per GB.

 

 

Some Tips For Minimizing Data Migration Costs

 

Keep your data transfer within a single AWS Region and Availability Zone

 

Utilize cost allocation tags to identify and analyse where you’re incurring your highest data transfer costs

 

Deploy Amazon CloudFront to reduce EC2 Instance/s to public Internet transfer costs, and utilize CloudFront’s free tier for the first year of use (note this is valid only up to 50 GB of outbound data transfer and 2 million HTTP requests per month)

 

Reduce the volume of data that you need to transfer whenever possible before starting the migration.

 

Deploy VPC endpoints to avoid routing traffic via the public Internet when connecting to AWS

 

 

AWS suggest the following schema for deciding on which migration method to choose:

 

 

 

 

 

Time Overhead for Migrating Data to AWS 

 

This is the formula suggested by AWS to determine how long it will take to transfer data to AWS from your on-premises site.

 

 

Number of Days = (Total Bytes)/(Megabits per second * 125 * 1000 * Network Utilization * 60 seconds * 60 minutes * 24 hours)

 

 

Let’s consider a very simple example consisting of just one virtual server machine of say 20GB in total size (no separate file server or other devices in this example)

 

 

So that will give us following calculation:

 

 

don’t forget to convert megabytes to megabits first. So our 20GBytes becomes using the table at https://convertlive.com/u/convert/gigabytes/to/megabits#20 for this:

 

 

Total Bytes will be:

 

 

20 Gigabytes = 21474836480 Bytes

 

 

21 474 836 480 Bytes

 

 

that’s just over 21.4 billion Bytes

 

so Number of Days = (Total Bytes)/(Megabits per second * 125 * 1000 * Network Utilization * 60 seconds * 60 minutes * 24 hours)

 

 

 

Connection & Data Scale Method Duration

 

 

Less than 10 Mbps & Less than 100 GB Self-managed ~ 3 days
Less than 10 Mbps & Between 100 GB – 1 TB AWS-Managed ~ 30 days
Less than 10 Mbps & Greater than 1 TB AWS Snow Family ~ weeks
Less than 1 Gbps & Between 100 GB – 1 TB Self-managed ~ days
Less than 1 Gbps & Greater than 1 TB AWS- Managed / Snow Family ~ weeks

 

 

Post AWS Migration Stage

 

After completing the migration process, make sure you run all necessary tests, and confirm everything is working correctly.

 

In particular you should look at configuring CloudWatch, CloudTrail and other monitoring services, plus AWS Auto Scaling and  CloudFront if required.

 

 

 

Continue Reading

AWS Security Services

Services that provide DDOS Protection on AWS

 

AWS Shield Standard, free of charge, is activated by default

 

AWS Shield Advanced – 24×7 premium protetion, fee charged and access to AWS DRP DDOS Response Team v expensive about 3000 USD per month.

 

AWS WAF filters specific requests based on rules – layer 7 http – for app load balancer, api gateway and CloudFront
you can define web acl : geo block, ip address blocks, sql injection etc

 

 

CloudFront and Route 53: uses global edge network, combined with Shield can provide attack mitigation at the edge

 

You can utilize AWS AutoScaling to leverage up if there is an attack.

 

You get full DDOS protection by combining Shield, WAF, CloudFront and Route53.

 

 

Penetration testing can be carried out by customers for 8 services eg EC2, RDS, CF etc – you don’t need any authorization to do this but you cannot do simulated ddos attacks on your system or dns zone walking on route 53, nor flooding tests

 

important note:
for any other simulated attacks, contact aws first, to check, otherwise it is not authorized – and could be seen as an infrastructure attack on aws!  

 

AWS Inspector:

 

chargable, first 15 days free. not cheap. cost per instance or image scanned.

 

does automated security assessments, eg for EC2

 

sends reports to security hub and event bridge

 

leverages the System Manager SSM agent

 

for Containers pushing to ECR –  assesses containers as they are moved to ECR

 

it is ONLY for EC2 and container infra. But only done when needed.

 

checks packages against CVE – package vulnerability scan

 

also does network reachability for EC2

 

that is all.

 

 

Logging on AWS – quick overview

 

aws services generate a wide range of logs

 

cloudtrail trails, config rules, cw logs vpc flow logs, elb access logs cloud front logs, waf logs,

 

 

exam question!
LOGS can be analyzed using AWS Athena if stored on S3.

 

you should encrypt logs stored on S3 and control the access to them by deploying iam and bucket policies plus mfa.

and always remember:
don’t log a server that is logging! otherwise you create an endless logging loop!

 

and move logs to glacier for cost saving

 

and also use glacier vault which locks the logs so they cant be tampered with.

 

 

AWS Guard Duty

 

this uses intelligent threat discovery and ML learning to detect

 

no need to install any software, works in the backend, only need to activate, but it chargeable
esp analyses cloudtrail, vpc flow logs, dns logs, kubernetes audit logs, looks for unusual api calls etc

you can set up cloudwatch events rules to connect to labda or sns

exam q
also can protect against cryptocurrency attacks, has a dedicated function for it. – comes up in exam

 

 

AWS Macie

a fully managed data security and data privacy service which uses ML pattern matching to protect your data

 

helps identify and alert esp re PII – personal identifiable information

 

can notify event bridge

 

 

 

AWS Trusted Advisor – only need to know overview for the exam

 

no need to install, is a service

 

core checks and recommendations — available for all customers, these are free

 

can send you a weekly email notification

 

full trusted advisor for business and enterprise – fee based
and can then create cloudwatch alarms or use apis

 

cost optimization

 

looks for underutilized resources – but cost optimizn is not in the free core checks, so you need to upgrade for this.

 

performance

ec2s ebs cloud front

 

security

 

mfa security used or not, iam key rotation etc, exposed access keys
s3 bucket permissions, security group issues, esp unrestricted ports

 

fault tolerance — eg abs snapshot age,

service limits

Continue Reading

AWS CloudTrail, CloudWatch, and Config Compared

CloudTrail is a services which provides governance, compliance, and auditing of your AWS account by logging and monitoring account activities.

 

What’s the difference between CloudWatch and CloudTrail

 

AWS CloudWatch

 

CloudWatch s a monitoring tool used for real-time monitoring of AWS resources and applications. It provides a monitoring service which analyzes the performance of the system.  

 

CloudWatch can be used to detect irregular behaviour in an AWS environments. It monitors various AWS resources including EC2, RDS,  S3, Elastic Load Balancer, etc. It can also be used with CloudWatch Alarms. 

 

 

2. AWS CloudTrail

 

CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account. It continuously logs and monitors the activities and actions across your AWS account. It provides the event history of your AWS account including data about who is accessing your system.  Remediation actions can also be taken by CloudTrail.      

 

While CloudWatch reports on the activity and health and performance of your AWS services and resources,  CloudTrail by contrast is a log of all the actions that have taken place inside your AWS environment.

 

CloudTrail can record API activity in your AWS account and reports an event within 15 minutes of the API call.

 

 

It provides auditing services for AWS accounts. In CloudTrail, Logs are saved in an S3 bucket.

 

However, you can receive notification of specific CloudTrail events immediately by sending them via the CloudWatch Event Bus.

 

While CloudTrail only writes to your S3 bucket once every five minutes, it sends events to the CloudWatch Event bus in near real-time as these API calls are observed.

 

CloudWatch monitors performance. For example, tracking metrics of an EC2 instance or keeping track of your Amazon DynamoDB performance or to see how Amazon S3 is performing. CloudWatch allows you to collect default metrics for over 70 AWS services.

 

It also has a “Custom Metrics” feature that enables you to collect a metric that is specifically important to your system. For example, to measure how people are using your application.

 

 

AWS CloudTrail

 

AWS CloudTrail is principally used for auditing API activity, tracking who did what and when, and securely logging this information to Amazon S3 for later analysis.

 

Thus CloudTrail keeps track of what is done in your AWS account, when, and by whom. For example, with CloudTrail you can view, search, and download latest activity in your AWS account to check it there are any abnormal or unusual actions and if so, by whom. This type of reporting is called auditing and it is the core service of CloudTrail.

 

 

CloudTrail tracks data events and management events:

 

Data events are object-level API requests made to your resources. For example, when an item is created or deleted in a DynamoDB table.

 

Management events log changes (mostly creation or deletion changes) to your environment, such as the creation or deletion of the entire DynamoDB itself.

 

CloudTrail tracks which applications or persons took these actions and stores the details in logfiles. These logfiles are encrypted and stored in S3.

 

 

Note that CloudWatch has CloudWatch Alarms which you can configure and metric data is retained for 15 months. CloudTrail on the other hand has no native alarms. However, you can configure CloudWatch Alarms for CloudTrail, but you have to store logs in S3.

 

In a nutshell:

 

CloudWatch is for performance. Think of CloudWatch as monitoring application metrics.
CloudTrail is for auditing. Think of CloudTrail as tracking API activity within an account.

 

 

 

 

AWS Config vs. CloudTrail

 

In the configuration and monitoring category AWS, there are two major AWS monitoring tools that are similar, and are easy to confuse. They are AWS Config and AWS CloudTrail.

 

Config and CloudTrail are different tools with different purposes.

 

What is AWS Config?

 

AWS Config is a service that lets you set configuration rules for your AWS resources to comply with. It then tracks whether the resources comply with those rules.

 

Whenever a resource has changed, Config records the change in a configuration history in an S3 bucket. It stores a snapshot of the system at a regular period of time set by you. It also has a dashboard that presents an overview of your resources and their configurations.

 

 

What is AWS CloudTrail?

 

CloudTrail is a logging service that records all API calls made to any AWS service. It records the details of the API call such as which user or application made the call, the time and date it happened and the IP address it originated from.

 

There is also another AWS logging service called CloudWatch Logs, but unlike CloudWatch Logs which reports application logs, CloudTrail reports on how AWS services are being used in your environment.

 

Where CloudTrail and Config are Similar

 

Config and CloudTrail have a number of things in common. Both are monitoring tools for your AWS resources. Both track changes and store a history of what happened to your resources in the past. Both are used for compliance and governance, auditing and security policies. If you notice something unusual or going wrong with your AWS resources, then chances are you’ll see it reported in both CloudTrail and Config.

 

Where CloudTrail and Config are Different

 

 

Note that AWS Config Rules is not a cheap service. There is no free tier, you pay a fee per config item per region.

 

Though both often report on the same events, their approach is different. Config reports on what has changed in the configuration, whereas CloudTrail reports on who made the change, and when, and from which IP address.

 

Config reports on the configuration of your AWS resources and creates detailed snapshots of how your resources have changed.

 

CloudTrail focuses on the events or API calls behind those changes, focusing on users, applications, and activities performed in your environment.

 

Where CloudTrail and Config work together

 

By taking a different approach to the same events, CloudTrail and Config make a good combination. Config is a great starting point for ascertaining what has happened to your AWS resources, while CloudTrail can give you more information from your CloudTrail logs.

 

Config watches and reports on instances of rules for your resources being violated. It doesn’t actually allow you to make changes to these resources from its own console.

 

By contrast, CloudTrail gives you more control by integrating with CloudWatch Events to allow you to set automated rule-based responses to any event affecting your resources.

 

In the case of security breaches, if multiple changes have been made by an attacker in a short period of time, Config might not report this in detail.

 

Config stores the most recent and important changes to resources but disregards smaller and more frequent changes.

CloudTrail by contrast records every single change in its logs. It also has an integrity validation feature that checks if the intruder or attacker manipulated the API logs to cover their activity track.

 

 

Should You Use AWS Config or CloudTrail for Security?

 

 

Both Config and CloudTrail have a role to play together. Config records and notifies about changes in your environment. CloudTrail helps you find out who made the change, from where, and when.

 

A good way to think of it is that AWS Config will tell you what your resource state is now or what it was at a specific point in the past whereas CloudTrail will tell you when specific events in the form of API calls have taken place.

 

So you ought to use both. Config Rules triggers on a change in the status of your system, but it will often only give you an update on the state of the system itself.

 

CloudTrail meanwhile provides you with a log of every event which details everything that has taken place and when and by whom. This helps identify all the causes that may have led to the security problem in the first place.

 

 

Remember also that AWS Config Rules does not prevent actions from happening – it is not a “deny”.

 

But – you can do “remediations” of resources that are identified as non-compliant. This can be done for example via SSM Automation Documents. Config then triggers an auto-remediation action that you define.

 

Notifications:

 

you can use EventBridge to receive notifications from Config, from there you can also send the notifications onto eg Lambda functions, SNS or SQS.

Continue Reading

AWS Additional Monitoring Tools

 

AWS Config

 

 

AWS Config is an AWS fully managed change management solution within AWS. It allows you to track the change history of individual resources and configure notifications when a resource changes.

 

This is achieved by means of config rules. A config rule represents the desired state that the resource should be in.

 

Config rules allow you to monitor for systems that fall outside of your set baselines and identify which changes caused the system to fall out of compliance with the baseline. AWS Config is enabled on a per-region basis, so you need to enable it for every region in which you want to use it.

 

Bear in mind that AWS Config is a monitoring tool and does not actually enforce baselines, nor does it prevent a user from making changes that cause a resource to move out of compliance.

 

AWS Config enables you to capture the configuration history for your AWS resources, maintain a resource inventory, audit and evaluate changes
in resource configuration, and enable security and governance by integrating notifications with these changes. You can use it to discover AWS resources in your account, continuously monitor resource configuration against desired resource configuration, and check the configuration details for a resource at a given point in time.

 

AWS Config is used to assess compliance as according to your set internal guidelines for maintaining resource configurations, as well as enabling compliance auditing, security analysis, resource change tracking, and assisting with operational troubleshooting.

 

AWS Trusted Advisor

 

AWS Trusted Advisor service analyzes and checks your AWS environment in real-time and gives recommendations for the following four areas:

 

Cost optimization
Performance
Security
Fault tolerance

 

Trusted Advisor or TA integrates with AWS IAM so you can control access to checks as well as to categories.

 

The current status of these checks is displayed in the TA dashboard as follows:

 

Red: Action recommended
Yellow: Investigation recommended
Green: No problem detected

 

Where the colour is red or yellow, TA provides alert criteria, recommended actions, and relevant resource details, such as details of the
security groups allowing unrestricted access via specific ports.

 

Six core checks are available for all AWS customers free of charge.

 

Five checks for security plus one check for performance:

 

eg:
service limits
IAM use
security groups-unrestricted ports
MFA on root account
Elastic block storage public snapshot
RDS public snapshot.

 

 

 

AWS Inspector

 

AWS Inspector provides for the automation of security assessments. The assessments can be set to run on a schedule or when an event occurs that is monitored by Amazon CloudWatch, or also via an API call. The dashboard shows the assessments, as well as the findings from the various scans that have run.

 

Amazon Inspector makes use of assessment templates that define which sets of rules you want to run against your environment.

 

Two types of assessments are offered by AWS Inspector: network assessments and host assessments.

 

Network assessments don’t require any agent to be installed. However if you want detailed information about processes running on a specific port then you need to install the AWS Inspector Agent.

 

Host assessments however require the Inspector Agent to be installed. These assessments are far more detailed and scan for things such as vulnerable versions of software, violations of security best practices, and areas that should be system hardened. You can select these assessments set up AWS Inspector.

 

You create an assessment template in Inspector which you then use to assess your environment by means of an Assessment Run which will then report on its findings.

 

Templates contain one or more rules packages. A rules package defines what you are checking for. Note that you can’t create custom rules packages; you can use only the rules packages provided by AWS. Currently, these are the rules packages available, listed by assessment type:

Network assessments

Network Reachability: This rules package checks your environment’s network configurations, including your security groups, network access control lists (NACLs), route tables, subnets, virtual private cloud (VPC), VPC peering, AWS Direct Connect and virtual private gateways (VPGs), Internet gateways (IGW), EC2 instances, elastic load balancers (ELBs), and elastic network interfaces (ENIs).

 

Host assessments

Common Vulnerabilities and Exposures (CVE): This rules package checks your systems to see if they are vulnerable to any of the CVEs reported.

 

Center for Internet Security (CIS) Benchmarks: This rules package assesses your systems against CIS benchmarks specific to your OS.

 

There are Level 1 and Level 2 checks. Level 1 is usually safe to implement; Level 2 is more risky as the settings in Level 2 may have unintended side effects. Level 2 is usually used in environments where a very high level of security is required.

 

Security Best Practices: This rules package assesses how well your environment confirms to security best practices. Eg, it will check that a Linux EC2 instance cannot be logged into via SSH.

 

Runtime Behavior Analysis: This rules package identifies risky behaviors on your systems, such as using insecure protocols for connecting or open ports that are not in use.

 

 

 

AWS GuardDuty

 

GuardDuty is the AWS intrusion detection system (IDS) or intrusion prevention system (IPS). It uses threat intelligence feeds and analyzes logs from multiple sources, such as VPC flow logs, AWS CloudTrail event logs, and DNS logs.

 

GuardDuty can alert you to suspicious activity that could indicate potential issues such as leaked user account credentials, privilege escalation attacks, and possible command-and-control type activities.

 

GuardDuty scans specifically for three types of activity:

 

Reconnaissance
Instance compromise
Account compromise

 

Reconnaissance is the first step of an attack and was defined in the “Cyber Kill Chain”, developed by Lockheed Martin. During the reconnaissance
phase, an attacker is learning about your environment through actions such as vulnerability scans to probe for IP addresses, hostnames, open ports, and misconfigured protocols.

 

GuardDuty can detect also utilize threat intelligence feeds to detect IP addresses known to be malicious. You can use findings reported by GuardDuty to automatically remediate the vulnerability become it develops into a security violation.

 

The next type of activity is instance compromise. This consists of several indicators that may be present, such as malware command and control, crypto miners, unusual traffic levels or unusual network protocols, or communication with a known malicious IP.

 

 

Continue Reading

AWS CloudWatch Monitoring Overview

 

AWS CloudWatch is the basic AWS monitoring service that collects metrics on your resources in AWS, including your applications, in real time.

 

You can also collect and monitor log files with AWS CloudWatch. You can set alarms for metrics in CloudWatch to continuously monitor performance, utilization, health, and other parameters of your AWS resources and take action when metrics cross set thresholds.

 

CloudWatch is a global AWS service, so it can monitor resources and services across all AWS regions via a single dashboard.

 

 

CloudWatch provides basic monitoring free of charge at 5-minute intervals as a serverless AWS service, thus there is no need to install any additional software to use it.

 

 

For an additional charge, you can set detailed monitoring that provides data at 1-minute intervals.

 

 

AWS CloudWatch has a feature that allows you to publish and retain custom metrics for a 1-second or 1-minute duration for your application, services, and resources, known as high-resolution custom metrics.

 

CloudWatch stores metrics data for 15 months, so even after terminatíng an EC2 instance or deleting an ELB, you can still retrieve historical metrics for these resources.

 

 

How CloudWatch Works

 

CW Monitoring Is Event-Driven

 

All monitoring in AWS is event-driven. An event is “something that happens in AWS and is captured.”

 

For example, when a new EBS volume is created, the createVolume event is triggered, with a result of either available or failed. This event and its result are sent to CloudWatch.

 

You can create a maximum of 5000 alarms in every region in your AWS account.

 

You can create alarms for functions such as starting, stopping, terminating, or recovering an EC2 instance, or when an instance is experiencing a service issue.

 

Monitoring Is Customizable

 

You can define custom metrics easily. A custom metric behaves just like a predefined one and can then be analyzed and interpreted in the same way as standard metrics.

One important limitation of CloudWatch – exam question! 

 

CloudWatch functions below the AWS Hypervisor, which means it functions below or the virtualization layer of AWS.

 

This means it can report on things like CPU usage and disk I/O…but it cannot see beyond what is happening above that layer.

 

This means CloudWatch cannot tell you what tasks or application processes are affecting performance.

 

Thus it cannot tell you about disk usage, unless you write code that checks disk usage and send that as a custom metric to CloudWatch.

 

This is an important aspect that can appear in the exam. You might be asked if CloudWatch can report on memory or disk usage by default; it cannot.

 

Monitoring Drives Action

 

The final piece of the AWS monitoring puzzle is alarms – this is what occurs after a metric has reported a value or result outside a set “everything is okay” threshold.

 

When this happens, an alarm is triggered. Note that an alarm is not necessarily the same as “something is wrong”; an alarm is merely a notification that something has happened at a particular point.

 

For example, it could be running some code in Lambda, or sending a message to an Auto Scaling group telling it to scale in, or sending an email via the AWS SNS message service.

 

Think of alarms as saving you from having to sit monitoring the CloudWatch dashboard 24×7.

 

One of your tasks as SysOp is to define these alarms.

 

 

CloudWatch Is Metric- and Event-Based

 

Know the difference between metrics and events.

An event is predefined and is something that happens, such as bytes coming into a network interface.

 

The metric is a measure of that event eg how many bytes are received in a given period of time.

 

Events and metrics are related, but they are not the same thing.

 

CloudWatch Events Are Lower Level

 

An event is something that happens, usually a metric changing or reporting to CloudWatch, but at a system level.

 

An event can then trigger further action, just as an alarm can.

 

Events are typically reported constantly from low-level AWS resources to CloudWatch.

 

CloudWatch Events Have Three Components

 

CloudWatch Events have three key components: events, rules, and targets.

 

An event:

 

the thing being reported. Events describe change in your AWS resources. They can be thought of as event logs for services, applications and resources.

 

A rule:

 

 

an expression that matches incoming events. If the rule matches an event, then the event is forwarded to a target for processing.

 

 

A target:

 

 

is another AWS component, for example a piece of Lambda code, or an Auto Scaling group, or an email or SNS/SQS message that is sent out.

 

 

Both alarms and events are important and it is essential to monitor both.

 

CloudWatch Namespaces

 

A CloudWatch Namespace is a container for a collection of related CloudWatch metrics. This provides for a way to group metrics together for easier understanding and recognition.

AWS provides a number of predefined namespaces, which all begin with AWS/[service].

 

Eg, AWS/EC2/CPUUtilization is CPU utilization for an EC2 instance,

 

 

AWS/DynamoDB/CPUUtilization is the same metric but for DynamoDB.

 

 

You can add your own custom metrics to existing AWS namespaces, or else create your own custom namespaces in CloudWatch.

 

 

exam question:

CloudWatch can accept metric data from 2 weeks earlier and 2 hours into the future  but make sure your EC2 instance time is set accurately for this to work correctly!

 

 

Monitoring EC2 Instances

 

CloudWatch provides some important often-encountered metrics for EC2.

 

Here are some of the most common EC2 metrics which you should be familiar with for the exam:

 

 

CPUUtilization – one of the fundamental EC2 instance metrics. It shows the percentage of allocated compute units currently in use.

 

DiskReadOps – reports a count of completed read operations from all instance store volumes.

 

DiskWriteOps – opposite of DiskReadOps, reports a count of completed read operations from all instance store volumes.

 

DiskReadBytes – reports the bytes read from all available instance store volumes.

 

DiskWriteBytes – reports the total of all bytes written to instance store volumes.

 

NetworkIn – total bytes received by all network interfaces.

 

NetworkOut – total bytes sent out across all network interfaces on the instance.

 

NetworkPacketsIn – total number of packets received by all network interfaces on the instance (available only for basic monitoring).

 

NetworkPacketsOut – number of packets sent out across all network interfaces on the instance. Also available only for basic monitoring.

 

 

 

S3 Metrics

 

There are many S3 metrics, but these are the most common ones you should know:

BucketSizeBytes – shows the daily storage of your buckets as bytes.

NumberOfObjects – the total number of objects stored in a bucket, across all storage classes.

 

AllRequests – the total number of all HTTP requests made to a bucket.

 

GetRequests – total number of GET requests to a bucket. There are also similar metrics for other requests: PutRequests , DeleteRequests , HeadRequests , PostRequests , and SelectRequests.

 

BytesDownloaded – total bytes downloaded for requests to a bucket.

 

BytesUploaded – total bytes uploaded to a bucket. These are the bytes that contain a request body.

 

FirstByteLatency – per-request time for a completed request, by first-byte millisecond.

 

TotalRequestLatency – the elapsed time in milliseconds from the first to the last byte of a request.

 

 

 

CloudWatch Alarms

 

 

Alarms Indicate a Notifiable Change

 

 

A CloudWatch alarm initiates action. You can set an alarm for when a metric is reported with a value outside of a set level.

 

Eg, for when your EC2 instance CPU utilization reaches 85 percent.

 

 

Alarms have three possible states at any given point in time:

 

OK : means the metric lies within the defined threshold.

ALARM : means the metric is below or above the defined threshold.

 

INSUFFICIENT_DATA : can have a number of reasons. The most common reasons are that the alarm has only just started or been created, that the metric it is monitoring is not available for some reason, or there is not enough data at this time to determine whether the alarm is OK or in ALARM state.

 

 

CloudWatch Logs

 

CloudWatch Logs stores logs from AWS systems and resources and can also handle the logs for on-premises systems provided they have the Amazon Unified CloudWatch Agent installed.

 

If you are monitoring AWS CloudTrail activity through CloudWatch, then that activity is sent to CloudWatch Logs.

 

If you need a long retention period for your logs, then CloudWatch Logs can also do this.

 

By default logs are kept forever and never expire. But you can adjust this based on your own retention policies.

You can choose to keep logs for only a single day or go up to 10 years.

 

Log Groups and Log Streams

 

You can group logs together that serve a similar purpose or from a similar resource type. For
example, EC2 instances that handle web traffic.

 

 

Log streams refer to data from instances within applications or log files or containers.4

 

 

CloudWatch Logs can send logs to S3, Kinesis Data Streams and Kinesis Data Firehose, Lambda and ElasticSearch

 

 

CloudWatch Logs – sources can be:

 

SDKs,

CloudWatch Logs Agent,

CloudWatch Unified Agent

Elastic Beanstalk

ECS – Elastic Container Service

Lambda function logs

VPC Flow Logs – these are VPC specific

API Gateway

CloudTrail based on filters

Route53 – logs DNS queries

 

 

Define Metric Filters and Insights for CloudWatch Logs

You can apply a filter expression eg to look for a specific IP in a log or the number of occurrences of “ERROR” in the log

Metric filters can be used to trigger CloudWatch Alarms

 

CloudWatch Logs Insights can be used to query logs and add queries to CloudWatch Dashboards

 

 

CloudWatch Logs – Exporting to S3

 

 

NOTE

this can take 12 hours for the data to become available for export – so it is not real time. For this you should use Log Subscriptions.

 

 

The API call for this is “CreateExportTask”

 

 

 

CloudWatch Log Subscriptions

 

You apply a “subscription filter” to the CloudWatch Log before sending it to eg a Lambda function managed by AWS/or to a custom-designed Lambda function and then from there as real-time data on to eg ElasticSearch. Or, you might send it from Subscription Filter and then to Kinesis.

 

 

You can also send or aggregate logs from different accounts and different regions to a subscription filter in each region and from there to a common single Kinesis Data Stream and Firehose and from there in near-real time on to eg S3.

 

 

 

 

 

 

 

Unified CloudWatch Agent

 

 

The AWS Unified CloudWatch Agent provides more detailed information than the standard free CloudWatch service.

 

You can also use it to gather logs from your on-premises servers in the case of a hybrid environment and then centrally manage and store them from within the CloudWatch console.

 

The agent is available for Windows and Linux operating systems.

 

When installed on a Windows machine, you can forward in-depth information to CloudWatch from the Windows Performance Monitor, which is built into the Windows operating system.

 

When CloudWatch is installed on a Linux system, you can receive more in-depth metrics about CPU, memory, network, processes, and swap memory usage. You can also gather custom logs from applications installed on servers.

 

To install the CloudWatch agent, you need to set up the configuration file.

 

 

Continue Reading

AWS NACLs – Network Access Control Lists

The AWS Network Access Control List (NACL) is a security layer for your VPC that acts as a firewall for controlling traffic in and out of one or more subnets.

 

NACLs vs. Security Groups

 

NACLs and Security Groups (SGs) both have similar purposes. They filter traffic according to rules, to ensure only authorized traffic is routed to its destination.

NACLs

 

NACLs are used to control access to network resources. They reside on subnets and evaluate traffic based on defined rules which you set, and use these rules to determine whether or not traffic should be allowed to pass through the subnet.

 

NACLs are “STATELESS” which means they require you to create separate rules for BOTH INCOMING AND OUTGOING traffic. Just because a particular data stream is allowed into the subnet, this doesn’t mean it will automatically be allowed out.

 

NACLs are processed in numerical ie serial order. Thus if you want traffic to be permitted both in and out of a subnet, you have to set network access rules for both directions.

 

NACLs are automatically applied to everything within that subnet, so there is no need to apply NACLs to individual resources as they are created. This means less network admin overhead for managers.

 

 

Security Groups

 

Security Groups apply to EC2 instances and operate like a host-based firewall. As with NACLs they apply rules that determine whether traffic to or from a given EC2 instance should be allowed.

 

This provides for more finely tuned traffic control for resources that have specific network traffic requirements.

 

Security Groups unlike NACLs are stateful; this means that any traffic that is allowed into your EC2 instance will automatically be allowed out again and vice versa.

 

All security groups rules are evaluated according to a default “deny everything unless allowed” policy. This means that if no ALLOW exists, then traffic will be blocked.

 

Security Groups must be applied at the time of resource creation and have to be explicitly configured.

 

 

Similarities and Differences Between NACLs and Security Groups

 

Both NACLs and Security Groups utilize rules that prevent unwanted traffic from accessing your network. The rules themselves also look similar. But a notable difference between them is that NACLs allow for DENY rules to be explicitly created.

 

It is important to ensure that your security group rules and your NACLs are not working against one another. Thus it is important to understand when it is best to use NACLs and when it is best to use SGs.

 

The major difference between them is in where they are applied. NACLs are applied at the SUBNET level, while Security Groups are applied at the EC2 instance level.

 

NACLs protect the network while Security Groups protect the resource.

 

As NACLs are higher up in the architecture, they apply to a much wider set of resources. Any NACL rule you create will therefore impact the operation of every resource located within the subnet.

 

Security Groups on the other hand only affect the EC2 instances to which they are attached.

 

 

When to Use NACLs

 

NACLs are best used sparingly. Because NACLs apply to the full set of resources in a subnet, their impact is wide and substantial.

 

NACLs are most effective for filtering external traffic to internal subnets. They can also be useful for applying traffic controls between the subnets themselves.

 

 

 

Best Practices for Using NACLs

 

Use NACLs sparingly and deploy them based on the function of the subnet they are attached to

 

Keep NACLs simple and only use them to deny traffic if possible

 

Restrict who can create or modify NACLs through IAM rules

 

Build your Security Group rules into your NACLs

 

Ensure that your inbound and outbound rules make sense ie that they match

 

When numbering your NACLs, be sure to leave room for future rules

 

Audit your rules frequently and delete any rules that are unused or redundant

 

Deploy NACLs to also control your subnet-to-subnet traffic and ensure logical separation between them

 

 

Continue Reading

AWS – Choosing an AWS Database

AWS offers several db solutions.

 

A number of questions need to be considered when choosing an AWS DB solution:

 

Questions to Consider

 

What is your need – read heavy, write heavy, balanced?

 

Throughput volume?

 

Will this change, does it need to scale, will it fluctuate up and down?, is it predictable or not?

 

How much data volume to store – and for how long

 

will it grow?

 

what is the average object size?

 

how do you need to access it?

 

what data durability do you require? what is the “source of truth” for the data?

 

are there compliance requirements?

 

what latency requirements – how many concurrent users

 

what is the data model – how will you query the data, will there be data joins? structured, semi-structured?

 

strong schema, flexible, reporting needs? searching? RBDMS or NoSQL?

 

what about license costs – cloud native db such as Aurora possible or not?

 

Overview of Database Types on AWS

 

RDBMS such as sql, oltp, this means: RDS, or Aurora, esp good for joins

NoSQL DBs such as DynamoDB – json, elasticache – key/value pairs or Nepture – good for graphs, but no joins and no sql

 

Object Stores: S3 for big objects, Glacier for backup and archives – may not seem like a DB but it works like one

 

Data Warehouse solutions eg SQL Analytics/BI, Redshift (OLAP), Athena

 

Search solutions: eg ElasticSearch (json), for free text unstructureed searches

 

Graph solutions: Neptune – this displays relationships between data

 

 

Overviews of AWS DB Solutions

 

RDS Overview

 

its a managed db on the postgresql/myswl/Oracle/SQL level

 

you must however an ec2 instance and ebs vol type and sufficient size

 

it supports read replicas and multi-AZ
security is via iam and security groups, kms, and ssl in transit
backup, snapshot and point in time restores all possible

 

managed and scheduled maintanance

 

monitoring available via cloudwatch

 

use cases include:

 

storing relational datasets rdbms/oltp performing sql queries, transactional inserts, update, delete is possible

 

rds for solutions architect, considerations include these “5 pillars”:

 

operations
security
reliability
performance
cost

 

 

operations_ small downtimes when failover happens, when maintenance happens, when scaling read replicas, ec2 instances, and restoring from ebs, this requires manual intervention, and when application changes

 

 

security: aws is responsible for os security, but we are responsible for setting up kms, security groups, iam policies, authorizing users in db and using ssl

 

 

reliability: the multi-az feature makes rds v reliable, good for failover in failure situations

 

performance: dependent on ec2 instance type, ebs vol type, can add read replicas, storage autoscaling is possible, and manual scaling of instances is also possible

 

costs: is pay per hour based on provisioned number and type of ec2 instances and ebs usage

 

 

Aurora Overview

 

Compatible API for PostgreSQL and MySQL

 

Data is held in 6 replicas across 3 AZs
It has auto-healing capability
Multi-AZ Auto Scaling Read Replicas
Read Replicas can be global

 

Aurora DB can be global for DR or latency purposes
auto Scaling of storage from 10GB to 128 TB

 

Define EC2 instance type for Aurora instances

 

same security/monitoring/maintenance as for RDS but also has

 

Aurora Serverless for unpredictable/intermittent workloads
Aurora Multi-Master for continuous writes failover

 

use cases: same as for RDS but with less maintenance/more flexibility and higher performance

 

Operations: less operations, auto scaling storage

 

security: AWS as per usual, but we are responsible for kms, security groups, iam policy, user auth and ssl

 

reliability: multi AZ, high availability, serverless and multimaster options

 

performance 5x performance, plus max 15 read replicas (vs only 5 for rds)

 

cost: pay per hour acc to ec2 and storage usage, can be lower cost than eg oracle

 

 

ElastiCache Overview

 

it is really just a cache not a database

 

is a managed Redis or Memcached – similar to RDS but for caches

 

its an in-memory data store with very low latency
you must provision an ec2 instance type to use it

 

supports redis clustering and multi-az, plus read replicas using sharding

 

security is via iam, security groups, kms, and redis auth – NO IAM

 

backup, snapshot, point in time restores
managed and scheduled maintenance
monitoring via cloudwatch

 

use case: key/value store, frequent reads, less writes, cache results for db queries, storing of session data for websites, but cannot use sql – latter is important to be aware of.

 

you retrieve the data only by key value, you can’t query on other fields

 

 

operations: same as rds
security: usual can use iam policies,for users Redis Auth, and ssl

 

reliability: clustering and multi AZ

 

performance: in memory so extremely fast, read replicas for sharding, very efficient

 

cost: similar to rds pricing based on ec2 and storage usage

 

 

DynamoDB

 

proprietary to AWS
a managed NoSQL DB
serverless, provisioned, auto-scaling, on-demand capacity

 

can replace elasticache as a key-value store, eg for storing session data

 

performance is slower than eg rds

 

highly available, multi-AZ by default, read–writes decoupled, DAW available for read cache

 

2 options for reads: eventually consistent or strongly consistent

 

security, authorization-authentication all done via iam

 

dynamodb streams integrate with lambda

 

backup-restore and global table feature – if you enable streams

 

monitoring via cloudwatch

 

important
but you can only query on primary key, sort key or indexes – exam q!
so you cannot query on “any” attribute – only the above.

 

use case:

 

serverless apps development, small docs 100s kb, distrib serverless cache, doesn’t have sql query language available, has transaction capability now built in

 

 

 

S3 Overview

 

acts as a simple key/value store for objects

 

great for big objects up to 5TB, not so good for small objects

 

serverless, scales infinitely, strong consistency for every operation

 

tiers for migrating data: s3 standard, s3 IA, s3 one-zone IA, Glacier for backups

 

features include: versioning, encryption, CRR – cross-region replication

 

security: IAM, bucket policies, ACL

 

encryption: SSE-S3, SSE-KMS, SSE-C, client side encryption, SSL in transit

 

use case: static files, key-value stores for big files and website hosting

 

operations: no operations necessary!
security: IAM, bucket policies, ACL, encryption set up correctly,

 

reliability: extremely high, durability also extremely good, multi-AZ and CRR

 

performance: scales easily, very high read/write, multipart for uploads

 

cost: only pay for storage used, no need to provision in advance, plus network costs for transfer/retrieval, plus charge per number of requests

 

 

Athena Overview

 

fully serverless database with SQL capability
used to query data in S3
pay per query
outputs results back to S3
is secured via IAM

 

use cases: one-time sql queries, serverless queries on S3, log analytics

 

 

Operations: no ops needed! is serverless

 

security: via S3 using bucket policies, IAM

 

reliability: is managed, uses Presto engine, highly available
performance: queries scale based on data size

 

cost: pay per query/per TB of data scanned, serverless

 

 

Redshift Overview

 

is a great data warehouse system

 

based on PostgreSQL but not used for oltp

 

instead, its an OLAP – online analytical processing — analytics and data warehousing

 

10x better performance than other data warehouses, scales to PBs of data

 

columnar storage of data rather than row-based

 

it is MPP – uses a massively parallel query execution engine which makes it extremely fast

pay as you go acc to no of instances provisioned
has an SQL interface for performing the queries.

 

data is loaded from S3, DynamoDB, DMS and other DBs,
1 to 128 nodes possible with up to 128 TB of space per node!

 

leader node used for query planning and results aggregation
compute node : performs the queries and sends results to leader node

 

Redshift Spectrum: performs queries directly against S3 with no load to load the data into the redshift cluster

 

Backup & Restore, SecurityVPC, IAM, KMS, monitoring

 

Redshift Enhanced VPC Routing: Copy / Unload goes through VPC, this avoids public internet

 

Redshift Snapshots and DR

 

has no multi-AZ node

 

the snapshots are p-i-t- point in time backups of a cluster, stored internally in S3

 

they are incremental — only changes are saved, makes it fast

can restore to a new cluster

automated snapshots: every 8 hrs every 5gb or according to a schedule, with a set retention period

 

manual snapshots: snapshot is retained until you delete it

 

neat feature:
you can configure Redshift to auto copy snapshots to a cluster of another region
either manually or automatically

 

 

loading data into redshift:

 

there are three possible ways:

 

1. use kinesis data firehouse loads data into redshift cluster via an s3 copy automatically to an S3 bucket

 

2. using the copy command manually without Kinesis
from S3 bucket via internet – ie without using enhanced vpc routing to redshift cluster
or via vpc not via internet, using enhanced vpc routing

 

 

3. from ec2 instance to redshift cluster using jdbc driver
in this case it is much better to write the data in batches rather than all at once.

 

 

Redshift Spectrum

 

must already have a redshift cluster operational

 

Spectrum is a way to query data that is already in S3 without having to load it into Redshift

 

you submit the query which is submitted to thousands of Redshift spectrum nodes for processing

 

 

Operations: similar to rds
security: uses iam, vpc, kms, ssl as for rds
reliability: auto healing, cross region snapshop copies possible
performance: 10x performance of other data warehousing systems, uses compression
cost: pay per node provisioned about 10% of cost of other dw systems
vs athena: is faster at querying, does joins, aggregations thanks to indexes

 

redshift = analytics/BI/Data Warehouse

 

 

Glue Overview

 

is a managed extract transform and load ETL service

 

used to prepare and transform data for analytics
fully serverless service

 

fetch data from s3 bucket or rds, send to glue which does extracting, transforming and loading to redshift data warehouse

 

glue data catalog: catalog of all the datasets you have in aws – ie metadata info about your data

 

you deploy glue data crawler to navigate s3, rds, dynamo DB, it then writes the metadata to the glue data catalog.

 

this can then be used b Glue Jobs eTL

 

data discovery -> Athena, Redshift Spectrum, EMR for analytics

 

all you need to know about Glue for exam

 

Finally,

 

Neptune Overview

 

tip:
if you hear graphs mentioned in exam, this refers to neptune!

 

is a fully managed graph database

 

used for
high relationship data
social networking, eg users friends with users, replied to comment on post of user and likes other comments

 

knowledge graphs eg for wikipedia – links to other wiki pages, lots of these links
this is graph data – giving a massive graph

 

highly available across 3 AZs with up to 15 read replicas available

 

point in time recovery, continuous backup to S3

 

support for kms encryption, https, and ssl

 

operations: similar to rds
security: iam, vpc, kms, ssl – similar to rds, plus iam authentication
reliability: multi-AZ and clustering
performance: best suited for graphs, clustering to improve performance

 

cost: similar to rds, pay per provisioned node

 

remember neptune = graphs database

 

 

AWS OpenSearch

 

– the successor to AWS ElasticSearch

 

eg dynamodb only allows search by primary key or index,

 

whereas with OpenSearch you can search ANY field

 

often used as a complement to other db

 

also has usage for big data apps
can provision a cluster of instances
built-in integrations: various – kinesis data firehose, aws IoT, CloudWatch Logs

 

 

comes with visualization dashboard

 

operations: similar to rds

 

security: is via Cognito, IAM, KMS encryption, SSL and VPC
reliability: multi-AZ and clustering
performance: based on elasticsearch project – open source, pbyte scale
cost: pay per provisioned node

 

all you need to remember:

 

used to search and index data

 

Question 1:
Which database helps you store relational datasets, with SQL language compatibility and the capability of processing transactions such as insert, update, and delete?

 

RDS

 

Question 2:
Which AWS service provides you with caching capability that is compatible with Redis API?

 

ElastiCache

Good job!
Amazon ElastiCache is a fully managed in-memory data store, compatible with Redis or Memcached.

 

 

Question 3:
You want to migrate an on-premises MongoDB NoSQL database to AWS. You don’t want to manage any database servers, so you want to use a managed NoSQL database, preferably Serverless, that provides you with high availability, durability, and reliability. Which database should you choose?

 

Amazon DynamoDB

 

Good job!

Amazon DynamoDB is a key-value, document, NoSQL database.

 

 

Question 4:
You are looking to perform Online Transaction Processing (OLTP). You would like to use a database that has built-in auto-scaling capabilities and provides you with the maximum number of replicas for its underlying storage. What AWS service do you recommend?

 

Amazon Aurora

 

Good job!

Amazon Aurora is a MySQL and PostgreSQL-compatible relational database. It features a distributed, fault-tolerant, self-healing storage system that auto-scales up to 128TB per database instance. It delivers high performance and availability with up to 15 low-latency read replicas, point-in-time recovery, continuous backup to Amazon S3, and replication across 3 AZs.

 

 

 

Question 5:
As a Solutions Architect, a startup company asked you for help as they are working on an architecture for a social media website where users can be friends with each other, and like each other’s posts. The company plan on performing some complicated queries such as “What are the number of likes on the posts that have been posted by the friends of Mike?”. Which database do you recommend?

 

Amazon Neptune

 

Good job!
Amazon Neptune is a fast, reliable, fully-managed graph database service that makes it easy to build and run applications that work with highly connected datasets.

 

 

Question 6:
You have a set of files, 100MB each, that you want to store in a reliable and durable key-value store. Which AWS service do you recommend?

 

Amazon S3

 

Good job!
Amazon S3 is indeed a key-value store! (where the key is the full path of the object in the bucket)

 

Question 7:
You would like to have a database that is efficient at performing analytical queries on large sets of columnar data. You would like to connect to this Data Warehouse using a reporting and dashboard tool such as Amazon QuickSight. Which AWS technology do you recommend?

 

Amazon Redshift

 

Good job!
Amazon Redshift

 

 

Question 8:
You have a lot of log files stored in an S3 bucket that you want to perform a quick analysis, if possible Serverless, to filter the logs and find users that attempted to make an unauthorized action. Which AWS service allows you to do so?

 

Amazon Athena

 

Good job!
Amazon Athena is an interactive serverless query service that makes it easy to analyze data in S3 buckets using Standard SQL.

 

 

Question 9:
As a Solutions Architect, you have been instructed you to prepare a disaster recovery plan for a Redshift cluster. What should you do?

 

enable automated snapshops, then configure your redshift cluster to autocopy the snapshots to another aws region

 

Good job!

 

 

Question 10:
Which feature in Redshift forces all COPY and UNLOAD traffic moving between your cluster and data repositories through your VPCs?

 

 

Enhanced VPC Routing

 

Good job!

 

Question 11:
You are running a gaming website that is using DynamoDB as its data store. Users have been asking for a search feature to find other gamers by name, with partial matches if possible. Which AWS technology do you recommend to implement this feature?

 

ElasticSearch

 

Good job!
Anytime you see “search”, think ElasticSearch.

 

 

Question 12:
An AWS service allows you to create, run, and monitor ETL (extract, transform, and load) jobs in a few clicks.

 

AWS Glue

 

Good job!
AWS Glue is a serverless data-preparation service for extract, transform, and load (ETL) operations.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Continue Reading

AWS DynamoDB

DynamoDB is a fully managed, highly-available database with replication across multiple AZs

 

NoSQL – not a relational database!

 

scales to massive workloads

 

100TBs of storage

 

fast and consistence, low latency retrieval

 

integrated with iam for security authorization and admin
enables event-driven programming via DynamoDB Streams

 

low cost and auto-scaling

 

standard and infrequent access 1A table class

 

 

Basics of DynamoDB

 

 

v important – also exam q:
made up of tables – there is NO SUCH THING AS A DATABASE – you only need to create tables!
the db already exists!

 

each table has a primary key – must be set at creation time
each table can have an infinite number of items ie rows
each item has attributes can be added over time or can be null

 

max size of an item is 400KB so not good for large objects

 

data types supported are:

 

scalar types: string, number, binary, boolean, null
document types: list, map
set types: string set, number set, binary set

 

tables:

 

have primary key – containing partition key (eg user_id) and sort key (eg game_id)
and
attributes: eg score, result – these are items which are NOT the primary key

 

read/write capacity modes

 

control how you manage your table capacity – ie the read and write throughput

 

we have 2 possible modes:

 

provisioned mode, where you specify number of read and writes per second, this is the default, you have to plan the capacity for this in advance, you pay for provisioned read capacity units RCUs and write capacity units (WCUs)

 

these “capacity units” are used to set your desired capacity for your database! – set these in the web dashboard of dynamodb when you are provisioning.

 

 

you can also add auto scaling mode for both rcu and wcu

 

on-demand mode:

read write automatically scales up and down acc to workload, so no capacity planning needed
you pay for your use, is more expensive 2-3x more

 

good for unpredictable workloads which can be large or small varying

 

need to know for exam!

 

remember always you just create tables, never a database with dynamodb!

 

DynamoDB Accelerator DAX

 

 

a fully manageable, highly available seamless in-memory cache for dynamodb

 

helps solve read congestion by means of memory caching

 

microseconds latency for cached data

 

does not require any application logic modification – so it is fully compatible with existing dynamodb api

 

applications can access the database via DAX for much faster reads

 

TTL is 5 mins default for dax data

dax is different from elasticache in that it is meant for dynamodb and does not change the api
food for individual object cache

 

elasticache
good for storing aggregation result where further processing required

 

 

DynamoDB Stream

 

very easy – is an ordered stream of data which represents items when created or updated or deleted

 

can then be sent on to kinesis datastreams
can be read by lambda or kinesis client library apps

 

data is retained for 24 hrs

 

 

use cases:

 

reacting to changes in real time eg welcome emails to new users
to do analytics
to insert into derivative tables or into elastic serarch
or implement cross region replication

 

 

DynamoDB Global Tables

 

 

are cross-region

 

two way or multi-way replication

 

to provide for low latency across regions

 

it is
active-active replication

 

this means apps can read and write to the table from any region

 

must enable dynamodb streams for this

 

TTL expiry: automatically deletes items after an expiry timestamp

 

eg 1 month for each row in the table

 

 

DynamoDB Indexes – only need to know at high level for exam

 

basically, these allow you to query on attributes other than the primary key

 

2 types:

 

gsi – global secondary and lsi – local secondary

 

(all you need to know for now)

 

by default you query on the primary key, but you can use gsi or lsi to query on other attributes.

 

 

Transactions

 

these allow you to write to 2 tables at the same time:

 

the transaction however MUST write to both or else none – in order to keep the tables accurate – so data consistency is maintained

 

 

 

 

 

 

 

 

Continue Reading

AWS Lambda

Serverless Services in AWS include:

 

Lambda
DynamoDB
Cognito
API Gateway
S3
SNS/SQS
Kinesis Data Firehose
Aurora Serverless
Step Functions
Fargate

 

exam tests heavily on serverless knowledge!

 

AWS Lambda

 

features:

 

virtual functions – server to manage
limited by time – short execution processes
runs on demand only, only billed when you are actually using it
the scaling is automated

 

the benefits of Lambda:

 

easy pricing – pay per request and compute time

 

free tier covers 1 million lalbda requests and 400k of GB compute time

 

integrated with all AWS services and programming languages
easy monitoring via CloudWatch
easy to allocate more resources per functions –
up to 10GB of RAM! possible

 

also, increasing RAM improves CPU and network

 

language support:

 

node.js – javascript
python
java 8
c .net core
golang
c powershell
ruby
custom runtime api eg rust

 

lambda container image — this must implement the lambda runtime api

 

ecs and fargate are preferred for running arbitrary docker images

 

 

lambda integrates with

 

api gateway
kinesis
dynamodb
s3

 

cloudfront
cloudwatch events and eventbridge

 

cloudwatch logs
sns and sqs
cognito – reacts when a user logs in eg to a database

 

lambda use case:

 

thumbnail image creation

 

new image uploaded to s3 then triggers a lambda function to generate a thumbnail of the image
this is pushed to s3 and meta data to dynamo db.

 

another example:

 

a serverless CRON job to run jobs

 

but for cron you usually need to have a server running, but with lambda you can do this without a server!

 

eg cloudwatch events or eventbridge every hour triggers a lambda function, this is instead of the cronjob!

 

pricing for lambda:

 

pay per calls first 1mill requests are free

 

then 20c per 1 mill requests

 

pay per duration in increments of 1 ms

 

400k GBseconds of compute time per month is free, charges thereafter on rising scale

 

very cheap to run lambda so it is very popular

 

 

you can run jobs using many different program languages

 

you enter your code in lambda web console and lambda then runs the code for you.

 

you can have lambda respond to events from various sources – eg data processing, streaming analytics, mobile or iot backends

 

lambda takes care of scaling for your load, you don’t have to do anything here!
ie seamless scaling

 

 

to create a lambda function you have 4 possibilities:

 

author from scratch
use a blueprint – these are pre configured functions
container image
browse serverless app repository

 

 

Lambda Limits per region
important for exam…

 

for
execution:

 

mem allocation 128 mb to 10 gb in 1mb increments

 

max exec time is 900 secs

 

env variables 4kb

 

disk capacity in the function container in /tmp is 512 mb

 

concurrency executions 1000 – can be increased

 

for
deployment:

 

function deployment size compressed .zip is 50mb

 

size of uncrompressed deployment code plus dependencies is 250mb

 

can use the /tmp to load other files at startup

 

size of env variables is 4kb

 

the exam may ask you question to see if you think lambda can be used or not acc to the requirement for the task… you need to know these above limits in order to judge suitability of lambda for the task.

 

 

Lambda@Edge

 

if you are deploying a CloudFront cdn and you want to deploy lambda globally

 

how to implement request filtering

 

you can use lambda@edge for this

 

you deploy it alongside each region in your cloudfront cdn

 

you can use lambda to modify the viewer/origin requests and responses of cloudfront:

 

this can be:

 

after cloud front receives a request – viewer request
before cloud front forwards the request to the origin – origin request

 

after cloudfront receives the response from the origin – origin response
before cloudfront forwards the response to the viewer – viewer response

 

plus,
you can also generate responses to viewers without having to send a request to the origin!

 

important to know this high level overview for exam.

 

use cases:

 

website security/privacy

 

dynamic web application at the Edge

SEO

intelligent routing across origins and data

centers
bot mitigation at the Edge

 

real-time image transformation
a/b testing
user authentication and authorization

 

user prioritization
user tracking and analytics

 

Continue Reading

AWS S3 Storage

S3 is the AWS object storage system. S3 stores objects in flat volumes or containers called buckets, rather than a hierarchical file system.

 

Buckets must have globally unique name – across ALL accounts of AWS – not just your own!

 

defined at region level – important!

 

 

bucket naming convention – must know this for exam!

 

no uppercase, no underscore,, 3-63 chars long,

 

must start with lowercase letter or number

 

must not be an ip address

 

objects:

 

are files, must have a key – the full path to thta file

 

eg

 

s3://my-bucket/my_file.txt -> my_file.txt is the key

 

you can add “folder” names but they are just a prefix ie tag not a real file directory system

 

 

so if you have

 

s3://my-bucket/my_folder/my_file.txt

 

then /my_folder/my_file.txt

is the object key for this file

 

 

object max size is 5TB but you cann only upload 5GB in one go, to upload larger objects, you have to use multi-part upload

 

metadata can be added, also tags

 

and version id system if enabled

 

you can block public access if you want for a bucket

 

you receive an ARN or amazon resource name for the bucket

 

 

2 ways to open an S3

 

in the console click on object action and open

 

or via the url public object url

 

but for this you must set the permission for access to the bucket

 

bucket must be public access

 

pre-signed url – you give the client temp credentials to open the file

 

 

you can version your files, but have to enable versioning at the bucket level. Best practice to use versioning, this protects against unintended deletes and you can roll back.

 

Any files existing before versioning is enabled will not have previous versions available.

 

if you click on objects -> list versions, you will see the available versions listed.

 

you can easily roll back in this way. So versioning should be enabled!

 

 

S3 Encryption

 

exam question!

4 methods of encryption for S3 are

 

SSE-S3 – encrypts S3 objects using keys managed by AWS

 

SSE-KMS – uses AWS key management service to manage the keys

 

SSE-C – to manage own keys yourself

 

Client-Side Encryption

 

important to know which is best suited for each situation!

 

SSE-S3

 

managed by AWS S3

 

object encrypted server side

 

uses AES-256 algorithm

 

must set header “x-amx server-side-encryption”: “AES256”

 

 

SSE-KMS

 

managed by AWS KMS

 

gives you control over users and an audit trail

 

object is encrypted server side

 

set header “x-amx server-side-encryption”: “aws:kms”

 

 

SSE-C

 

server side with your own keys outside of aws

 

so s3 does NOT store the key

 

https has to be used for this as you will be sending your key

 

the actual client side data encryption key in every http header

 

s3 then uses the key to encrypt the data on s3 at the bucket.

 

 

Client Side Encryption

 

this happens before transmitting data to s3 at the client side

 

and decryption on the client side.

 

customer fully manages the keys and encryption/decryption.

 

there is a client library called aws s3 encryption client which you can use on your clients.

 

 

encryption in transit ssl or tls

 

https is “in flight” – you always use https for your in flight: mandatory

 

uses ssl or tls certificates

 

 

 

S3 Security

 

User-based

first there is user-based

 

uses IAM poliicies, which api calls are allowed from a specific user

 

Resource-based

 

sets bucked wide rules from S3 console, allows cross account

 

 

not very common (NOT in exam)

oacl – object access control list acl – this is finer grain

 

bacl – bucket access control list acl – this is less common

 

 

IAM principal can access an S3 object if

 

user iam permissions allow it or the resource policy allows it

 

and no explicit deny exists

 

 

s3 bucket policies

 

they are json based policies

 

actions: they allow a set of api to allow or deny

 

principle is the account or user the policy applies to

 

use the s3 bucket policy to

 

grant public access to the bucket

force encryption at upload to the bucket

grant access to another account – cross account access

 

 

Bucket settings for block public access

 

used to block public access

 

3 kinds:

 

new acls
any acl
new public bucket or access point policies

 

block public or cross account access to buckets or objects through ANY public bucket or access point policies.

 

but exam will not test you on these.

 

created to prevent company data leaks

 

can also be set at account level

 

networking: supports vpc endpoints

 

S3 Logging and Audit

 

s3 access logs can be stored in other s3 buckets
api calls can be logged in CloudTrail

 

user security:

 

MFA Delete can be required to delete objects for versioned buckets

 

pre-signed urls valid for a limited time only

 

use case

 

eg to download a premium product or service eg video if user is logged in as a paid up user or has purchased the vid or service

 

 

 

S3 Websites

 

s3 can host static websites for public access

 

url will be

 

bucket-name.s3-website-aws-region.amazonaws.com

 

 

 

if you get 403 forbidden error then make sure bucket policy allows public reads – bucket must be publicly accessible for this.

 

 

CORS Cross-Origin Resource Sharing

 

web browser mechanism to allow requests to other origins while visiting the main origin

 

eg http://example.com/app1 and http://example.com/app2

 

a CORS header is needed for this – and the other origin must also allow the request.

 

 

the web-browser does a “pre-flight request” first – asking the cross origin site if it is permitted – then if yes, then eg get put delete

 

these are the cors method access-control-allow-methods

 

 

S3 CORS

 

exam question!
if a client does a cross-origin request to an “3 bucket then you must enable the correct CORS headers.

 

you can allow for a specific origin, or for * ie all origins

 

 

S3 MFA Delete

 

forces a user to generate a code from a device eg mobile phone before doing some operations on S3

 

to activate MFA-Delete enable versioning on the S3 bucket

 

it is required to permanently delete an object version

and to suspend versioning on the bucket

 

not needed to enable versioning or list deleted versions

 

 

 

only bucket owner ie root account can enable/disable MFA-Delete

 

only possible via CLI at present.

 

 

 

first create an access key for the bucket in the web console of iam

 

 

then configure the aws cli to use this key

 

download the key file, and then set up a cli with your access key id and secret access key

 

command:

 

aws configure –profile root-mfa-delete-demo

 

you are then prompted to enter the access key id and secret access key

 

then you run

 

aws s3 ls –prifle root-mfa-delete-demo to display

 

 

then do:

 

aws s3api put-bucket-versioning –bucket demo-mfa-2020 –versioning-configuration Status=Enabled, MFADelete=Enabled –mfa “<here enter the arn-of-mfa-device and-the-mfa-code-for-the-device>” –profile root-mfa-delete-demo

 

you can then test by uploading an object

 

and try deleting the version – you should get a message saying you cannot delete as mfa authentication delete is enabled for this bucket…

 

so to delete you must use the cli mfa-delete command and your chosen device eg mobile phone mfa – or alternatively for this demo just disable the mfa-delete again. then you can delete as per usual.

 

 

 

To force encryption you can set a bucket policy which refuses any api “put” calls to an object that does not have encryption headers

 

alternatively you can use the default encryption option of S3

 

important for exam: bucket policies are evaluated *before* default encryption settings!

 

 

 

S3 Access Logs

 

– you can log to another bucket, or you can use AWS Athena

first, very important – and potential exam question!

 

NEVER set your logging bucket to be the monitored bucket or one of the monitored buckets! because this will create a big infinite loop! which means a huge AWS bill!

 

always keep logging bucket and the monitored bucket/s separate! – ie set a separate different target bucket that is NOT being logged!

 

tip: make sure you define a bucket with the word “access-logs” or similar in the name, so that you can easily identify your login bucket to avoid logging it by mistake.

 

S3 Replication – Cross Region (CRR) and Single Region (SRR)

 

– must enable versioning

 

the copying is asynchronous

 

buckets can belong to different accounts

 

must grant proper iam permissions to S3 for this

 

CRR: you synchronize against different regions

 

used for compliance, lower latency access, replicating across different accounts

 

SRR: used for log aggregation, live replication between eg production and test and development accounts

 

note: after activating only the new objects get replicated. to replicate existing objects, you need to use…

S3 Batch Replication feature

 

for DELETEs: you can replicate the delete markers from source to target

but deletions with a version id are not replicated – this is to avoid malicious deletes

 

and there is no “chaining” of replication…

this means eg if bucket 1 replicates to bucket 2 and 2 replicates to bucket 3, then bucket 1 is not automatically replicated to 3 – you have to explicitly set each replication for each pair of buckets.

 

first you have to enable versioning for the bucket

 

then create your target bucket for the replication if not already in existence – can be same region for SRR or a different region for CRR

 

then select in origin bucket:

 

management -> replication rules, you create a replication rule and you set the source and destination.

 

then you need to create an iam role:

 

and specify if you want to replicate existing objects or not

 

for existing objects you must use a one time batch operation for this

 

 

S3 Pre-Signed URLs

 

can generate using sdk or cli

 

uploads: must use sdk
downloads can use cli, easy

 

valid fo 3600 secs ie 1hr default, can change

 

users are given a pre-signed url for get or put

 

use cases:
eg to only allow logged in or premium uses to download a product eg video or service
allow a user temporary right to upload a file to your bucket

 

 

S3 Storage Classes

 

need to know for exam!

S3 offers

 

Standard General Purpose, and
Standard-Infrequent-Access (IA)
One Zone-IA
Glacier Instant Retrieval
Glacier Flexible Retrieval
Glacier Deep Archive
Intelligent Tiering

 

 

can move objects between classes or use S3 Lifecycle Management service

 

Durability and Availability

 

difference:

 

S3 has very high durability, 99.9 11×9!

 

Availability

 

how readily available the service is available.

 

S3 standard is about 1 hr per year out of availability

 

standard:

 

big data, mobile, gaming, content distribution

 

IA:

 

less frequent access
lower cost than standard

 

99.9% available

 

good for DR and backups

 

1-Zone-IA

 

v high durability but 99.5% availability – not so high

 

thus best used for secondary backup copies or recreatable data

 

Glacier storage classes:

 

low-cost object storage for archiving or backup

 

you pay for storage plus a retrieval charge

 

Glacier Instant Retrieval IR

 

millisecond retrieval, min storage 90 days

 

Glacier Flexible Retrieval

 

1-5 mins to recover

 

standard 3-5 hrs

 

 

bulk 5-12 hrs – is free
min storage duration 90 days

 

Glacier Deep Archive

 

best for long-term only storage

 

12 hrs or bulk 48 hrs retrieval
lowest cost

 

min 180 days storage time

 

 

intelligent tiering

 

allows you to move objects between tiers based on monitoring
no retrieval charges

enables you to leave the moving between tiers to the system

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Continue Reading

AWS ElastiCache for Redis/Memcached

ElastiCache is an in-memory DB cache. 

 

ElastiCache supports two open-source in-memory caching engines: Memcached and Redis

 

applications query ElastiCache (EC), called a “cache hit”

 

– if not available it seeks from RDS and then stores in EC

 

this relieves load on RDS

 

cache must have an invalidation strategy set in order to ensure the most relevant data is cached -not so easy in practice

 

is used for managed Redis or 1memcached

 

is an in-memory db with v high performance and low latency,

 

reduces load for read-intensive workloads

 

helps the application operate stateless

 

AWS takes care of maintenance, upgrades, patching, backups etc.

 

BUT you have to make major application code changes to query EC!

 

EC can also be used for DB user session store

 

which avoids users having to reauthenticate and relogin to the DB

 

 

Difference between Redis and Memcached

 

REDIS

 

is similar to RDS:

 

allows for multi-az with auto-failover

 

use read replicas

 

data durability and high availability rather like RDS

 

backup and restore features

 

 

MEMCACHED

 

uses multi-node for partitioning of data, known as sharding

 

NO high availability and no replication

 

it is not a processing cache, so not persistent

 

no backup and no restore

 

has multi-threaded architecture

 

 

pure cache with no high availability or data protection for failure – a simpler option

 

Deployment Options

 

Amazon ElastiCache can use on-demand cache nodes or reserved cache nodes.

 

On-demand nodes provide cache capacity by the hour, resources are assigned when a cache node is provisioned.

 

You can terminate an on-demand node at any time. Billing is monthly for the actual hours used.

 

Reserved nodes use 1-year or 3-year contracts.

 

Hourly cost of reserved nodes is much lower than hourly cost for on-demand nodes.

 

 

 

REDIS Use Case:

 

need to know for exam!

 

Gaming Leaderboards, as these require intensive processing-

 

REDIS uses “sorted-sets”, this ensures accurate real-time data is generated for the leaderboard – important!

 

 

EC Security – important for exam

 

The EC caches do not use IAM policies, except for AWS API-level security

 

REDIS AUTH

 

you use Redis Auth to authenticate – sets a password/token when creating the redis cluster

 

on top of security groups

 

supports inflight ssl

 

 

Memcached

 

uses SASL authentication (more advanced)

 

3 kinds of patterns for EC – need for exam

 

lazy loading – all read data is put in the cache, but can become stale, but future data is only loaded when it cant find the data present in the cache, it reads from db and then copies to cache

write-through – adds or updates cahce data when written to DB – there is no stale data

 

Session Store: stores temp session data in cache using a TTL

 

 

 

DB TCP Ports

 

PostgreSQL: 5432

 

MySQL: 3306

 

Oracle RDS: 1521

 

MSSQL Server: 1433

 

MariaDB: 3306 (same as MySQL)

 

Aurora: 5432 (if PostgreSQL compatible) or 3306 (if MySQL compatible)

 

For the exam make sure to be able to differentiate an “Important Port” vs an “RDS database Port”.

 

Continue Reading

AWS Aurora

AWS Aurora is a high-performance highly available database engine for AWS.

 

Proprietary AWS tech, not open source

 

supports Postgres and MySQL db

 

is AWS cloud optimized and claims 5x performance improvement over MySQL on RDS

 

and 3x improvement on Postgres on RDS

 

Storage grows automatically in blocks of 10GB to 128TB

 

can have up to 15 replicas vs 5 mysql replicas

 

replication also much faster and failover is almost instantaneous,
failover very fast within 30 secs, and it is HA high availability native

 

self-healing with peer-to-peer replication

 

and supports cross-region replication

 

stores 6 copies of your data across 3 AZs

 

4 copies of 6 needed for writes
3 out of 6 needed for reads

 

provides a writer endpoint that points to the master db.

you can have asg auto-scaling on top (but max 15 replicas)

 

 

patching, updating etc is done by AWS

 

also provides BackTrack service – you can restore a point in time without any extra backup routine

 

important for the exam!

 

READER ENDPOINT connects automatically to ALL the read replicas, so this provides for connection level load balancing for the read replicas

 

storage is highly striped across 100s of volumes

 

security similar to RDS

 

encryption at rest using KMS

 

 

automated backups, snapshots and replicas are also encrypted

 

encryption in flight uses SSL

 

you can authenticate using IAM, same as with RDS

 

but you are responsible for protecting your instance with security groups

 

 

also important – for exam!

 

you cannot SSH into Aurora

 

 

 

 

Aurora Read Replica Auto Scaling

 

You create a writer endpoint for DB writes, while reads go to a single separate reader endpoint, which connects to multiple aurora DBs.

 

As auto scaling takes place and more read DBs are added, these are connected to the reader endpoint

 

However, you can also create a separate custom endpoint for specific other traffic purposes, eg a read analytics software which needs to connect and which generates intensive traffic load

 

or you might want to have a set of read replicas which have different instance type to the others, again, you can use custom endpoints for this, this creates an additional endpoint.

 

Aurora Serverless

 

automated db instance and auto-scaling

 

no need for capacity planning

 

use case:

 

infrequent, unpredictable workloads

 

billing is pay per second, can be more cost-effective

 

client talks to a proxy-fleet and in the backend Aurora creates the necessary instances.

 

 

Important for EXAM!

Aurora Multi-Master

 

every node is Read-Write

 

useful for immediate failover for the WRITE node – offers high availability for the writer node.

 

Global Aurora

 

you can have

 

Cross Region Read Replicas

 

useful for Disaster Recovery (DR)

 

easy to set up

 

Global Database (recommended)

 

you have one primary region for read-write

 

and up to 5 secondary read only regions with v low replication lag
plus up to 16 read replicas per secondary region

 

provides for very low latency

 

when you promote another region for DR the RTO recovery time overhead is less than 1 minute.

 

Aurora Machine Learning

 

you can add ML predictions to apps via sql

supports

 

AWS
– sagemaker
– comprehend – for sentiment analysis

 

can be used for fraud detection, ad targeting, sentiment analysis, product recommendations

 

Continue Reading

ASG Auto Scaling Groups

Auto Scaling Groups or ASGs provide a way to scale in and out with your instances and infra.

 

Scale out: add instances for increased workload

 

Scale in: remove instances for decreased workload

 

you can set a minimum, desired, and maximum capacity no of instances

 

automatically connect new instances to a load balancer

 

recreate an instance if a previous one is terminated eg if unhealthy

 

ASG is FREE! – but you pay for the underlying instances

 

ASGs also work with load balancers

 

You create a Launch Template (used to be called Launch Configuration – now deprecated)

 

in this you set:

 

AMI and instance type

 

EC2 user data script, if used
EBS volumes
security groups
ssh key pair
IAM roles for the instances
network/subnet info
lb info

 

plus also set
min,max,initial capacity
scaling policies

 

CloudWatch Alarms also integrate with ASGs.

 

you set a metric to be monitored by CW and this then triggers the ASG activity eg scale in policy or scale out policy.

 

 

 

ASG Termination Policy

by default, ASG looks for the AZ which has the most number of instances, and then deletes the one within that AZ which has the oldest launch configuration.

ASG always seeks to balance number of instances across AZs by default.

 

 

Lifecycle Hooks

 

when an instance is launched you can first determine that it goes first into a pending state, you then move it into pending proceed.. then it goes into in service state.

 

if no pending state, then it goes straight to in-service state.

 

 

also for teminating, you can set a terminating wait state, this is so you have time to carry out some other actions first.

 

 

Launch Configs – these are legacy deprecated, you have to recreate each time.
Launch Templates – new, are versionable, recommended by AWS

 

only use Launch Templates from now on!

 

 

 

 

 

 

Continue Reading

AWS Load Balancers

NOTE: health checks for EC2 instances are crucial when using  load balancers, because you do not want to send traffic to an EC2 instance or other service if it is not working properly.

 

You set up your security group for the load balancer, your endpoints eg EC2 instances should only accept traffic from the load balancer security group and not from the external internet. This is an enhanced security mechanism.

 

 

 

Types of Load Balancer in AWS

 

 

ELB Elastic Load Balancer

CLB Classic Load Balancer (deprecated)

ALB Application Load Balancer

NLB Network Load Balancer

GWLB Gateway Load Balancer

 

 

 

 

ELB Elastic Load Balancer

 

is a managed load balancer,

aws guarantees it will work, takes care of upgrades and availability

costs more than setting up your own load balancer, but is more convenient and less overhead for you

is integrated with many aws services

 

 

 

CLB Classic Load Balancer

 

 

is deprecated, don’t use for new installs

 

operates on tcp layer 4 and http/https layer 7

 

health checks are based on above

 

fixed hostname

 

ALB Application Load Balancer

 

works at layer 7 http

 

balances to multiple http servers machines ie target groups

 

also can be multiple applications on SAME machine eg via containers

 

supports websocket as well

 

and redirects from http to https

 

can route acc to target url path eg example.com/users and example.com/posts

 

also based on hostname eg

 

one.example.com and two.example.com

 

also query string or headers in the url

 

good for micro services and container-based apps eg docker and amazon ecs

 

also have port mapping feature

 

comparison with old classic lb: you would need additional clbs to do the same with one alb if you want different routing

 

 

 

NLB Network Load Balancers

 

operates at layer 4

 

forwards TCP/UDP traffic to instances

 

high volume traffic, millions of requests per sec

low latency 100ms vs 400ms for ALB

 

NLB has one static ip per AZ, supports Elastic IP

 

Useful for having 2 incoming points for traffic to your network

 

use case:

 

when you need extreme performance or tcp udp traffic

 

Note: NLB is NOT in the free-tier pricing!

 

 

GWLB  Gateway Load Balancer 

 

esp used for firewalls, intrusion detection, prevention system (IDS/IDPS), deep packet inspection systems etc

 

can also be used to manage a fleet of 3rd party network virtual appliances running on aws

 

operates at layer 3 network layer ip packets

 

has 2 functions:

 

1. transparent network gateway – a single point of entry/exit for traffic

 

2. load balancer to distribute traffic to your virtual appliances

 

exam tip:
GENEVE protocol port 6081 is the gateway load balancer

 

EC2s must be private addresses for GWLB

 

 

Sticky Sessions or Session Affinity

 

this means the same client is always connected to the same instance behind a load balancer to complete a transaction

 

this works for CLBs and ALBs

 

uses a cookie with an expiry date.

 

this is to ensure a user does not lose his session data

 

but – it can cause an imbalance within the balanced load cluster

 

 

types:

 

application-based cookie

– custom cookie, is generated by the target, can include any attribute

 

– application cookie – generated by load balancer, cookie name is AWSALBAPP

 

but some names are reserved: AWSALB, AWSALBAPP AWSALBTG

 

duration-based cookie

 

– generated by load balancer
cookie name is AWSALB for ALB and AWSELB for CLB

 

 

 

 

Cross-Zone Load Balancing

 

a point to note about cross-zone load balancing…

 

if this feature is  ON, then it will ensure each INSTANCE gets the equal amount of share of traffic as all other instances.

 

but if this feature is OFF , then it will vary between the instances depending on how many instances in each AZ, if this is unequal eg one AZ has fewer EC2s than others, then it will be unequally divided among the actual EC2s although equally shared out at the AZ 1lb level.

 

Be aware:

 

CZ-LB is enabled by default for ALB – and cannot be disable –  but for NLB it is disabled by default – but you pay extra if you want to enable it. 

 

but for CLB: it is disabled, but you can enable, and it is free to enable

 

 

 

SSL/TLS and AWS Load Balancers

 

encrypts via “in-flight” in-transit encryption

 

SSL: secure sockets layer

TLS: transport layer security, the newer ssl version

 

public SSL certificates are issued by certificate authorities (CAs)

 

eg Globalsign, Digicert, GoDaddy etc

 

have an expiry date, must be renewed

 

Load Balancer uses an X.509 SSL certificates, can be managed via ACM – the AWS certificate manager

 

you can create your own certificate

 

clients can also use SNI server name indication – client must declare which hostname it wants in the SSL handshake. Server then finds the correct SSL certificate or else returns the default one.

 

 

 

SNI Server Name Indication for SSL

 

solves problem of loading multiple SSL certificates onto one webserver to serve multiple websites.

 

only works with alb and nlb and cloudfront, not with clb

 

 

Elastic load balancer elb only supports

 

CLB – only 1 SSL certificate

 

must use multiple clbs for more than one certificate

 

ALB and NLB 

 

supports multiple SSL certificates and uses sni to make it work

 

 

Connection Draining and load balancers

 

CLB call it connection draining
ALB and NLB: call it deregistration delay

 

it allows some time for instances to complete in-flight SSL requests while instance is unhealthy or de-registering

 

it stops lb sending requests to the instance during this period

 

you can set a period of between 1 and 3600 secs, default is 300 secs, or disable, by setting to 0.

 

set a low value if requests are short

 

if there tend to be longer requests, eg for uploads, downloads etc… then set a higher value.

 

Continue Reading

AWS Snow Family

AWS Snow Family

 

for data migration in and out of AWS:

 

Snowcone

 

Snowball Edge

 

Snowmobile

 

for Edge computing data migration:

 

Snowcone

Snowball Edge

 

These are offline devices that perform data migration for migration that would take more than a week by network transfer.

 

Snowcone and Snowball Edge are sent by post using a physical route rather than digitally.

 

Snowball Edge, used for TB or PB data transfer

 

pay per data transfer job

 

 

2 versions

 

Snowball Edge Storage Optimized:

 

80TB max of HDD capacity for block volume and S3 compatible object storage

 

Snowball Edge Compute Optimized:

 

42TB of HDD capacity for block volume and S3 compatible object storage

 

 

use case for Edge:

 

large data cloud migration, Disaster recovery prep, disaster recovery action

 

For Snowcone:

 

a much smaller device, can withstand harsh environments, very light and secure

 

used for smaller size transfers, up to 8TB storage

 

10 times less than snowball edge.

 

you must provide own battery and cables

 

can be sent to AWS in post or via internet – and use AWS DataSync to send data.

 

 

Snowmobile is a truck – very high-scale transfers.

 

request snowball devices on the AWS website

 

install a snowball client on servers

 

connect the snowball to your servers, then copy the file,

 

then ship the device back

 

many EBs of data 1EB = 1PB

 

very high security, staffed,

 

best for more than 10PB data transfer

 

Edge Computing:

 

somewhere creating data or needing data but has no internet access or limited access…

 

snowball edge or snowcone can be embedded into your edge site

so it gives you a way to transfer data despite having no strong internet connection

 

The Snowcone and Edge can run EC2 instances and Lambda functions!

good for preprocessing data, transcoding media screens, machine learning at the edge

 

Snowcone 2 CPUs 4GB RAM, wired/wi-fi, powered by usb-c or optional battery

 

Snowcone Edge:

many more vCPUs up to 52vCPUs and 208 GiB RAM
optional GPU
object storage clustering available

 

can get discount pricing 1 or 3 years

 

 

Also AWS OpsHub – management software for snow family

 

This enables you to manage the device from your client pc or laptop, you install on your client

 

 

 

 

Continue Reading

AWS CloudFront

CloudFront is the AWS CDN Content Delivery Service

 

CF offers:

 

ddos protection against webserver attacks
web application firewalls WAF and Shield
improved web content read service through edge region caching (currently 216 edge locations globally, number steadily increasing) ie content caching

 

can use both https for external traffic and also forward https traffic internally – latter not usually possible due to TLS certificate limitations

 

can be used for:

 

s3 buckets CF origin access identity or CF OAI

 

– this is an IAM role used for CloudFront connections to S3.

 

 

two ways to upload content to S3:

 

first,

you can  use CF to upload files to S3 – ie ingress traffic

 

secondly,
other option is to use Custom Origin http

 

can use ALB
can use EC2 Instance
can use S3 website – must enable the bucket as a static web instance
any http backend eg on premises webserver

 

two different architectures are possible acc to whether you have an ALB or not with CF.

 

if EC2 is the origin, (origin means your website content original site) then the security group for the EC2 must allow public access

 

ie the edge location sites to access the EC2 instance

 

if an ALB is attached, then the ALB security group MUST be public – but the EC2 instance can in this case be private access, ie need not give public access. in this case your ALB is your “origin”. 

 

 

these are two different architectures – be sure to understand for the exam!

 

 

 

the edge location always serves cached content originally from your S3 site ie your “origin”.

 

 

so, web users send request to your CDN CF IP at the edge location – if the content is not already present on the CF Edge Location then it forwards the request to your S3 bucket and sends the result back to the edge location of your CF ip… 

 

 

it traverses the security group, so the ip of the ec2 instances ie the origin, must be public and allow public ip of the edge location

 

 

 

CF Geo Restriction

 

You can whitelist or blacklist users from specific countries, eg for copyright etc reasons.

 

What is the difference between CF and S3 Cross Region Replication?

 

CF:

 

a global edge network

caches for a short TTL
good for static content

 

S3 Cross Region Replication:

 

must be set up for each region you want to replicate

 

files updated in near real-time

 

read only

 

good for dynamic content that needs to be available at low-latency in a few regions

 

 

Continue Reading

AWS EBS & EFS Elastic Block Storage & Elastic File System

EBS is a network drive you can attach to ONLY ONE INSTANCE at any one time

 

 

important: EBS is not a network file system!

is bound to an AZ

 

think of it like a network USB stick

 

30GB per month free of type SSD or magnetic GP2 or GP3 volume

 

EBS can have some latency on the AWS network

 

can be detached from one EC2 and attached to another, eg for failovers, big advantage!

 

but cannot be moved to other AZs, unless you do a snapshot and then move.

 

have to provision in advance and the level of IOPS you want

 

you pay according to this after the first 30GB.

 

 

BUT you can have 2 EBS volumes attached to an EC2 – that is not limited.

 

however they are bound to an AZ

 

they can be attached on demand, do not have to be actively attached.

 

 

EBS Delete on Termination attribute – is enabled by default for EBS root volume

 

but not for other EBS volumes as it is disabled for the latter by default

 

you can change this…
advantage: to preserve root volume when EC2 instance is terminated.

 

You can transfer an EBS volume to another region or az by means of using snapshots.

 

You can move snapshots to archive which is much cheaper.

 

Snapshots are usually gone once deleted, but if you have recycle bin enabled then you can retrieve them for a limited time period according to retention rule you set.

 

 

EBS Volume Types

 

6 types:

 

gp2 /gp3 SSD general purpose balances price/performance

 

io1/io2 SSD high performance for mission critical low latency/high throughput workloads

 

st1 HDD low cost HDD for frequently accessed throughput intensive workloads

 

sc1 HDD lowest cost HDD for less frequently accessed workloads

 

EBS volumes are by size, throughput, iops

 

NOTE: only gp2/gp3 and io1/op2 can be used as boot volumes

 

 

General Purpose SSD

 

cost-effective storage

 

system boot vols, virtual desktops, dev and test env

 

1GiB – 16TiB

 

gp3

 

3000 iops and 125 miB/s

 

can go up to 16k iops and 1000 MiB/s

 

gp2

 

small vols can do burst iops to 3000

 

size of volume and iops are linked max iops is 16k

 

3 iops per GB means at 5,334 GB we have max iops

 

 

Provisioned iops ssd

 

for critical biz apps with sustained iops performance

or apps that need more than 16k iops

 

good for db workloads

 

io1/io2 4GiB – 16TiB

 

max piops 64k for nitro ec2 and 32k for others

 

can increase piops indep of storage size

 

io2 gives more durability and more iops per gib

 

io2 block express 4gib – 64tib

 

very low latency
max piops 256k

 

 

with hdd:

 

cannot be a boot vol

125 mib to 16tib

 

throughput optimized hdd st1:

 

big data, data warehousing and log processing

 

max througput is 500 mib/s max iops 500

 

cold hdd sc1

 

for infreq accessed data
where low cost is important
max throughput is 250 mib/s max iops 250

 

 

 

you *dont* need to know these details for the exam, but be aware of the main difference in the variations

 

 

 

EBS Multi-Attach – for io1/io2 family

 

attaches same ebs volume to multiple instances in same az at same time

 

each instance has full r/w permissions to the vol

 

use case:

 

for high app availability in clustered linux apps eg teradata

 

apps must be able to manage concurrent write ops

 

only thus for very specific workloads,

 

you must use a cluster aware file system ie NOT ext4 xfs etc.

 

 

EFS Elastic File System

 

is a managed NFS-type system that can be mounted on multiple EC2s in multiple AZs.

 

highly available, scalable, but expensive, you pay per usage per gigabyte

 

use cases:

 

content management,

 

web and data sharing, wordpress

 

uses nfs4.1

 

uses security groups to control access

 

important!

compatible only with linuxbased ami’s and not windows!

 

can enable enctryption at rest using kms

 

is a posix linux system with standard file api

 

scales automatically by itself!

 

exam question:

 

efs performance and storage classes:

 

efs scaling:

 

can support 1000s of concurrent nfs clients 10gb throughput

 

can grow to petabyle size automatically

 

performance mode

 

– set at efs creation time
general purpose default – latency sensive use cases such as webserver, cms etc

 

to maximum i/o — gives higher latency, highly paralleluse for this io1 : this is best for big data applications, eg media processing etc

 

throughput mode
— bursting 1tb = 50mib/s and up to 100 mib/s

 

provisioned: set your throughput regardless of storage size eg 1 1gib/s per 1tb storage

 

 

EFS Storage Classes

 

you can set up storage tiers for lifecycle management

 

eg move to another tier after N days…

 

 

 

– standard tier – used for frequently accessed files
– infrequent access tier (IA) -efs-ia: costs to retrieve files

 

but lower price to store, enable efs-ia by means of a lifecycle policy

 

 

Availability and Durability of EFS

 

2 options:

 

standard: can set up EFS to be multi AZ
one-zone: you use one AZ only, backups are default enabled, compatible with IA

 

90% cost saving

important:
exam will ask you which tier /storage class you should use for which use case, and you need to be aware of the cost implications!

 

 

 

Differences Between EBS & EFS

 

must know for exam!

 

EBS

 

can only be attached to only one instance at a time
are locked into an AZ

 

gp2. io increases if disk size increase
io1: can increase the io independently.

 

to migrate an EBS across AZs

 

– take a snapshot
– restore the snapshot to the other desired AZ

 

note that ebs backups use up io and so you should not run them when your app has heavy traffic overhead

 

also, root ebs volumes get terminated by default if or when the ec2 instance gets terminated! very important to be aware of this
– but you can disable this

 

 

EFS by comparison:

 

can mount on 100
s of instances across AZs!

 

it is multi-AZ, multi-client/instance

 

can be used to share data

 

only available for linux posix, not windows!

 

 

efs more expensive than EBS but can use efs-ia to reduce costs

 

 

so: efs is more for multi instances

 

ebs is more for one instance

 

 

and

 

Instance Store

 

instance store: is an ephemeral local instance drive just for an instance – you lose it with the instance when the instance is deleted.

 

 

 

 

Continue Reading

AWS Route 53

Route 53 is the AWS DNS service.

 

Highly available
scalable
fully-managed
authoritative DNS – you the customer can update the dns records
is also a domain registrar

 

AWS provides 100% SLA guarantee availability

 

You define how you route traffic to a specific domain.

 

domain name
record type eg A or AAAA
value ie ip number
routing policy
TTl – time to live for the record caching time

 

different record types

 

A, AAA, CNAME, NS – essential to know

 

A – maps hostname to ipv4 address
AAAA – maps hostname to ipv6 address
CNAME – maps hostname to another hostname
you cannot create CNAMES for top record of domain eg example.com but you can for eg www.example.com

 

NS – the name servers for the hosted zone

 

Hosted Zones

 

are a container for dns records

 

 

public hosted zones

for internet available ips. Any client can request

 

private hosted zones

for not publicly available ips, within VPCs which can only be accessed within the subnet
this enables you to make ips resolvable within the private network ie internally, not publicly via internet.

 

 

otherwise they work the same way – public hosted and private hosted.

 

you pay 50c per month per hosted zone

 

 

from your CLI you can then check your domain registration and ip records with
nslookup or dig

 

do apt install bind-utils -y to install them if not installed yet on the machine.

 

dig <your domain name and machine>

 

nslookup <your domain name and machine>

 

 

 

TTL Time To Live

 

 

set in seconds

 

TTL: client will cache a lookup for the set TTL time period, this is to relieve DNS server from too much request and response traffic

 

high TTL:

 

less traffic, lower costs
but possibly not up-to-date records

 

low TTL:

 

more traffic, more costly
records more likely to be fully accurate

 

 

CNAME vs Alias

 

you need to use Alias for mapping to an alias hostname, not CNAME!

 

sometimes you may want to map a hostname to another hostname

 

CNAME does this, but only for non-root domains ie eg www.example.com not example.cmm
The CNAME maps to the root domain name only!

 

Alias: this works for both root and non-root domains
always either A for ipv4 or AAAA for ipv6

 

you cannot set the TTL for Alias, this is set by Route 53 automatically

 

you can use as aliases things like

 

elastic load balancers ELB
CloudFront
API Gateways
Elastic Beanstalk
S3 websites
VPC interface endpoints
Global accelerator
Route 53 record IN THE SAME HOSTED ZONE

 

Important: You *cannot* set an ALIAS for an EC2 DNS name!

 

 

 

Route 53 routing policies

 

simple
weighted
failover
latency-based
geolocation
multi-value
geoproximity

 

you set the routing policy in the Route 53 Dashboard for the dns record

 

 

simple policy

 

you can specify multiple records in the same record but then a random one is chosen by the client

 

can’t be associated with health checks

 

 

weighted policy

 

you set a % of requests to go to each resource you specify

 

eg to different EC2 instances

 

to do this you assign each record a relative weight

 

the weights don’t need to add up to 100

 

but the DNS records involved must have same name and type

CAN be associated with Health Checks

use cases: load balancing between regions, testing new application versions

 

NOTE if you assign a weight of 0 to a record then the resource will not receive any traffic!

 

Also, if ALL records have a weight of 0 then all records will be equal! ie balanced responses

 

 

 

Latency-based

 

you want to redirect to the resource with the least latency, ie closest to us in terms of SPEED of internet

latency based on traffic between users and AWS Regions

 

so depends on traffic speed, not necessarily same as geographical closeness

 

Can use Health Checks

 

 

 

 

Health Checks

 

HTTP Health Checks are only for PUBLIC resources

 

If one region is down, then we can use a Health Check in Route 53.

 

These provide for automated DNS failover

 

the check can be against:

an endpoint eg app server

an other health check eg calculated health checks

 

cloudwatch alarms eg for dynamodb, rds

 

To pass an HTTP health check the endpoint must respond with 2xx or 3xx status codes

 

you can combine up to 256 health checks into a single health check using OR, AND, or NOT 

 

and define how many must pass 

 

How to perform health checks for private hosted zones

use a CloudWatch Metric and Alarm then create a Health Check that monitors the alarm!

 

 

 

 

Failover Policy (Active-Passive Failover)

 

you associate your DNS record with a health check – essential for this

 

but you can only have one primary and one secondary record

 

so, you need to first set up your health check for each machine,

 

then you reference them in your dns records

 

policy: set to failover, and the type: primary or secondary

 

and then associate the respective health check you have defined.

 

then do the same for the secondary record and instance.

 

 

 

Geolocation

 

this is where user is physically based

 

use cases:

 

website localization
restrict content distribution
simple load balancing method

 

Geoproximity

 

enables you to specify “bias values” for specific geo regions

 

1- 99: more traffic to the resource
-1 to -99: less traffic to the resource

 

can be for AWS resources, specifying aws-region or non-AWS resources , specified by latitude/longitude

 

exam tip:

 

this can be useful when you need to shift traffic from one region to another

 

 

 

Multi-Value Policy

 

multi-value or multi-value answer is similar to an ELB but it is a client-side load balancer in effect.

 

used to route traffic to multiple resources but the client chooses which to use

 

can associate with Health Checks – up to 8 checks for each multi-value query

 

NOT a substitute though for an ELB!

 

 

 

Route 63 Traffic Policies

 

You can use these to define your DNS policy.

 

 

These make it easier to set policies.

 

Continue Reading