Tags Archives: monitoring

AWS Additional Monitoring Tools

 

AWS Config

 

 

AWS Config is an AWS fully managed change management solution within AWS. It allows you to track the change history of individual resources and configure notifications when a resource changes.

 

This is achieved by means of config rules. A config rule represents the desired state that the resource should be in.

 

Config rules allow you to monitor for systems that fall outside of your set baselines and identify which changes caused the system to fall out of compliance with the baseline. AWS Config is enabled on a per-region basis, so you need to enable it for every region in which you want to use it.

 

Bear in mind that AWS Config is a monitoring tool and does not actually enforce baselines, nor does it prevent a user from making changes that cause a resource to move out of compliance.

 

AWS Config enables you to capture the configuration history for your AWS resources, maintain a resource inventory, audit and evaluate changes
in resource configuration, and enable security and governance by integrating notifications with these changes. You can use it to discover AWS resources in your account, continuously monitor resource configuration against desired resource configuration, and check the configuration details for a resource at a given point in time.

 

AWS Config is used to assess compliance as according to your set internal guidelines for maintaining resource configurations, as well as enabling compliance auditing, security analysis, resource change tracking, and assisting with operational troubleshooting.

 

AWS Trusted Advisor

 

AWS Trusted Advisor service analyzes and checks your AWS environment in real-time and gives recommendations for the following four areas:

 

Cost optimization
Performance
Security
Fault tolerance

 

Trusted Advisor or TA integrates with AWS IAM so you can control access to checks as well as to categories.

 

The current status of these checks is displayed in the TA dashboard as follows:

 

Red: Action recommended
Yellow: Investigation recommended
Green: No problem detected

 

Where the colour is red or yellow, TA provides alert criteria, recommended actions, and relevant resource details, such as details of the
security groups allowing unrestricted access via specific ports.

 

Six core checks are available for all AWS customers free of charge.

 

Five checks for security plus one check for performance:

 

eg:
service limits
IAM use
security groups-unrestricted ports
MFA on root account
Elastic block storage public snapshot
RDS public snapshot.

 

 

 

AWS Inspector

 

AWS Inspector provides for the automation of security assessments. The assessments can be set to run on a schedule or when an event occurs that is monitored by Amazon CloudWatch, or also via an API call. The dashboard shows the assessments, as well as the findings from the various scans that have run.

 

Amazon Inspector makes use of assessment templates that define which sets of rules you want to run against your environment.

 

Two types of assessments are offered by AWS Inspector: network assessments and host assessments.

 

Network assessments don’t require any agent to be installed. However if you want detailed information about processes running on a specific port then you need to install the AWS Inspector Agent.

 

Host assessments however require the Inspector Agent to be installed. These assessments are far more detailed and scan for things such as vulnerable versions of software, violations of security best practices, and areas that should be system hardened. You can select these assessments set up AWS Inspector.

 

You create an assessment template in Inspector which you then use to assess your environment by means of an Assessment Run which will then report on its findings.

 

Templates contain one or more rules packages. A rules package defines what you are checking for. Note that you can’t create custom rules packages; you can use only the rules packages provided by AWS. Currently, these are the rules packages available, listed by assessment type:

Network assessments

Network Reachability: This rules package checks your environment’s network configurations, including your security groups, network access control lists (NACLs), route tables, subnets, virtual private cloud (VPC), VPC peering, AWS Direct Connect and virtual private gateways (VPGs), Internet gateways (IGW), EC2 instances, elastic load balancers (ELBs), and elastic network interfaces (ENIs).

 

Host assessments

Common Vulnerabilities and Exposures (CVE): This rules package checks your systems to see if they are vulnerable to any of the CVEs reported.

 

Center for Internet Security (CIS) Benchmarks: This rules package assesses your systems against CIS benchmarks specific to your OS.

 

There are Level 1 and Level 2 checks. Level 1 is usually safe to implement; Level 2 is more risky as the settings in Level 2 may have unintended side effects. Level 2 is usually used in environments where a very high level of security is required.

 

Security Best Practices: This rules package assesses how well your environment confirms to security best practices. Eg, it will check that a Linux EC2 instance cannot be logged into via SSH.

 

Runtime Behavior Analysis: This rules package identifies risky behaviors on your systems, such as using insecure protocols for connecting or open ports that are not in use.

 

 

 

AWS GuardDuty

 

GuardDuty is the AWS intrusion detection system (IDS) or intrusion prevention system (IPS). It uses threat intelligence feeds and analyzes logs from multiple sources, such as VPC flow logs, AWS CloudTrail event logs, and DNS logs.

 

GuardDuty can alert you to suspicious activity that could indicate potential issues such as leaked user account credentials, privilege escalation attacks, and possible command-and-control type activities.

 

GuardDuty scans specifically for three types of activity:

 

Reconnaissance
Instance compromise
Account compromise

 

Reconnaissance is the first step of an attack and was defined in the “Cyber Kill Chain”, developed by Lockheed Martin. During the reconnaissance
phase, an attacker is learning about your environment through actions such as vulnerability scans to probe for IP addresses, hostnames, open ports, and misconfigured protocols.

 

GuardDuty can detect also utilize threat intelligence feeds to detect IP addresses known to be malicious. You can use findings reported by GuardDuty to automatically remediate the vulnerability become it develops into a security violation.

 

The next type of activity is instance compromise. This consists of several indicators that may be present, such as malware command and control, crypto miners, unusual traffic levels or unusual network protocols, or communication with a known malicious IP.

 

 

Continue Reading

AWS CloudWatch Monitoring Overview

 

AWS CloudWatch is the basic AWS monitoring service that collects metrics on your resources in AWS, including your applications, in real time.

 

You can also collect and monitor log files with AWS CloudWatch. You can set alarms for metrics in CloudWatch to continuously monitor performance, utilization, health, and other parameters of your AWS resources and take action when metrics cross set thresholds.

 

CloudWatch is a global AWS service, so it can monitor resources and services across all AWS regions via a single dashboard.

 

 

CloudWatch provides basic monitoring free of charge at 5-minute intervals as a serverless AWS service, thus there is no need to install any additional software to use it.

 

 

For an additional charge, you can set detailed monitoring that provides data at 1-minute intervals.

 

 

AWS CloudWatch has a feature that allows you to publish and retain custom metrics for a 1-second or 1-minute duration for your application, services, and resources, known as high-resolution custom metrics.

 

CloudWatch stores metrics data for 15 months, so even after terminatíng an EC2 instance or deleting an ELB, you can still retrieve historical metrics for these resources.

 

 

How CloudWatch Works

 

CW Monitoring Is Event-Driven

 

All monitoring in AWS is event-driven. An event is “something that happens in AWS and is captured.”

 

For example, when a new EBS volume is created, the createVolume event is triggered, with a result of either available or failed. This event and its result are sent to CloudWatch.

 

You can create a maximum of 5000 alarms in every region in your AWS account.

 

You can create alarms for functions such as starting, stopping, terminating, or recovering an EC2 instance, or when an instance is experiencing a service issue.

 

Monitoring Is Customizable

 

You can define custom metrics easily. A custom metric behaves just like a predefined one and can then be analyzed and interpreted in the same way as standard metrics.

One important limitation of CloudWatch – exam question! 

 

CloudWatch functions below the AWS Hypervisor, which means it functions below the virtualization layer of AWS.

 

This means it can report on things like CPU usage and disk I/O…but it cannot see beyond what is happening *above* that layer.

 

This means CloudWatch CANNOT tell you what tasks or application processes are affecting performance. Remember this point!

 

Thus it cannot tell you about disk usage, unless you write code that checks disk usage and send that as a custom metric to CloudWatch.

 

This is an important aspect that can appear in the exam. You might be asked if CloudWatch can report on memory or disk usage by default; it cannot.

 

Monitoring Drives Action

 

The final piece of the AWS monitoring puzzle is alarms – this is what occurs after a metric has reported a value or result outside a set “everything is okay” threshold.

 

When this happens, an alarm is triggered. Note that an alarm is not necessarily the same as “something is wrong”; an alarm is merely a notification that something has happened at a particular point.

 

For example, it could be running some code in Lambda, or sending a message to an Auto Scaling group telling it to scale in, or sending an email via the AWS SNS message service.

 

Think of alarms as saving you from having to sit monitoring the CloudWatch dashboard 24×7.

 

One of your tasks as SysOp is to define these alarms.

 

 

CloudWatch Is Metric- and Event-Based

 

Know the difference between metrics and events.

An event is predefined and is something that happens, such as bytes coming into a network interface.

 

The metric is a measure of that event eg how many bytes are received in a given period of time.

 

Events and metrics are related, but they are not the same thing.

 

CloudWatch Events Are Lower Level

 

An event is something that happens, usually a metric changing or reporting to CloudWatch, but at a system level.

 

An event can then trigger further action, just as an alarm can.

 

Events are typically reported constantly from low-level AWS resources to CloudWatch.

 

CloudWatch Events Have Three Components

 

CloudWatch Events have three key components: events, rules, and targets.

 

An event:

 

the thing being reported. Events describe change in your AWS resources. They can be thought of as event logs for services, applications and resources.

 

A rule:

 

 

an expression that matches incoming events. If the rule matches an event, then the event is forwarded to a target for processing.

 

 

A target:

 

 

is another AWS component, for example, a piece of Lambda code, or an Auto Scaling group, or an email or SNS/SQS message that is sent out.

 

 

Both alarms and events are important and it is essential to monitor both.

 

CloudWatch Namespaces

 

A CloudWatch Namespace is a container for a collection of related CloudWatch metrics. This provides for a way to group metrics together for easier understanding and recognition.

AWS provides a number of predefined namespaces, which all begin with AWS/[service].

 

Eg, AWS/EC2/CPUUtilization is CPU utilization for an EC2 instance,

 

 

AWS/DynamoDB/CPUUtilization is the same metric but for DynamoDB.

 

 

You can add your own custom metrics to existing AWS namespaces, or else create your own custom namespaces in CloudWatch.

 

 

exam question:

CloudWatch can accept metric data from 2 weeks earlier and 2 hours into the future but make sure your EC2 instance time is set accurately for this to work correctly!

 

 

Monitoring EC2 Instances

 

CloudWatch provides some important often-encountered metrics for EC2.

 

Here are some of the most common EC2 metrics which you should be familiar with for the exam:

 

 

CPUUtilization – one of the fundamental EC2 instance metrics. It shows the percentage of allocated compute units currently in use.

 

DiskReadOps – reports a count of completed read operations from all instance store volumes.

 

DiskWriteOps – opposite of DiskReadOps, reports a count of completed read operations from all instance store volumes.

 

DiskReadBytes – reports the bytes read from all available instance store volumes.

 

DiskWriteBytes – reports the total of all bytes written to instance store volumes.

 

NetworkIn – total bytes received by all network interfaces.

 

NetworkOut – total bytes sent out across all network interfaces on the instance.

 

NetworkPacketsIn – total number of packets received by all network interfaces on the instance (available only for basic monitoring).

 

NetworkPacketsOut – number of packets sent out across all network interfaces on the instance. Also available only for basic monitoring.

 

 

 

S3 Metrics

 

There are many S3 metrics, but these are the most common ones you should know:

BucketSizeBytes – shows the daily storage of your buckets as bytes.

NumberOfObjects – the total number of objects stored in a bucket, across all storage classes.

 

AllRequests – the total number of all HTTP requests made to a bucket.

 

GetRequests – total number of GET requests to a bucket. There are also similar metrics for other requests: PutRequests , DeleteRequests , HeadRequests , PostRequests , and SelectRequests.

 

BytesDownloaded – total bytes downloaded for requests to a bucket.

 

BytesUploaded – total bytes uploaded to a bucket. These are the bytes that contain a request body.

 

FirstByteLatency – per-request time for a completed request, by first-byte millisecond.

 

TotalRequestLatency – the elapsed time in milliseconds from the first to the last byte of a request.

 

 

 

CloudWatch Alarms

 

 

Alarms Indicate a Notifiable Change

 

 

A CloudWatch alarm initiates action. You can set an alarm for when a metric is reported with a value outside of a set level.

 

Eg, for when your EC2 instance CPU utilization reaches 85 percent.

 

 

Alarms have three possible states at any given point in time:

 

OK : means the metric lies within the defined threshold.

ALARM : means the metric is below or above the defined threshold.

 

INSUFFICIENT_DATA : can have a number of reasons. The most common reasons are that the alarm has only just started or been created, that the metric it is monitoring is not available for some reason, or there is not enough data at this time to determine whether the alarm is OK or in ALARM state.

 

 

CloudWatch Logs

 

CloudWatch Logs stores logs from AWS systems and resources and can also handle the logs for on-premises systems provided they have the Amazon Unified CloudWatch Agent installed.

 

If you are monitoring AWS CloudTrail activity through CloudWatch, then that activity is sent to CloudWatch Logs.

 

If you need a long retention period for your logs, then CloudWatch Logs can also do this.

 

By default logs are kept forever and never expire. But you can adjust this based on your own retention policies.

You can choose to keep logs for only a single day or go up to 10 years.

 

Log Groups and Log Streams

 

You can group logs together that serve a similar purpose or from a similar resource type. For
example, EC2 instances that handle web traffic.

 

 

Log streams refer to data from instances within applications or log files or containers.4

 

 

CloudWatch Logs can send logs to S3, Kinesis Data Streams and Kinesis Data Firehose, Lambda and ElasticSearch

 

 

CloudWatch Logs – sources can be:

 

SDKs,

CloudWatch Logs Agent,

CloudWatch Unified Agent

Elastic Beanstalk

ECS – Elastic Container Service

Lambda function logs

VPC Flow Logs – these are VPC specific

API Gateway

CloudTrail based on filters

Route53 – logs DNS queries

 

 

Define Metric Filters and Insights for CloudWatch Logs

You can apply a filter expression eg to look for a specific IP in a log or the number of occurrences of “ERROR” in the log

Metric filters can be used to trigger CloudWatch Alarms

 

CloudWatch Logs Insights can be used to query logs and add queries to CloudWatch Dashboards

 

 

CloudWatch Logs – Exporting to S3

 

 

NOTE

this can take 12 hours for the data to become available for export – so it is not real time. For this you should use Log Subscriptions.

 

 

The API call for this is “CreateExportTask”

 

 

 

CloudWatch Log Subscriptions

 

You apply a “subscription filter” to the CloudWatch Log before sending it to eg a Lambda function managed by AWS/or to a custom-designed Lambda function and then from there as real-time data on to eg ElasticSearch. Or, you might send it from Subscription Filter and then to Kinesis.

 

 

You can also send or aggregate logs from different accounts and different regions to a subscription filter in each region and from there to a common single Kinesis Data Stream and Firehose and from there in near-real time on to eg S3.

 

 

 

 

 

 

 

Unified CloudWatch Agent

 

 

The AWS Unified CloudWatch Agent provides more detailed information than the standard free CloudWatch service.

 

You can also use it to gather logs from your on-premises servers in the case of a hybrid environment and then centrally manage and store them from within the CloudWatch console.

 

The agent is available for Windows and Linux operating systems.

 

When installed on a Windows machine, you can forward in-depth information to CloudWatch from the Windows Performance Monitor, which is built into the Windows operating system.

 

When CloudWatch is installed on a Linux system, you can receive more in-depth metrics about CPU, memory, network, processes, and swap memory usage. You can also gather custom logs from applications installed on servers.

 

To install the CloudWatch agent, you need to set up the configuration file.

 

 

Continue Reading

Network Speed Testing Using iperf3

iperf is a command line utility which provides info about bandwidth, network delays and datagram loss. It can test both TCP and UDP throughput speeds.

 

To perform an iperf test you establish both a server machine as one end-point, which discards the test traffic, and a client machine, which generates test traffic.

 

 

iperf must be installed on both machines. For Debian/Ubuntu:

 

(in this case we are installing version iperf3)

 

apt update

 

apt install iperf3 -y

 

Re Firewalling

 

You must open TCP port 5001 on the server machine.

 

Using Ubuntu/Debian Linux you can do:

 

ufw allow from 10.147.18.0/24 to 10.147.18.0/24 port 5001 proto tcp

 

On CentOS/RHEL/Fedora:

 

firewall-cmd –zone=public –add-port=5001/tcp –permanent

 

 

For the purpose of the test I simply briefly disabled fwall on gemini.

 

I found I only needed to temporarily disable ufw on gemini not on the client.

 

Or you can do:

 

root@gemini:/home/kevin# ufw allow 5201
Rule added
Rule added (v6)
root@gemini:/home/kevin#

 

 

You can also change the port by passing the -p option (e.g. in this case to open and use TCP port 2456):

 

iperf3 -s -p 2456

 

 

Then start an iperf server on the server machine using iperf3 server mode:

iperf3 -s

 

so we do:

 

root@gemini:/home/kevin# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

 

 

Next, start an iperf server on the client machine in iperf3 client mode:

 

iperf3 -c {ip-address-of-server}
iperf3 -c {ip-address-of-server} -p {tcp-port}

 

in our case, we want to test our VPN, so we enter the geminivpn IP, not the gemini external internet IP!

 

so we do:

 

iperf3 -c 10.147.18.185

 

the result:

 

root@gemini:/home/kevin# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.147.18.65, port 49458
[ 5] local 10.147.18.185 port 5201 connected to 10.147.18.65 port 49460
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 953 KBytes 7.80 Mbits/sec 
[ 5] 1.00-2.00 sec 1.18 MBytes 9.87 Mbits/sec 
[ 5] 2.00-3.00 sec 971 KBytes 7.96 Mbits/sec 
[ 5] 3.00-4.00 sec 1.13 MBytes 9.48 Mbits/sec 
[ 5] 4.00-5.00 sec 1.15 MBytes 9.61 Mbits/sec 
[ 5] 5.00-6.00 sec 1014 KBytes 8.31 Mbits/sec 
[ 5] 6.00-7.00 sec 1.10 MBytes 9.25 Mbits/sec 
[ 5] 7.00-8.00 sec 1.14 MBytes 9.59 Mbits/sec 
[ 5] 8.00-9.00 sec 1.14 MBytes 9.54 Mbits/sec 
[ 5] 9.00-10.00 sec 1.06 MBytes 8.88 Mbits/sec 
[ 5] 10.00-10.03 sec 32.2 KBytes 8.91 Mbits/sec 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.03 sec 10.8 MBytes 9.03 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------





root@len:/home/kevin# iperf3 -c 10.147.18.185
Connecting to host 10.147.18.185, port 5201
[ 5] local 10.147.18.65 port 49460 connected to 10.147.18.185 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.23 MBytes 10.3 Mbits/sec 1 88.6 KBytes 
[ 5] 1.00-2.00 sec 1.15 MBytes 9.61 Mbits/sec 5 77.8 KBytes 
[ 5] 2.00-3.00 sec 1.08 MBytes 9.10 Mbits/sec 7 51.0 KBytes 
[ 5] 3.00-4.00 sec 1.08 MBytes 9.10 Mbits/sec 0 77.8 KBytes 
[ 5] 4.00-5.00 sec 1.08 MBytes 9.10 Mbits/sec 0 96.6 KBytes 
[ 5] 5.00-6.00 sec 1.08 MBytes 9.10 Mbits/sec 4 48.3 KBytes 
[ 5] 6.00-7.00 sec 1.08 MBytes 9.10 Mbits/sec 0 75.1 KBytes 
[ 5] 7.00-8.00 sec 1.08 MBytes 9.10 Mbits/sec 0 93.9 KBytes 
[ 5] 8.00-9.00 sec 1.08 MBytes 9.10 Mbits/sec 1 83.2 KBytes 
[ 5] 9.00-10.00 sec 1.08 MBytes 9.10 Mbits/sec 5 34.9 KBytes 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.1 MBytes 9.27 Mbits/sec 23 sender
[ 5] 0.00-10.03 sec 10.8 MBytes 9.03 Mbits/sec receiver

iperf Done.
root@len:/home/kevin#





root@intel:~# iperf3 -c 10.147.18.185
Connecting to host 10.147.18.185, port 5201
[ 5] local 10.147.18.84 port 46324 connected to 10.147.18.185 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.45 MBytes 12.1 Mbits/sec 10 48.3 KBytes 
[ 5] 1.00-2.00 sec 926 KBytes 7.58 Mbits/sec 3 51.0 KBytes 
[ 5] 2.00-3.00 sec 1.08 MBytes 9.10 Mbits/sec 0 77.8 KBytes 
[ 5] 3.00-4.00 sec 1.08 MBytes 9.10 Mbits/sec 13 32.2 KBytes 
[ 5] 4.00-5.00 sec 926 KBytes 7.58 Mbits/sec 1 53.7 KBytes 
[ 5] 5.00-6.00 sec 1.27 MBytes 10.6 Mbits/sec 1 56.4 KBytes 
[ 5] 6.00-7.00 sec 1.15 MBytes 9.61 Mbits/sec 1 53.7 KBytes 
[ 5] 7.00-8.00 sec 1.08 MBytes 9.10 Mbits/sec 1 64.4 KBytes 
[ 5] 8.00-9.00 sec 1.08 MBytes 9.10 Mbits/sec 1 64.4 KBytes 
[ 5] 9.00-10.00 sec 1.08 MBytes 9.10 Mbits/sec 2 45.6 KBytes 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.1 MBytes 9.30 Mbits/sec 33 sender
[ 5] 0.00-10.03 sec 10.8 MBytes 9.03 Mbits/sec receiver

iperf Done.
root@intel:~#


root@asus:/home/kevin# iperf3 -c 10.147.18.185
Connecting to host 10.147.18.185, port 5201
[ 5] local 10.147.18.14 port 43012 connected to 10.147.18.185 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.31 MBytes 10.9 Mbits/sec 1 93.9 KBytes

[ 5] 1.00-2.00 sec 1.08 MBytes 9.10 Mbits/sec 0 113 KBytes 
[ 5] 2.00-3.00 sec 1.21 MBytes 10.1 Mbits/sec 2 93.9 KBytes 
[ 5] 3.00-4.00 sec 741 KBytes 6.07 Mbits/sec 11 26.8 KBytes 
[ 5] 4.00-5.00 sec 802 KBytes 6.57 Mbits/sec 1 51.0 KBytes 
[ 5] 5.00-6.00 sec 1.08 MBytes 9.10 Mbits/sec 0 75.1 KBytes 
[ 5] 6.00-7.00 sec 1.02 MBytes 8.60 Mbits/sec 1 72.5 KBytes 
[ 5] 7.00-8.00 sec 1.02 MBytes 8.59 Mbits/sec 1 64.4 KBytes 
[ 5] 8.00-9.00 sec 1.08 MBytes 9.11 Mbits/sec 0 85.9 KBytes 
[ 5] 9.00-10.00 sec 1.08 MBytes 9.10 Mbits/sec 1 77.8 KBytes 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.4 MBytes 8.73 Mbits/sec 18 sender
[ 5] 0.00-10.07 sec 10.2 MBytes 8.51 Mbits/sec receiver

iperf Done.
root@asus:/home/kevin#

 

I then closed down the zerotier VPN on len and did a test from len via the external internet to gemini, ie not using the VPN:

 

root@len:/home/kevin# systemctl stop zerotier-one

(obviously at that point the vpn connection lost as I was connected via an ssh session).

 

root@gemini:/home/kevin# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 5.146.192.17, port 59644
[ 5] local 45.76.140.242 port 5201 connected to 5.146.192.17 port 59646
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 974 KBytes 7.97 Mbits/sec 
[ 5] 1.00-2.00 sec 1.14 MBytes 9.55 Mbits/sec 
[ 5] 2.00-3.00 sec 1.16 MBytes 9.73 Mbits/sec 
[ 5] 3.00-4.00 sec 1.15 MBytes 9.61 Mbits/sec 
[ 5] 4.00-5.00 sec 1.12 MBytes 9.43 Mbits/sec 
[ 5] 5.00-6.00 sec 1.16 MBytes 9.73 Mbits/sec 
[ 5] 6.00-7.00 sec 1.11 MBytes 9.27 Mbits/sec 
[ 5] 7.00-8.00 sec 1.16 MBytes 9.70 Mbits/sec 
[ 5] 8.00-9.00 sec 1.08 MBytes 9.07 Mbits/sec 
[ 5] 9.00-10.00 sec 1.14 MBytes 9.57 Mbits/sec 
[ 5] 10.00-10.03 sec 28.9 KBytes 9.35 Mbits/sec 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.03 sec 11.2 MBytes 9.36 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

 

 

There was no difference in speed. No difference in speed either if len is disconnected from vpn, the other clients show same speeds.


root@gemini:/home/kevin# ufw allow 5201
Rule added
Rule added (v6)
root@gemini:/home/kevin# 
root@gemini:/home/kevin# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.147.18.84, port 46580
[ 5] local 10.147.18.185 port 5201 connected to 10.147.18.84 port 46582
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.07 MBytes 8.97 Mbits/sec 
[ 5] 1.00-2.00 sec 1.00 MBytes 8.40 Mbits/sec 
[ 5] 2.00-3.00 sec 1.01 MBytes 8.46 Mbits/sec 
[ 5] 3.00-4.00 sec 1.13 MBytes 9.48 Mbits/sec 
[ 5] 4.00-5.00 sec 1.14 MBytes 9.56 Mbits/sec 
[ 5] 5.00-6.00 sec 1.07 MBytes 8.99 Mbits/sec 
[ 5] 6.00-7.00 sec 993 KBytes 8.13 Mbits/sec 
[ 5] 7.00-8.00 sec 1.13 MBytes 9.45 Mbits/sec 
[ 5] 8.00-9.00 sec 1.13 MBytes 9.52 Mbits/sec 
[ 5] 9.00-10.00 sec 1.12 MBytes 9.43 Mbits/sec 
[ 5] 10.00-10.05 sec 53.7 KBytes 9.37 Mbits/sec 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.05 sec 10.8 MBytes 9.04 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^X^Z
[1]+ Stopped iperf3 -s
root@gemini:/home/kevin# ufw enable
Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup
root@gemini:/home/kevin#




root@intel:~# iperf3 -c 10.147.18.185
Connecting to host 10.147.18.185, port 5201
[ 5] local 10.147.18.84 port 46582 connected to 10.147.18.185 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.46 MBytes 12.3 Mbits/sec 1 85.9 KBytes 
[ 5] 1.00-2.00 sec 926 KBytes 7.58 Mbits/sec 11 26.8 KBytes 
[ 5] 2.00-3.00 sec 1.08 MBytes 9.10 Mbits/sec 0 59.0 KBytes 
[ 5] 3.00-4.00 sec 1.08 MBytes 9.10 Mbits/sec 1 59.0 KBytes 
[ 5] 4.00-5.00 sec 1.08 MBytes 9.10 Mbits/sec 0 83.2 KBytes 
[ 5] 5.00-6.00 sec 1.08 MBytes 9.10 Mbits/sec 11 32.2 KBytes 
[ 5] 6.00-7.00 sec 1.08 MBytes 9.10 Mbits/sec 1 56.4 KBytes 
[ 5] 7.00-8.00 sec 1.08 MBytes 9.10 Mbits/sec 0 80.5 KBytes 
[ 5] 8.00-9.00 sec 1.15 MBytes 9.61 Mbits/sec 1 75.1 KBytes 
[ 5] 9.00-10.00 sec 1.08 MBytes 9.10 Mbits/sec 5 53.7 KBytes 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.1 MBytes 9.32 Mbits/sec 31 sender
[ 5] 0.00-10.05 sec 10.8 MBytes 9.04 Mbits/sec receiver

iperf Done.
root@intel:~# 
root@gemini:/home/kevin# ufw deny 5201
Rule updated
Rule updated (v6)
root@gemini:/home/kevin#

 

I then also did a speed test using two laptops, rather than the server. speed test with iperf3 from asus to intel is much faster:

 

 

root@asus:/home/kevin/LOCAL# iperf3 -c intelvpn 
Connecting to host intelvpn, port 5201
[ 5] local 10.147.18.14 port 50668 connected to 10.147.18.84 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 4.00 MBytes 33.6 Mbits/sec 1 252 KBytes 
[ 5] 1.00-2.00 sec 4.82 MBytes 40.5 Mbits/sec 2 271 KBytes 
[ 5] 2.00-3.00 sec 4.64 MBytes 38.9 Mbits/sec 0 295 KBytes 
[ 5] 3.00-4.00 sec 5.49 MBytes 46.0 Mbits/sec 1 317 KBytes 
[ 5] 4.00-5.00 sec 5.85 MBytes 49.0 Mbits/sec 0 344 KBytes 
[ 5] 5.00-6.00 sec 6.21 MBytes 52.1 Mbits/sec 9 263 KBytes 
[ 5] 6.00-7.00 sec 4.28 MBytes 35.9 Mbits/sec 1 293 KBytes 
[ 5] 7.00-8.00 sec 5.42 MBytes 45.5 Mbits/sec 1 317 KBytes 
[ 5] 8.00-9.00 sec 7.11 MBytes 59.7 Mbits/sec 3 338 KBytes 
[ 5] 9.00-10.00 sec 6.81 MBytes 57.1 Mbits/sec 9 255 KBytes 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 54.6 MBytes 45.8 Mbits/sec 27 sender
[ 5] 0.00-10.04 sec 53.8 MBytes 44.9 Mbits/sec receiver

iperf Done.
root@asus:/home/kevin/LOCAL#




root@intel:/home/kevin# ufw allow 5201
Rule added
Rule added (v6)
root@intel:/home/kevin# iperf3 -s 
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.147.18.14, port 50666
[ 5] local 10.147.18.84 port 5201 connected to 10.147.18.14 port 50668
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 3.60 MBytes 30.2 Mbits/sec 
[ 5] 1.00-2.00 sec 4.62 MBytes 38.7 Mbits/sec 
[ 5] 2.00-3.00 sec 4.53 MBytes 38.0 Mbits/sec 
[ 5] 3.00-4.00 sec 5.47 MBytes 45.9 Mbits/sec 
[ 5] 4.00-5.00 sec 6.03 MBytes 50.6 Mbits/sec 
[ 5] 5.00-6.00 sec 5.80 MBytes 48.7 Mbits/sec 
[ 5] 6.00-7.00 sec 4.20 MBytes 35.3 Mbits/sec 
[ 5] 7.00-8.00 sec 5.86 MBytes 49.2 Mbits/sec 
[ 5] 8.00-9.00 sec 6.74 MBytes 56.5 Mbits/sec 
[ 5] 9.00-10.00 sec 6.71 MBytes 56.3 Mbits/sec 
[ 5] 10.00-10.04 sec 215 KBytes 48.3 Mbits/sec 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.04 sec 53.8 MBytes 44.9 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------








a bit slower with the len to intel:


root@len:/home/kevin# iperf3 -c intelvpn
Connecting to host intelvpn, port 5201
[ 5] local 10.147.18.65 port 55332 connected to 10.147.18.84 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 4.24 MBytes 35.6 Mbits/sec 1 204 KBytes 
[ 5] 1.00-2.00 sec 3.98 MBytes 33.4 Mbits/sec 2 225 KBytes 
[ 5] 2.00-3.00 sec 3.92 MBytes 32.9 Mbits/sec 1 247 KBytes 
[ 5] 3.00-4.00 sec 4.10 MBytes 34.4 Mbits/sec 7 188 KBytes 
[ 5] 4.00-5.00 sec 4.22 MBytes 35.4 Mbits/sec 0 233 KBytes 
[ 5] 5.00-6.00 sec 4.16 MBytes 34.9 Mbits/sec 0 258 KBytes 
[ 5] 6.00-7.00 sec 4.04 MBytes 33.9 Mbits/sec 0 268 KBytes 
[ 5] 7.00-8.00 sec 3.68 MBytes 30.8 Mbits/sec 0 282 KBytes 
[ 5] 8.00-9.00 sec 3.92 MBytes 32.9 Mbits/sec 0 301 KBytes 
[ 5] 9.00-10.00 sec 3.74 MBytes 31.3 Mbits/sec 0 319 KBytes 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 40.0 MBytes 33.5 Mbits/sec 11 sender
[ 5] 0.00-10.08 sec 39.7 MBytes 33.0 Mbits/sec receiver

iperf Done.
root@len:/home/kevin#


#



from len to asus:

 

root@len:/home/kevin# iperf3 -c asusvpn
Connecting to host asusvpn, port 5201
[ 5] local 10.147.18.65 port 59526 connected to 10.147.18.14 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.41 MBytes 28.6 Mbits/sec 1 169 KBytes 
[ 5] 1.00-2.00 sec 1.81 MBytes 15.2 Mbits/sec 18 102 KBytes 
[ 5] 2.00-3.00 sec 3.92 MBytes 32.9 Mbits/sec 1 145 KBytes 
[ 5] 3.00-4.00 sec 4.64 MBytes 38.9 Mbits/sec 0 185 KBytes 
[ 5] 4.00-5.00 sec 2.35 MBytes 19.7 Mbits/sec 8 102 KBytes 
[ 5] 5.00-6.00 sec 1.63 MBytes 13.7 Mbits/sec 0 129 KBytes 
[ 5] 6.00-7.00 sec 3.44 MBytes 28.8 Mbits/sec 1 156 KBytes 
[ 5] 7.00-8.00 sec 4.10 MBytes 34.4 Mbits/sec 4 150 KBytes 
[ 5] 8.00-9.00 sec 2.53 MBytes 21.2 Mbits/sec 6 126 KBytes 
[ 5] 9.00-10.00 sec 1.57 MBytes 13.1 Mbits/sec 2 145 KBytes 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 29.4 MBytes 24.6 Mbits/sec 41 sender
[ 5] 0.00-10.01 sec 29.1 MBytes 24.4 Mbits/sec receiver

iperf Done.
root@len:/home/kevin#

root@asus:/# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.147.18.65, port 59524
[ 5] local 10.147.18.14 port 5201 connected to 10.147.18.65 port 59526
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 3.01 MBytes 25.2 Mbits/sec 
[ 5] 1.00-2.00 sec 1.71 MBytes 14.4 Mbits/sec 
[ 5] 2.00-3.00 sec 3.99 MBytes 33.5 Mbits/sec 
[ 5] 3.00-4.00 sec 4.57 MBytes 38.4 Mbits/sec 
[ 5] 4.00-5.00 sec 2.43 MBytes 20.4 Mbits/sec 
[ 5] 5.00-6.00 sec 1.64 MBytes 13.7 Mbits/sec 
[ 5] 6.00-7.00 sec 3.41 MBytes 28.6 Mbits/sec 
[ 5] 7.00-8.00 sec 4.10 MBytes 34.4 Mbits/sec 
[ 5] 8.00-9.00 sec 2.60 MBytes 21.8 Mbits/sec 
[ 5] 9.00-10.00 sec 1.67 MBytes 14.0 Mbits/sec 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 29.1 MBytes 24.4 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------




root@len:~# iperf3
iperf3: parameter error - must either be a client (-c) or server (-s)

Usage: iperf3 [-s|-c host] [options]
iperf3 [-h|--help] [-v|--version]

Server or Client:
-p, --port # server port to listen on/connect to
-f, --format [kmgtKMGT] format to report: Kbits, Mbits, Gbits, Tbits
-i, --interval # seconds between periodic throughput reports
-F, --file name xmit/recv the specified file
-A, --affinity n/n,m set CPU affinity
-B, --bind <host> bind to the interface associated with the address <host>
-V, --verbose more detailed output
-J, --json output in JSON format
--logfile f send output to a log file
--forceflush force flushing output at every interval
-d, --debug emit debugging output
-v, --version show version information and quit
-h, --help show this message and quit
Server specific:
-s, --server run in server mode
-D, --daemon run the server as a daemon
-I, --pidfile file write PID file
-1, --one-off handle one client connection then exit
--rsa-private-key-path path to the RSA private key used to decrypt
authentication credentials
--authorized-users-path path to the configuration file containing user
credentials
Client specific:
-c, --client <host> run in client mode, connecting to <host>
--sctp use SCTP rather than TCP
-X, --xbind <name> bind SCTP association to links
--nstreams # number of SCTP streams
-u, --udp use UDP rather than TCP
--connect-timeout # timeout for control connection setup (ms)
-b, --bitrate #[KMG][/#] target bitrate in bits/sec (0 for unlimited)
(default 1 Mbit/sec for UDP, unlimited for TCP)
(optional slash and packet count for burst mode)
--pacing-timer #[KMG] set the timing for pacing, in microseconds (default 1000)
--fq-rate #[KMG] enable fair-queuing based socket pacing in
bits/sec (Linux only)
-t, --time # time in seconds to transmit for (default 10 secs)
-n, --bytes #[KMG] number of bytes to transmit (instead of -t)
-k, --blockcount #[KMG] number of blocks (packets) to transmit (instead of -t or -n)
-l, --length #[KMG] length of buffer to read or write
(default 128 KB for TCP, dynamic or 1460 for UDP)
--cport <port> bind to a specific client port (TCP and UDP, default: ephemeral port)
-P, --parallel # number of parallel client streams to run
-R, --reverse run in reverse mode (server sends, client receives)
--bidir run in bidirectional mode.
Client and server send and receive data.
-w, --window #[KMG] set window size / socket buffer size
-C, --congestion <algo> set TCP congestion control algorithm (Linux and FreeBSD only)
-M, --set-mss # set TCP/SCTP maximum segment size (MTU - 40 bytes)
-N, --no-delay set TCP/SCTP no delay, disabling Nagle's Algorithm
-4, --version4 only use IPv4
-6, --version6 only use IPv6
-S, --tos N set the IP type of service, 0-255.
The usual prefixes for octal and hex can be used,
i.e. 52, 064 and 0x34 all specify the same value.
--dscp N or --dscp val set the IP dscp value, either 0-63 or symbolic.
Numeric values can be specified in decimal,
octal and hex (see --tos above).
-L, --flowlabel N set the IPv6 flow label (only supported on Linux)
-Z, --zerocopy use a 'zero copy' method of sending data
-O, --omit N omit the first n seconds
-T, --title str prefix every output line with this string
--extra-data str data string to include in client and server JSON
--get-server-output get results from server
--udp-counters-64bit use 64-bit counters in UDP test packets
--repeating-payload use repeating pattern in payload, instead of
randomized payload (like in iperf2)
--username username for authentication
--rsa-public-key-path path to the RSA public key used to encrypt
authentication credentials

[KMG] indicates options that support a K/M/G suffix for kilo-, mega-, or giga-

iperf3 homepage at: https://software.es.net/iperf/
Report bugs to: https://github.com/esnet/iperf
root@len:~#


 

Continue Reading

LPIC3 DIPLOMA Linux Clustering – LAB NOTES: Lesson Monit

 

Monit is an open source utility for monitoring services on Linux systems and keeping them running.

 

If for any reason a monitored service shuts down, Monit will attempt to bring it back online.

 

Monit also comes with a web interface which can also be used to control and monitor services.

 

To install Monit

 

(instructions for Debian/Ubuntu systems):

 

apt-get install monit

 

systemctl enable –now monit

 

 

 

How To Configure Monit

 

Monit configuration files are located under /etc/monit/ directory.

 

The main configuration file is /etc/monit/monitrc.

 

All files in /etc/monit/conf.d/ and /etc/monit/conf-enabled/ are read by monit when started.

 

 

Monit has an embedded HTTP interface for viewing service status via a web interface.

 

By default monit HTTP interface is not enabled. To enable uncomment the following lines in /etc/monit/monitrc

 

nano /etc/monit/monitrc

 

set httpd port 2812 and
use address localhost # only accept connection from localhost
allow localhost # allow localhost to connect to the server and
allow admin:monit # require user ‘admin’ with password ‘monit’

 

# NOTE: make sure you change these to something else in online or production environments!

 

 

You can change admin:monit to use another username and password. To connect from a different IP, add:

 

allow <IP Address>

 

then restart:

 

systemctl restart monit

 

 

How To Use Monit

 

 

To display system status with monit:

 

monit status

 

root@intel:~# monit summary
Monit 5.26.0 uptime: 0m
┌─────────────────────────────────┬────────────────────────────┬───────────────┐
│ Service Name │ Status │ Type │
├─────────────────────────────────┼────────────────────────────┼───────────────┤
│ intel │ OK │ System │
└─────────────────────────────────┴────────────────────────────┴───────────────┘
root@intel:~#

 

root@intel:~# monit status
Monit 5.26.0 uptime: 0m

 

System ‘intel’
status OK
monitoring status Monitored
monitoring mode active
on reboot start
load average [0.22] [0.48] [0.57]
cpu 0.0%us 0.0%sy 0.0%wa
memory usage 2.0 GB [26.3%]
swap usage 0 B [0.0%]
uptime 26m
boot time Mon, 17 May 2021 14:04:37
data collected Mon, 17 May 2021 14:31:10

 

root@intel:~#

 

 

 

To check config:

 

monit -t

 

root@intel:~# monit -t
Control file syntax OK
root@intel:~#

 

 

To reload config after changes:

 

monit reload

 

root@intel:~# monit reload
Reinitializing monit daemon
root@intel:~#

 

to start running all monitored programs:

 

monit start all

 

 

To access Monit Web Interface:

 

http://[ip-address|domain]:2812

Login with username “admin” and password “monit”.

 

To allow access to port from remote IPs through the firewall, run:

 

root@intel:~# ufw allow 2812
Rules updated
Rules updated (v6)
root@intel:~#

 

 

How to Configure Monit Web Interface to use SSL/TLS HTTPS

 

 

In directory  /etc/monit/  prepare the config file monit.cnf:

 

# create RSA certs – Server

RANDFILE = ./openssl.rnd

[ req ]
default_bits = 2048
default_md = sha256
encrypt_key = yes
distinguished_name = req_dn
x509_extensions = cert_type

[ req_dn ]
countryName = Country Name (2 letter code)
countryName_default = UK

stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = England

localityName = Locality Name (eg, city)
localityName_default = London

organizationName = Organization Name (eg, company)
organizationName_default = kevwells.com

organizationalUnitName = Organizational Unit Name (eg, section)
organizationalUnitName_default = kevwells.com

commonName = Common Name (FQDN of your server)
commonName_default = kevwells.com

emailAddress = Email Address
emailAddress_default = mmonit@kevwells.com

[ cert_type ]
nsCertType = server

 

 

save above as monit.cnf

 

then still within the /etc/monit directory where you have just saved monit.cnf  run these commands to generate the pemfile :

 

 

# Generates the private key and the certificate
openssl req -new -x509 -days 365 -nodes -config ./monit.cnf -out /etc/ssl/certs/monit.pem \
-keyout /etc/ssl/certs/monit.pem

 

# Generates the Diffie-Hellman Parameters
openssl dhparam -2 2048 >> /etc/ssl/certs/monit.pem

 

# Set mode
chmod 600 /etc/ssl/certs/monit.pem

 

# Prints out the certificate information
openssl x509 -text -noout -in /etc/ssl/certs/monit.pem

 

 

root@gemini:/etc/monit# openssl dhparam -2 2048 >> /etc/ssl/certs/monit.pem
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
……………………………………………………………+.

 

 

then after doing

 

openssl x509 -text -noout -in /etc/ssl/certs/monit.pem

 

do:

 

root@gemini:/etc/monit# monit -t
Control file syntax OK

root@gemini:/etc/monit# systemctl restart monit
root@gemini:/etc/monit# systemctl status monit
● monit.service – LSB: service and resource monitoring daemon
Loaded: loaded (/etc/init.d/monit; generated)
Active: active (running) since Mon 2021-05-17 14:09:10 BST; 5s ago
Docs: man:systemd-sysv-generator(8)
Process: 13001 ExecStart=/etc/init.d/monit start (code=exited, status=0/SUCCESS)
Tasks: 2 (limit: 2280)
Memory: 1.2M
CGroup: /system.slice/monit.service
└─13018 /usr/bin/monit -c /etc/monit/monitrc

 

May 17 14:09:10 gemini systemd[1]: Starting LSB: service and resource monitoring daemon…
May 17 14:09:10 gemini monit[13001]: * Starting daemon monitor monit
May 17 14:09:10 gemini monit[13001]: …done.
May 17 14:09:10 gemini systemd[1]: Started LSB: service and resource monitoring daemon.
root@gemini:/etc/monit#

 

You can then access the monitoring web interface with:

 

http://kevwells.com:2812

 

(enter username and password when prompted – these have been changed from the standard)

 

Continue Reading