Tags Archives: ssh

AWS How To Create A New User With SSH Access For An EC2 Instance

How To Create A New User With SSH Access For An EC2 Instance

Each Linux EC2 instance is provided with a default Linux system user account. This default user name is determined by the AMI selected when you launched the instance. For Linux Ubuntu this user is called “ubuntu”.


You can also add and delete user accounts for your instance.


By default for security reasons, password authentication and root logins are disabled, and sudo is enabled.


Thus to log in to your instance, you need to use an SSH key pair.


You can allow password authentication and root login for your instance but this is not recommended.


To create a new user on the instance, do:



adduser <username>


adduser kevin

root@ip-172-31-24-137:~# adduser kevin
Adding user `kevin’ …
Adding new group `kevin’ (1001) …
Adding new user `kevin’ (1001) with group `kevin’ …
Creating home directory `/home/kevin’ …
Copying files from `/etc/skel’ …
New password:
Retype new password:
passwd: password updated successfully
Changing the user information for kevin
Enter the new value, or press ENTER for the default
Full Name []: Kevin
Room Number []:
Work Phone []:
Home Phone []:
Other []:
Is the information correct? [Y/n] Y


Then we create the .ssh directory for kevin:


root@ip-172-31-24-137:/home/kevin# mkdir .ssh


Next we are going to deploy the same private/public key pair for this new user kevin as we have for the user ubuntu. (user ubuntu is a standard EC2 user which is automatically created for logins to the instance when creating the instance).


So, we copy the /home/ubuntu/.ssh/authorized_keys to /home/kevin/.ssh/


and then modify the permissions so they are correct for this user:


chown -R kevin.kevin .ssh; chmod 700 .ssh; touch .ssh/authorized_keys; chmod 600 .ssh/authorized_keys


Then, you can login to the instance from outside AWS by using:


root@gemini:/home/kevin/DATAVOLUME/DATA/AWS/LOGINSANDKEYS/EC2_KevinVM1_created.17.11.22# ssh kevin@ec2-3-85-124-194.compute-1.amazonaws.com -i ./KevinVM1.pem


Note that you must be in the correct local folder to reference the KevinVM1.pem ssh private key file


Keep this file confidential and secure at all times!


and also note the kevin@ – this is essential.



Troubleshooting the SSH


If you cannot connect via ssh, then login to the EC2 instance using the AWS Instance Connect web console,


and check that the sshd service is running on the instance:


systemctl status sshd


if not started, start with


systemctl start sshd


root@ip-172-31-24-137:/home/kevin# systemctl status sshd
● ssh.service – OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Drop-In: /usr/lib/systemd/system/ssh.service.d
Active: active (running) since Thu 2022-11-24 14:54:34 UTC; 44min ago
Docs: man:sshd(8)
Process: 760 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 762 (sshd)
Tasks: 1 (limit: 1143)


if the system


Connection timeout


This is a security group issue. Any timeout (not just for SSH) is related to security groups or firewall issues.


Make sure your security group is correctly assigned to your EC2 instance and allows inbound connections from all IPs or the IP you are trying to connect from on port SSH 22.


If you still cannot connect receiving a time out message, then this means a firewall is blocking the connection.


So this will need to be modified first.


If SSH does not work on Windows and replies with “ssh command not found” then you need to use or install Putty


If you get “connection refused” this means the instance is reachable, but no SSH client service is running on the instance


Try to restart the instance and if SSH still does not work, terminate the instance and create a new one. Make sure you’re using Amazon Linux 2 for the EC2 instance.


If you get “Permission denied (publickey,gssapi-keyex,gssapi-with-mic)”


This means you are using the wrong security key or not using a security key at all. Check the EC2 instance configuration to ensure you have assigned the correct key to it.

Alternatively permission refused can happen if you are using the wrong user. Check that you have started an Amazon Linux 2 EC2 instance, and ensure you are using the user ec2-user. This is speficied as ec2-user@<public-ip> (ex: ec2-user@ in your SSH command.


As a workaround you can connect to the instance by using EC2 Instance Connect.


If you could connect yesterday, but not today then this can be because you have stopped your EC2 instance and then started it again today.


When you do this, the public IP of that EC2 instance changes.


Be sure that you are trying to connect using the correct current public IP for the instance.



You can access your EC2 instance using AWS EC2 Instance Connect as a workaround while you fix the problem.


Continue Reading

AWS – Connecting to AWS

AWS Remote Connectivity


you have following options to connect to your AWS EC2 instances:


SSH to an instance. Default port is 22


AWS Instance Connect is a web-based console you can also use




To connect using SSH



download your .pem access key.


remove the space within the filename if there is a space.


Then place in a directory on your local client machine and from that directory,


get your public ip from your ec2 instance that you want to ssh into.


and also check the security of your instance to check that port 22 from anywhere exists as a rule and allows access.


then do ssh ec2user@<yourEC2publicIP>


you will see an authentication error initially, because you have to specify your access key… so,


we need to reference our .pem file into our command…


so cd to the directory where your .pem file is located, then do:


ssh -i <your.pem file> ec2user@<yourEC2publicIP>


then you will see the error that you have to change your permissions of your key file:


so on your local client do


chmod 0400 *.pem


then you should be able to login ok.



Resolving SSH Connectivity Problems


if you still have a connection problem…


if it is “connection refused”, then this means the instance is reachable, but no SSH utility is running on the instance




Try to restart the instance


If it doesn’t work, terminate the instance and create a new one. Make sure you’re using Amazon Linux 2


If your security group is properly configured as above, and you still have connection timeout issues, then that means a corporate firewall or a personal firewall is blocking the connection.



In this case use EC2 Instance Connect as described below.



Permission denied (publickey,gssapi-keyex,gssapi-with-mic)


This means either two things:


You are using the wrong security key or not using a security key. Please look at your EC2 instance configuration to make sure you have assigned the correct key to it.


You are using the wrong user. Make sure you have started an Amazon Linux 2 EC2 instance, and make sure you’re using the user ec2-user.




if you were able to connect yesterday, but not today…


This is probably because you have stopped your EC2 instance and then started it again today. When you do so, the public IP of your EC2 instance will change. So you need to get the new public ip of the instance and try connecting to that.



EC2 Instance Connect


login to your AWS Web Management Desktop and navigate to the EC2 Dashboard, and select the instance you want to connect to, and then click on CONNECT.


This will open a CLI shell console session on your EC2 instance via the web-browser window.






Use IAM Roles instead of AWS Credentials (access keys) to login to your EC2 Instances





You CAN enter your access key id and secret access key into your aws credentials within your ssh session, using the “aws credentials” cli command – but you are strongly advised NOT to do this!


if you do this, then anyone who has access to the file on your laptop or pc can access your instance… and they can retrieve your access keys from your instance!



Therefore, always use IAM Roles instead to grant users access to EC2 instances!



To do this, we attach the role we created earlier to our ec2 instance, via the ec2 dashboard:


in your EC2 Dashboard, select the EC2 instance  then select Actions – Security – Modify iam role 


then add the role you want, then choose the iam role you want to add… in this case our demo role.



you can check with:


[ec2-user@ip-172-31-87-242 ~]$
[ec2-user@ip-172-31-87-242 ~]$ aws iam list-users
“Users”: [
“UserName”: “kevin”,
“PasswordLastUsed”: “2022-08-08T09:21:00Z”,
“CreateDate”: “2022-08-08T09:17:45Z”,
“Path”: “/”,
“Arn”: “arn:aws:iam::042663301467:user/kevin”
[ec2-user@ip-172-31-87-242 ~]$




Continue Reading

SSH Reverse Tunnel – A Practical Example – Public Internet Server to Client Laptop on NAT WiFi

Instructions for setting up an ssh reverse tunnel from my laptop to my internet server.


This was necessary as I could not connect to my NFS shares on gemini from my laptop when using external wifi as my server denies connections that are not made via my private VPN network. This is for security reasons to prevent third-party access to my NFS server shares.




However, I can ssh in to the gemini server.  So I set up a reverse SSH tunnel.  I did the following:



on laptop:


ssh -f -N -T -R22000:localhost:22 gemini


I can then copy files through the tunnel using the port 22000 address on gemini…




root@gemini:~# scp -P 22000 filefromgemini gemini:/tmp
root@gemini’s password:
filefromgemini 100% 47 3.8KB/s 00:00


I want to copy the WordPress databases from gemini to laptop:


on gemini do:


wp cache flush –path=/var/www/wordpress –allow-root


# create the database export from server kevwells.com:


mysqldump –all-databases > /home/kevin/all_databases.sql



then scp the .sql file to port 22000 on gemini – this is the ssh reverse tunnel entrance to my laptop!


scp -P 22000 /home/kevin/all_databases.sql gemini:/home/kevin/


(the “gemini:/home/kevin” in this case is actually my /home/kevin on the laptop! – ie the other end ie exit of the tunnel)



so, to summarize we do:


on laptop client:


root@asus:/usr/local/bin# ssh -f -N -T -R22000:localhost:22 gemini


on gemini server:


root@gemini:~# wp cache flush –path=/var/www/wordpress –allow-root
Success: The cache was flushed.
root@gemini:~# mysqldump –all-databases > /home/kevin/all_databases.sql


root@gemini:/home/kevin# ls
all_databases.sql BACKUP DATA Downloads NEXTCLOUD readme.locationofmyphp.ini
root@gemini:/home/kevin# scp -P 22000 /home/kevin/all_databases.sql gemini:/home/kevin/
root@gemini’s password:
all_databases.sql 100% 28MB 2.0MB/s 00:14


then on laptop, sure enough the file has arrived:


root@asus:/home/kevin# ll
total 253272
drwxrwxr-x 34 kevin kevin 16384 Apr 21 16:02 ./
drwxr-xr-x 3 root root 4096 Aug 28 2021 ../
-rw-r–r– 1 root root 29555359 Apr 21 16:02 all_databases.sql



Mission accomplished.



Continue Reading

How To Set Up An SSH Reverse Tunnel

Reverse SSH tunneling, otherwise known as remote port forwarding via SSH. This is a way to connect to a machine by having the other computer call you first and then activating a second connection over the initial one in the opposite direction.


How SSH Remote Tunneling Works


To connect via SSH or some other service, such as HTTP, from the Internet into a machine behind a firewall, you need the machine in question to open an SSH connection to a machine outside the firewall and include an SSH -R tunnel whose “entry” point is the “remote” side of this connection.


This means, from the firewalled host, you need to execute something like:


ssh -f -N -T -R22000:localhost:22 yourpublichost.example.com


This instructs your client machine to establish a tunnel with an SSH -R remote entry point.


Anything that then attaches to port 22000 at the other end of the tunnel will be automatically forwarded to “localhost port 22”, where “localhost” is defined from the point of view of the exit point of the tunnel – namely your SSH client machine.



The other options here are:


-f instructs SSH to background itself after authentication. T
-N instructs SSH to create a connection, but without actually running any remote commands. This saves processing resources and time.
-T instructs SSH to disable pseudo-tty allocation, which is useful if you do not want an interactive command line shell.


Standard local port forwarding does not work if incoming SSH requests in the remote server are disabled. For security reasons, administrators might entirely block inbound SSH requests but allow outbound SSH requests.


In such situations, you can use remote port forwarding to create an outbound SSH connection and let the clients connect to the local port even if inbound connections are blocked.



Now, when users from distant internet visit port 80 of the remote server as http://<remote_server_ip>, the request is redirected back to the client’s local server (port 3000) via SSH tunnel where the local server handles the request and response.


By default, the remote port forwarding tunnel will bind to the localhost of the remote server. To enable it to listen on the public interface (for a scenario like above), set the SSH configuration GatewayPorts yes in sshd_config.



This is the command, set from the outgoing machine, in this case my laptop:


ssh -NR 8080:localhost:8081 kevin@kevwells.com


it will not return to the command line – that is correct behaviour.


To run a webserver over the tunnel you need to set it up to run on eg port 8081. You then set the tunnel to run on eg 8080.


Configure sshd_config


You need to set the following directives in /etc/ssh/sshd_config:


AllowTcpForwarding yes
GatewayPorts yes


Make sure you restart ssh service after editing the file: ie


systemctl restart ssh


You can then connect to the asus website using http://kevwells.com:8080


Make sure apache is running on 8081 on the localhost.


Points to Watch with SSH Reverse Tunneling


This only works here with http.


https will NOT work – since one cannot obtain an SSL/TLS certificate for a local NAT IP address – only for publicly accessible IPs ie domain names accessible without NAT.



By default, TCP listening sockets on the server are bound only to the loopback interface. This can be overridden by specifying a bind_address. An empty bind_address, or the address *, directs that the remote socket should listen on all interfaces.



If you specify a remote bind_address then this will only function if the server’s GatewayPorts option is enabled in sshd_config.


In addition, the GatewayPorts directive in the sshd_config on the server must be set.


To quote from the sshd_config man page:


This specifies whether remote hosts are allowed to connect to ports forwarded for the client. By default, sshd(8) binds remote port forwardings to the loopback address. This prevents other remote hosts from connecting to forwarded ports. GatewayPorts can be used to specify that sshd should allow remote port forwardings to bind to non-loopback addresses, thus allowing other hosts to connect. The argument may be no to force remote port forwardings to be available to the local host only, yes to force remote port forwardings to bind to the wildcard address, or client specified to allow the client to select the address to which the forwarding is bound. The default is no.



Remember to restart the ssh server service after all config changes.


Make sure also that all ports in use are accessible through any firewall or routers.


Also if you are running a local apache webserver on the NAT machine, then make sure you have the respective port open in the ports.conf at /etc/apache2/ports.conf


in this example




AND set a virtual hosts definition in your active sites-enabled config file for the port you are using for the webserver on the local machine ie:


(this definition is on the LOCAL NAT machine apache config)


<VirtualHost *.:8081>


You do NOT need any definition for this, nor for (in this case 8080 but it could be set differently) 8080 port on the receiving machine.


and be sure to restart the apache server on the 8081 port machine ie the local NAT machine after all config changes.


To test if a port is accessible, eg port 80:


you can use


nc -l 80


this starts a simple process to receive traffic on port 80:


root@gemini:/etc/ssh# nc -l 80


you can then test it from another machine:


root@asus:~# telnet kevwells.com 80
Connected to kevwells.com.
Escape character is ‘^]’.



(you have to terminate the nc process in order to exit the telnet session)



You need root to listen on ports ≤1024, since you’re using -R and not -L the lack of permission is at the far end connection, thus running the SSH command as root won’t help.


If you use -R 8080:localhost:80 it will run, but obviously not listening on the standard http port. 


However, attempting to do  


ssh -R 80: kevin@kevwells.com


will give the error message:


Warning: remote port forwarding failed for listen port 80

This is because you can’t bind on <1024 without using sudo.  OpenSSH will refuse to bind to privileged ports unless the user id of the logged in user is 0 ie root.


Continue Reading

SSH Security Hardening Notes

SSH is a favorite attack vector among hackers.


To harden ssh and protect against ssh-hacker attempts, make following modifications to sshd_config:


root@gemini:/etc/ssh# cat sshd_config

PermitRootLogin no

#Port 22
#AddressFamily any

#we only allow access from our zonetier-one vpn  (IP commented out here for security reasons):

ListenAddress 10.******  

# Disable password authentication forcing use of keys only to login:
PasswordAuthentication no

Then restart sshd.


From then on ssh logins can only be made from inside the Zonetier VPN.  And they cannot in any case be done using root, nor with password-entry logins, only with ssh keys.


Users must first of all transfer their ssh keys from their client to the server either using ssh-copy-id if authorized, or alternatively copy-pasting their id_rsa.pub to the authorized_keys file located in their /home/<user>/.ssh directory.


This means non-users do not have any admittance to the server.


In addition to these measures, I also installed and activated fail2ban and reviewed all ports in the ufw /iptables firewalling.


Continue Reading

How to Install the SSLH Multiplexer

My server is using the sslh multiplexer daemon, this uses incoming port 443 for both ssh and https.


Reason for this is to avoid ssh connection problems when trying to connect to the server from outgoing routers which do not permit outgoing ssh port 22 connections.


If you have administrator access to the router you can modify this, but if you don’t have access, eg when using a router in a residential or commercial building complex to which you don’t have admin access yourself, then a viable workaround is to use port 443 for outgoing ssh connections.


This is because port 443 is hardly ever blocked by routers and can thus be relied upon to be accessible.


On my server, incoming port 443 ssh connections are therefore redirected to sshd on port 22, while incoming https 443 connections are redirected to https port 444 on apache.


Apache must then be configured to listen on port 444 instead of the default 443.



sslh accepts connections on specified ports, and forwards them further based on tests performed on the first data packet sent by the remote client.


sslh acts as a protocol demultiplexer, or a switchboard. Its name comes from its original function to serve SSH and HTTPS on the same port.


First, install sslh:


apt install sslh


then in /etc/default/sslh






and set the DAEMON_OPS to use the desired port for sslh and to forward to ssh:


Here we want incoming ssh connections to come in on port 444, and forward ssh calls to 22 and all other calls (which will be https for apache) to be forwarded to 444:


DAEMON_OPTS=”–user sslh –listen –ssh –ssl –pidfile /var/run/sslh/sslh.pid”


so it will look like this:


root@gemini:/# cat /etc/default/sslh
# Default options for sslh initscript
# sourced by /etc/init.d/sslh

# Disabled by default, to force yourself
# to read the configuration:
# – /usr/share/doc/sslh/README.Debian (quick start)
# – /usr/share/doc/sslh/README, at “Configuration” section
# – sslh(8) via “man sslh” for more configuration details.
# Once configuration ready, you *must* set RUN to yes here
# and try to start sslh (standalone mode only)


# binary to use: forked (sslh) or single-thread (sslh-select) version
# systemd users: don’t forget to modify /lib/systemd/system/sslh.service

#DAEMON_OPTS=”–user sslh –listen <change-me>:443 –ssh –ssl –pidfile /var/run/sslh/sslh.pid”
DAEMON_OPTS=”–user sslh –listen –ssh –ssl –pidfile /var/run/sslh/sslh.pid”


then config systemctl to autostart sslh, then start sslh:


systemctl enable sslh
systemctl start sslh


Check that its running and listening correctly:


root@gemini:/# ps -ef | grep sslh
sslh 611 1 0 Jun28 ? 00:00:00 /usr/sbin/sslh –foreground –user sslh –listen 443 –ssh 22 –tls 444 –pidfile /var/run/sslh/sslh.pid
sslh 612 611 0 Jun28 ? 00:00:00 /usr/sbin/sslh –foreground –user sslh –listen 443 –ssh 22 –tls 444 –pidfile /var/run/sslh/sslh.pid
sslh 937 612 0 Jun28 ? 00:00:00 /usr/sbin/sslh –foreground –user sslh –listen 443 –ssh 22 –tls 444 –pidfile /var/run/sslh/sslh.pid
sslh 1073 612 0 Jun28 ? 00:00:00 /usr/sbin/sslh –foreground –user sslh –listen 443 –ssh 22 –tls 444 –pidfile /var/run/sslh/sslh.pid
sslh 9093 612 0 19:22 ? 00:00:00 /usr/sbin/sslh –foreground –user sslh –listen 443 –ssh 22 –tls 444 –pidfile /var/run/sslh/sslh.pid
root 9503 9199 0 20:00 pts/2 00:00:00 grep –color=auto sslh
root@gemini:/# netstat -tulpn | grep sslh
tcp 0 0* LISTEN 611/sslh


then, you can access ssh by using:


ssh -p 443 username@server-ip


and sslh will forward the ssh connection to sshd on port 22 on the server


Don’t forget also you must reconfigure apache to listen on port 444 (/etc/apache2/ports.conf and the appropriate sites-enabled conf file – and restart apache after modifying the files).



sslh Syntax

sslh [ -t num ] [-p listening address] [-l target address for SSL] [-s target address for SSH] [-u username] [-P pidfile] [-v] [-i] [-V] [-f]


-t num
Timeout before a connection is considered to be SSH. Default is 2s.
-p listening address
Interface and port on which to listen, e.g. foobar:443, where foobar is the name of an interface (typically the IP address on which the Internet connection ends up).

Defaults to (listen to port 443 on all available interfaces).
-l target address for SSL
Interface and port on which to forward SSL connection, typically localhost:443.

Defaults to localhost:443 (this assumes you would configure your httpd process to listen to port 443).

Note that you can set sslh to listen on ext_ip:443 and httpd to listen on localhost:443: this allows clients inside your network to just connect directly to httpd.
-s target address for SSH
Interface and port on which to forward SSH connection, defaults to localhost:22.
Increase verboseness.
Prints sslh version.
-u username
Requires to run under the specified username. Defaults to nobody (which is not perfect — ideally sslh should run under its own UID).
-P pidfile
Specifies the file in which to write the PID of the main server. Defaults to /var/run/sslh.pid.
Runs as an inetd server. Options -P (PID file), -p (listen address), -u (user) are ignored.
Runs in foreground. The server will not fork and will remain connected to the terminal. Messages normally sent to syslog will also be sent to stderr.



root@gemini:/etc# systemctl status sslh
● sslh.service – SSL/SSH multiplexer
Loaded: loaded (/lib/systemd/system/sslh.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-03-22 14:58:58 UTC; 17min ago
Docs: man:sslh(8)
Main PID: 823 (sslh)
Tasks: 2 (limit: 2274)
Memory: 812.0K
CGroup: /system.slice/sslh.service
├─823 /usr/sbin/sslh –foreground ”–user sslh –listen 443 –ssh 22 –tls 4>
└─824 /usr/sbin/sslh –foreground ”–user sslh –listen 443 –ssh 22 –tls 4>



root@gemini:/etc# netstat -tulpn | grep sslh
tcp 0 0* LISTEN 823/sslh




A Note Regarding Apache and sslh


Note also that some programs, such as Lets Encrypt’s SSL Certbot SSL certificate sourcing program automatically define the https port as the default 443, so you then need to remove this and set it to 444, otherwise apache will not start.


Apache ports.conf needs to look like this. Note that port 443 is not used:


root@gemini:/etc/apache2# cat ports.conf
# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default.conf

Listen 80

<IfModule mod_ssl.c>
#Listen 443
Listen 444


The sites-enabled will also use 444 instead of 443 for virtual host definitions:



ServerName kevwells.com
ServerAlias www.kevwells.com
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined


Include /etc/letsencrypt/options-ssl-apache.conf


SSLCertificateFile /etc/letsencrypt/live/kevwells.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/kevwells.com/privkey.pem





Continue Reading