Tags Archives: ssh

SSH Reverse Tunnel – A Practical Example – Public Internet Server to Client Laptop on NAT WiFi

Instructions for setting up an ssh reverse tunnel from my laptop to my internet server.

 

This was necessary as I could not connect to my NFS shares on gemini from my laptop when using external wifi as my server denies connections that are not made via my private VPN network. This is for security reasons to prevent third-party access to my NFS server shares.

 

 

 

However, I can ssh in to the gemini server.  So I set up a reverse SSH tunnel.  I did the following:

 

 

on laptop:

 

ssh -f -N -T -R22000:localhost:22 gemini

 

I can then copy files through the tunnel using the port 22000 address on gemini…

 

example: 

 

 
root@gemini:~# scp -P 22000 filefromgemini gemini:/tmp
root@gemini’s password:
filefromgemini 100% 47 3.8KB/s 00:00
root@gemini:~#

 

I want to copy the WordPress databases from gemini to laptop:

 

on gemini do:

 

wp cache flush –path=/var/www/wordpress –allow-root

 

# create the database export from server kevwells.com:

 

mysqldump –all-databases > /home/kevin/all_databases.sql

 

 

then scp the .sql file to port 22000 on gemini – this is the ssh reverse tunnel entrance to my laptop!

 

scp -P 22000 /home/kevin/all_databases.sql gemini:/home/kevin/

 

(the “gemini:/home/kevin” in this case is actually my /home/kevin on the laptop! – ie the other end ie exit of the tunnel)

 

 

so, to summarize we do:

 

on laptop client:

 

root@asus:/usr/local/bin#
root@asus:/usr/local/bin# ssh -f -N -T -R22000:localhost:22 gemini
root@asus:/usr/local/bin#

 

on gemini server:

 

root@gemini:~#
root@gemini:~# wp cache flush –path=/var/www/wordpress –allow-root
Success: The cache was flushed.
root@gemini:~#
root@gemini:~# mysqldump –all-databases > /home/kevin/all_databases.sql

 

root@gemini:/home/kevin# ls
all_databases.sql BACKUP DATA Downloads NEXTCLOUD readme.locationofmyphp.ini
root@gemini:/home/kevin#
root@gemini:/home/kevin#
root@gemini:/home/kevin# scp -P 22000 /home/kevin/all_databases.sql gemini:/home/kevin/
root@gemini’s password:
all_databases.sql 100% 28MB 2.0MB/s 00:14
root@gemini:/home/kevin#

 

then on laptop, sure enough the file has arrived:

 

root@asus:/home/kevin# ll
total 253272
drwxrwxr-x 34 kevin kevin 16384 Apr 21 16:02 ./
drwxr-xr-x 3 root root 4096 Aug 28 2021 ../
-rw-r–r– 1 root root 29555359 Apr 21 16:02 all_databases.sql

 

 

Mission accomplished.

 

 

Continue Reading

How To Set Up An SSH Reverse Tunnel

Reverse SSH tunneling, otherwise known as remote port forwarding via SSH. This is a way to connect to a machine by having the other computer call you first and then activating a second connection over the initial one in the opposite direction.

 

How SSH Remote Tunneling Works

 

To connect via SSH or some other service, such as HTTP, from the Internet into a machine behind a firewall, you need the machine in question to open an SSH connection to a machine outside the firewall and include an SSH -R tunnel whose “entry” point is the “remote” side of this connection.

 

This means, from the firewalled host, you need to execute something like:

 

ssh -f -N -T -R22000:localhost:22 yourpublichost.example.com

 

This instructs your client machine to establish a tunnel with an SSH -R remote entry point.

 

Anything that then attaches to port 22000 at the other end of the tunnel will be automatically forwarded to “localhost port 22”, where “localhost” is defined from the point of view of the exit point of the tunnel – namely your SSH client machine.

 

 

The other options here are:

 

-f instructs SSH to background itself after authentication. T
-N instructs SSH to create a connection, but without actually running any remote commands. This saves processing resources and time.
-T instructs SSH to disable pseudo-tty allocation, which is useful if you do not want an interactive command line shell.

 

Standard local port forwarding does not work if incoming SSH requests in the remote server are disabled. For security reasons, administrators might entirely block inbound SSH requests but allow outbound SSH requests.

 

In such situations, you can use remote port forwarding to create an outbound SSH connection and let the clients connect to the local port even if inbound connections are blocked.

 

 

Now, when users from distant internet visit port 80 of the remote server as http://<remote_server_ip>, the request is redirected back to the client’s local server (port 3000) via SSH tunnel where the local server handles the request and response.

 

By default, the remote port forwarding tunnel will bind to the localhost of the remote server. To enable it to listen on the public interface (for a scenario like above), set the SSH configuration GatewayPorts yes in sshd_config.

 

 

This is the command, set from the outgoing machine, in this case my laptop:

 

ssh -NR 8080:localhost:8081 kevin@kevwells.com

 

it will not return to the command line – that is correct behaviour.

 

To run a webserver over the tunnel you need to set it up to run on eg port 8081. You then set the tunnel to run on eg 8080.

 

Configure sshd_config

 

You need to set the following directives in /etc/ssh/sshd_config:

 

AllowTcpForwarding yes
GatewayPorts yes

 

Make sure you restart ssh service after editing the file: ie

 

systemctl restart ssh

 

You can then connect to the asus website using http://kevwells.com:8080

 

Make sure apache is running on 8081 on the localhost.

 

Points to Watch with SSH Reverse Tunneling

 

This only works here with http.

 

https will NOT work – since one cannot obtain an SSL/TLS certificate for a local NAT IP address – only for publicly accessible IPs ie domain names accessible without NAT.

 

 

By default, TCP listening sockets on the server are bound only to the loopback interface. This can be overridden by specifying a bind_address. An empty bind_address, or the address *, directs that the remote socket should listen on all interfaces.

 

 

If you specify a remote bind_address then this will only function if the server’s GatewayPorts option is enabled in sshd_config.

 

In addition, the GatewayPorts directive in the sshd_config on the server must be set.

 

To quote from the sshd_config man page:

 

This specifies whether remote hosts are allowed to connect to ports forwarded for the client. By default, sshd(8) binds remote port forwardings to the loopback address. This prevents other remote hosts from connecting to forwarded ports. GatewayPorts can be used to specify that sshd should allow remote port forwardings to bind to non-loopback addresses, thus allowing other hosts to connect. The argument may be no to force remote port forwardings to be available to the local host only, yes to force remote port forwardings to bind to the wildcard address, or client specified to allow the client to select the address to which the forwarding is bound. The default is no.

 

 

Remember to restart the ssh server service after all config changes.

 

Make sure also that all ports in use are accessible through any firewall or routers.

 

Also if you are running a local apache webserver on the NAT machine, then make sure you have the respective port open in the ports.conf at /etc/apache2/ports.conf

 

in this example

 

LISTEN 8081

 

AND set a virtual hosts definition in your active sites-enabled config file for the port you are using for the webserver on the local machine ie:

 

(this definition is on the LOCAL NAT machine apache config)

 

<VirtualHost *.:8081>

 

You do NOT need any definition for this, nor for (in this case 8080 but it could be set differently) 8080 port on the receiving machine.

 

and be sure to restart the apache server on the 8081 port machine ie the local NAT machine after all config changes.

 

To test if a port is accessible, eg port 80:

 

you can use

 

nc -l 80

 

this starts a simple process to receive traffic on port 80:

 

root@gemini:/etc/ssh# nc -l 80
^C
root@gemini:/etc/ssh#

 

you can then test it from another machine:

 

root@asus:~# telnet kevwells.com 80
Trying 78.141.200.190…
Connected to kevwells.com.
Escape character is ‘^]’.

 

 

(you have to terminate the nc process in order to exit the telnet session)

 

 

You need root to listen on ports ≤1024, since you’re using -R and not -L the lack of permission is at the far end connection, thus running the SSH command as root won’t help.

 

If you use -R 8080:localhost:80 it will run, but obviously not listening on the standard http port. 

 

However, attempting to do  

 

ssh -R 80:127.0.0.1:8081 kevin@kevwells.com

 

will give the error message:

 

Warning: remote port forwarding failed for listen port 80

This is because you can’t bind on <1024 without using sudo.  OpenSSH will refuse to bind to privileged ports unless the user id of the logged in user is 0 ie root.

 

Continue Reading

SSH Security Hardening Notes

SSH is a favorite attack vector among hackers.

 

To harden ssh and protect against ssh-hacker attempts, make following modifications to sshd_config:

 

root@gemini:/etc/ssh# cat sshd_config

PermitRootLogin no

#Port 22
#AddressFamily any

#we only allow access from our zonetier-one vpn  (IP commented out here for security reasons):

ListenAddress 10.******  

# Disable password authentication forcing use of keys only to login:
PasswordAuthentication no


Then restart sshd.

 

From then on ssh logins can only be made from inside the Zonetier VPN.  And they cannot in any case be done using root, nor with password-entry logins, only with ssh keys.

 

Users must first of all transfer their ssh keys from their client to the server either using ssh-copy-id if authorized, or alternatively copy-pasting their id_rsa.pub to the authorized_keys file located in their /home/<user>/.ssh directory.

 

This means non-users do not have any admittance to the server.

 

In addition to these measures, I also installed and activated fail2ban and reviewed all ports in the ufw /iptables firewalling.

 

Continue Reading

How to Install the SSLH Multiplexer

My server is using the sslh multiplexer daemon, this uses incoming port 443 for both ssh and https.

 

Reason for this is to avoid ssh connection problems when trying to connect to the server from outgoing routers which do not permit outgoing ssh port 22 connections.

 

If you have administrator access to the router you can modify this, but if you don’t have access, eg when using a router in a residential or commercial building complex to which you don’t have admin access yourself, then a viable workaround is to use port 443 for outgoing ssh connections.

 

This is because port 443 is hardly ever blocked by routers and can thus be relied upon to be accessible.

 

On my server, incoming port 443 ssh connections are therefore redirected to sshd on port 22, while incoming https 443 connections are redirected to https port 444 on apache.

 

Apache must then be configured to listen on port 444 instead of the default 443.

 

 

sslh accepts connections on specified ports, and forwards them further based on tests performed on the first data packet sent by the remote client.

 

sslh acts as a protocol demultiplexer, or a switchboard. Its name comes from its original function to serve SSH and HTTPS on the same port.

 

First, install sslh:

 

apt install sslh

 

then in /etc/default/sslh

 

change:

 

#RUN=no
RUN=yes

 

and set the DAEMON_OPS to use the desired port for sslh and to forward to ssh:

 

Here we want incoming ssh connections to come in on port 444, and forward ssh calls to 22 and all other calls (which will be https for apache) to be forwarded to 444:

 

DAEMON_OPTS=”–user sslh –listen 0.0.0.0:443 –ssh 127.0.0.1:22 –ssl 127.0.0.1:444 –pidfile /var/run/sslh/sslh.pid”

 

so it will look like this:

 

root@gemini:/# cat /etc/default/sslh
# Default options for sslh initscript
# sourced by /etc/init.d/sslh

# Disabled by default, to force yourself
# to read the configuration:
# – /usr/share/doc/sslh/README.Debian (quick start)
# – /usr/share/doc/sslh/README, at “Configuration” section
# – sslh(8) via “man sslh” for more configuration details.
# Once configuration ready, you *must* set RUN to yes here
# and try to start sslh (standalone mode only)

#RUN=no
RUN=yes

# binary to use: forked (sslh) or single-thread (sslh-select) version
# systemd users: don’t forget to modify /lib/systemd/system/sslh.service
DAEMON=/usr/sbin/sslh

#DAEMON_OPTS=”–user sslh –listen <change-me>:443 –ssh 127.0.0.1:22 –ssl 127.0.0.1:443 –pidfile /var/run/sslh/sslh.pid”
DAEMON_OPTS=”–user sslh –listen 0.0.0.0:443 –ssh 127.0.0.1:22 –ssl 127.0.0.1:444 –pidfile /var/run/sslh/sslh.pid”
root@gemini:/#

 

then config systemctl to autostart sslh, then start sslh:

 

systemctl enable sslh
systemctl start sslh

 

Check that its running and listening correctly:

 

root@gemini:/# ps -ef | grep sslh
sslh 611 1 0 Jun28 ? 00:00:00 /usr/sbin/sslh –foreground –user sslh –listen 0.0.0.0 443 –ssh 127.0.0.1 22 –tls 127.0.0.1 444 –pidfile /var/run/sslh/sslh.pid
sslh 612 611 0 Jun28 ? 00:00:00 /usr/sbin/sslh –foreground –user sslh –listen 0.0.0.0 443 –ssh 127.0.0.1 22 –tls 127.0.0.1 444 –pidfile /var/run/sslh/sslh.pid
sslh 937 612 0 Jun28 ? 00:00:00 /usr/sbin/sslh –foreground –user sslh –listen 0.0.0.0 443 –ssh 127.0.0.1 22 –tls 127.0.0.1 444 –pidfile /var/run/sslh/sslh.pid
sslh 1073 612 0 Jun28 ? 00:00:00 /usr/sbin/sslh –foreground –user sslh –listen 0.0.0.0 443 –ssh 127.0.0.1 22 –tls 127.0.0.1 444 –pidfile /var/run/sslh/sslh.pid
sslh 9093 612 0 19:22 ? 00:00:00 /usr/sbin/sslh –foreground –user sslh –listen 0.0.0.0 443 –ssh 127.0.0.1 22 –tls 127.0.0.1 444 –pidfile /var/run/sslh/sslh.pid
root 9503 9199 0 20:00 pts/2 00:00:00 grep –color=auto sslh
root@gemini:/# netstat -tulpn | grep sslh
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 611/sslh
root@gemini:/#

 

then, you can access ssh by using:

 

ssh -p 443 username@server-ip

 

and sslh will forward the ssh connection to sshd on port 22 on the server

 

Don’t forget also you must reconfigure apache to listen on port 444 (/etc/apache2/ports.conf and the appropriate sites-enabled conf file – and restart apache after modifying the files).

 

 

sslh Syntax

sslh [ -t num ] [-p listening address] [-l target address for SSL] [-s target address for SSH] [-u username] [-P pidfile] [-v] [-i] [-V] [-f]

OPTIONS

-t num
Timeout before a connection is considered to be SSH. Default is 2s.
-p listening address
Interface and port on which to listen, e.g. foobar:443, where foobar is the name of an interface (typically the IP address on which the Internet connection ends up).

Defaults to 0.0.0.0:443 (listen to port 443 on all available interfaces).
-l target address for SSL
Interface and port on which to forward SSL connection, typically localhost:443.

Defaults to localhost:443 (this assumes you would configure your httpd process to listen to port 443).

Note that you can set sslh to listen on ext_ip:443 and httpd to listen on localhost:443: this allows clients inside your network to just connect directly to httpd.
-s target address for SSH
Interface and port on which to forward SSH connection, defaults to localhost:22.
-v
Increase verboseness.
-V
Prints sslh version.
-u username
Requires to run under the specified username. Defaults to nobody (which is not perfect — ideally sslh should run under its own UID).
-P pidfile
Specifies the file in which to write the PID of the main server. Defaults to /var/run/sslh.pid.
-i
Runs as an inetd server. Options -P (PID file), -p (listen address), -u (user) are ignored.
-f
Runs in foreground. The server will not fork and will remain connected to the terminal. Messages normally sent to syslog will also be sent to stderr.

 

 

root@gemini:/etc# systemctl status sslh
● sslh.service – SSL/SSH multiplexer
Loaded: loaded (/lib/systemd/system/sslh.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-03-22 14:58:58 UTC; 17min ago
Docs: man:sslh(8)
Main PID: 823 (sslh)
Tasks: 2 (limit: 2274)
Memory: 812.0K
CGroup: /system.slice/sslh.service
├─823 /usr/sbin/sslh –foreground ”–user sslh –listen 0.0.0.0 443 –ssh 127.0.0.1 22 –tls 127.0.0.1 4>
└─824 /usr/sbin/sslh –foreground ”–user sslh –listen 0.0.0.0 443 –ssh 127.0.0.1 22 –tls 127.0.0.1 4>

 

 

root@gemini:/etc# netstat -tulpn | grep sslh
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 823/sslh
root@gemini:/etc

 

 

 

A Note Regarding Apache and sslh

 

Note also that some programs, such as Lets Encrypt’s SSL Certbot SSL certificate sourcing program automatically define the https port as the default 443, so you then need to remove this and set it to 444, otherwise apache will not start.

 

Apache ports.conf needs to look like this. Note that port 443 is not used:

 

root@gemini:/etc/apache2# cat ports.conf
# If you just change the port or add more ports here, you will likely also
# have to change the VirtualHost statement in
# /etc/apache2/sites-enabled/000-default.conf

Listen 80

<IfModule mod_ssl.c>
#Listen 443
Listen 444
</IfModule>

 

The sites-enabled will also use 444 instead of 443 for virtual host definitions:

 

<VirtualHost 127.0.0.1:444>

ServerName kevwells.com
ServerAlias www.kevwells.com
DocumentRoot /var/www/html
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

 

Include /etc/letsencrypt/options-ssl-apache.conf

 

SSLCertificateFile /etc/letsencrypt/live/kevwells.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/kevwells.com/privkey.pem

 

</VirtualHost>
</IfModule>

 

 

Continue Reading