How Can We Help?
LPIC3 DIPLOMA Linux Clustering – LAB NOTES: Lesson VLANs
LAB on VLANs
These are my notes made during my lab practical as part of my LPIC3 Diploma course in Linux Clustering. They are in “rough format”, presented as they were written.
LPIC3 Syllabus for VLANs
364.4 Network High Availability
Weight: 5
Description: Candidates should be able to configure redundant networking connections and manage VLANs.
Furthermore, candidates should have a basic understanding of BGP.
Key Knowledge Areas:
• Understand and configure bonding network interface
• Network bond modes and algorithms (active-backup, blance-tlb, balance-alb,
802.3ad, balance-rr, balance-xor, broadcast)
• Configure switch configuration for high availability, including RSTP
• Configure VLANs on regular and bonded network interfaces
• Persist bonding and VLAN configuration
• Understand the principle of autonomous systems and BGP to manage external
redundant uplinks
• Awareness of traffic shaping and control capabilities of Linux
Partial list of the used files, terms and utilities:
• bonding.ko (including relevant module options)
• /etc/network/interfaces
• /etc/sysconfig/networking-scripts/ifcfg-*
• /etc/systemd/network/*.network
• /etc/systemd/network/*.netdev
• nmcli
• /sys/class/net/bonding_masters
• /sys/class/net/bond*/bonding/miimon
• /sys/class/net/bond*/bonding/slaves
• ifenslave
• ip
Cluster Overview
The cluster comprises four nodes installed with CentOS 7 and housed on a KVM virtual machine system on a Linux Ubuntu host.
For this lab I am creating a vlan called vlan-1, for just two machines, ie:
ceph-mon
ceph-osd0
NOTE: You do NOT need to create a new physical NAT network on KVM, since the VLAN subnet is purely virtual.
VLAN Tagging
Each VLAN is identified by a VID (VLAN Identifier) between 1 and 4094 inclusive. Ports on switches are assigned to a VLAN ID.
All ports assigned to a single VLAN are virtually located in their own separate broadcast domain. This reduces network traffic overhead.
The VID is stored in a 4-byte header that gets added to the packet, known as the Tag. Hence the name for this procedure is VLAN tagging.
Configuring VLAN Tagging Using nmcli
First ensure the 802.1Q kernel module is loaded. In practice, this module is often automatically loaded if you configure a VLAN subinterface.
This is the procedure to manually load it:
[root@ceph-mon ~]# modprobe 8021q
[root@ceph-mon ~]#
[root@ceph-mon ~]# lsmod | grep 8021q
8021q 33080 0
garp 14384 1 8021q
mrp 18542 1 8021q
[root@ceph-mon ~]#
1. You can use the nmcli connection command to create a VLAN connection.
Include the “add type vlan” arguments and any additional information to create a VLAN connection. For example:
[root@ceph-mon network-scripts]# nmcli con add type vlan con-name vlan-1 ifname eth0.100 dev eth0 id 100 ip4 192.168.133.40/24
Connection ‘vlan-1’ (25a01a92-740b-481e-8c88-033d6ace0227) successfully added.
[root@ceph-mon network-scripts]#
note we create a NEW ifname with eg eth0.100
nmcli con add type vlan con-name vlan-1 ifname eth0.100 dev eth0 id 100 ip4 192.168.133.40/24
The example defines the following attributes of the VLAN connection:
con-name vlan-1: Specifies the name of the new VLAN connection
ifname eth0.100: Specifies the VLAN interface to bind the connection to
dev eth0: Specifies the actual physical (parent) device this VLAN is on
id 100: Specifies the VLAN ID
ip4 192.168.133.1/24: Specifies IPv4 address to assign to the interface
This command automatically generates the respective network interface config file for the VLAN, so it is preferred to the manual config file method which is documented further below.
The nmcli con command shows the new VLAN connection.
# nmcli connection
[root@ceph-mon network-scripts]# nmcli connection
NAME UUID TYPE DEVICE
Wired connection 1 70ed8ab9-f6e1-3180-8d1b-b7c3cb827c8c ethernet eth3
eth0 d1840d20-4b54-49b7-8eb8-305bd11aa5eb ethernet eth0
vlan-1 25a01a92-740b-481e-8c88-033d6ace0227 vlan eth0.100
[root@ceph-mon network-scripts]#
this also creates the config file:
/etc/sysconfig/network-scripts/ifcfg-vlan-1
check with:
[root@ceph-mon network-scripts]# ls /sys/class/net
bond0 bonding_masters eth0 eth0.100 eth1 eth2 eth3 lo
[root@ceph-mon network-scripts]#
and
[root@ceph-mon network-scripts]# cat ifcfg-vlan-1
VLAN=yes
TYPE=Vlan
PHYSDEV=eth0
VLAN_ID=100
REORDER_HDR=yes
GVRP=no
MVRP=no
HWADDR=
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=192.168.133.40
PREFIX=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=vlan-1
UUID=25a01a92-740b-481e-8c88-033d6ace0227
DEVICE=eth0.100
ONBOOT=yes
[root@ceph-mon network-scripts]#
Manual Configuration of Network Interface File for VLAN Tagging
To manually create the network interface file for the VLAN you need to specify the interface name in the form of parentInterface.vlanID.
This associates the VLAN with the appropriate parent network interface. Also set the VLAN=yes directive to define this subinterface as a VLAN.
Then restart the network.
[root@ceph-mon network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-vlan-1
VLAN=yes
TYPE=Vlan
PHYSDEV=eth0
VLAN_ID=100
REORDER_HDR=yes
GVRP=no
MVRP=no
HWADDR=
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=192.168.133.40
PREFIX=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=vlan-1
UUID=25a01a92-740b-481e-8c88-033d6ace0227
DEVICE=eth0.100
ONBOOT=yes
[root@ceph-mon network-scripts]#
To delete a wifi connection type :
nmcli connection delete id <connection name>
nmcli connection delete id vlan-1
[root@ceph-mon network-scripts]# nmcli connection delete id vlan-1
Connection ‘vlan-1’ (56c10845-07a6-4245-bc95-24c17e991082) successfully deleted.
[root@ceph-mon network-scripts]#
How to Verify the VLAN Connection
[root@ceph-mon network-scripts]# ip add show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:93:ca:03 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.40/24 brd 192.168.122.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::6e18:9a8a:652c:1700/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::127d:ea0d:65b7:30e5/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::4ad9:fabb:aad4:9468/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
link/ether 52:54:00:d7:a5:b0 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
link/ether 52:54:00:d7:a5:b0 brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:22:42:1e brd ff:ff:ff:ff:ff:ff
inet6 fe80::5b5f:1ce3:13:7a74/64 scope link noprefixroute
valid_lft forever preferred_lft forever
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:d7:a5:b0 brd ff:ff:ff:ff:ff:ff
inet 10.0.9.45/24 brd 10.0.9.255 scope global bond0
valid_lft forever preferred_lft forever
inet6 fe80::5054:ff:fed7:a5b0/64 scope link
valid_lft forever preferred_lft forever
7: eth0.100@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:93:ca:03 brd ff:ff:ff:ff:ff:ff
inet 192.168.133.40/24 brd 192.168.133.255 scope global noprefixroute eth0.100
valid_lft forever preferred_lft forever
inet6 fe80::d5c6:9aa5:6996:1635/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@ceph-mon network-scripts]#
Note in the above, we can see the newly created vlan interface:
7: eth0.100@eth0:
[root@ceph-mon network-scripts]# nmcli connection
NAME UUID TYPE DEVICE
Wired connection 1 70ed8ab9-f6e1-3180-8d1b-b7c3cb827c8c ethernet eth3
eth0 d1840d20-4b54-49b7-8eb8-305bd11aa5eb ethernet eth0
vlan-1 25a01a92-740b-481e-8c88-033d6ace0227 vlan eth0.100
[root@ceph-mon network-scripts]# nmcli device
DEVICE TYPE STATE CONNECTION
eth0 ethernet connected eth0
eth0.100 vlan connected vlan-1
eth3 ethernet disconnected —
bond0 bond unmanaged —
eth1 ethernet unmanaged —
eth2 ethernet unmanaged —
lo loopback unmanaged —
[root@ceph-mon network-scripts]#
we can also do:
ls /sys/class/net/eth0.100
[root@ceph-mon network-scripts]# ls /sys/class/net/eth0.100
addr_assign_type broadcast dev_id duplex ifalias link_mode netdev_group phys_port_name proto_down statistics type
address carrier dev_port flags ifindex lower_eth0 operstate phys_switch_id queues subsystem uevent
addr_len carrier_changes dormant gro_flush_timeout iflink mtu phys_port_id power speed tx_queue_len
[root@ceph-mon network-scripts]#
and
ls /proc/net/vlan
[root@ceph-mon network-scripts]# ls /proc/net/vlan
config eth0.100
[root@ceph-mon network-scripts]#
Configuring Further VLAN Member Nodes
I then created a VLAN interface on node ceph-osd0 as follows, so that the two nodes (ceph-mon and ceph-osd0) can communicate via the VLAN:
[root@ceph-osd0 ~]#
[root@ceph-osd0 ~]# modprobe 8021q
[root@ceph-osd0 ~]# lsmod | grep 8021q
8021q 33080 0
garp 14384 1 8021q
mrp 18542 1 8021q
[root@ceph-osd0 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.122.50 netmask 255.255.255.0 broadcast 192.168.122.255
inet6 fe80::127d:ea0d:65b7:30e5 prefixlen 64 scopeid 0x20<link>
inet6 fe80::6e18:9a8a:652c:1700 prefixlen 64 scopeid 0x20<link>
inet6 fe80::4ad9:fabb:aad4:9468 prefixlen 64 scopeid 0x20<link>
ether 52:54:00:03:66:58 txqueuelen 1000 (Ethernet)
RX packets 40679 bytes 2147951 (2.0 MiB)
RX errors 0 dropped 39457 overruns 0 frame 0
TX packets 817 bytes 54247 (52.9 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.9.10 netmask 255.0.0.0 broadcast 10.255.255.255
inet6 fe80::9a5f:c1fc:8228:8d16 prefixlen 64 scopeid 0x20<link>
inet6 fe80::61d0:9d9f:ccc3:9f2e prefixlen 64 scopeid 0x20<link>
inet6 fe80::c466:3844:d978:b3d8 prefixlen 64 scopeid 0x20<link>
ether 52:54:00:a2:a4:1d txqueuelen 1000 (Ethernet)
RX packets 181745 bytes 11234531 (10.7 MiB)
RX errors 0 dropped 39454 overruns 0 frame 0
TX packets 130505 bytes 1040879191 (992.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 24888 bytes 2206620 (2.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 24888 bytes 2206620 (2.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@ceph-osd0 ~]# nmcli con add type vlan con-name vlan-1 ifname eth0.100 dev eth0 id 100 ip4 192.168.133.41/24
Connection ‘vlan-1’ (6c39b373-e1f5-46c2-9137-768f53e5ed22) successfully added.
[root@ceph-osd0 ~]# nmcli connection
NAME UUID TYPE DEVICE
eth0 d1840d20-4b54-49b7-8eb8-305bd11aa5eb ethernet eth0
eth1 9c92fad9-6ecb-3e6c-eb4d-8a47c6f50c04 ethernet eth1
vlan-1 6c39b373-e1f5-46c2-9137-768f53e5ed22 vlan eth0.100
[root@ceph-osd0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-vlan-1
VLAN=yes
TYPE=Vlan
PHYSDEV=eth0
VLAN_ID=100
REORDER_HDR=yes
GVRP=no
MVRP=no
HWADDR=
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=192.168.133.41
PREFIX=24
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=vlan-1
UUID=6c39b373-e1f5-46c2-9137-768f53e5ed22
DEVICE=eth0.100
ONBOOT=yes
[root@ceph-osd0 ~]# ip add show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:03:66:58 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.50/24 brd 192.168.122.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet6 fe80::6e18:9a8a:652c:1700/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::127d:ea0d:65b7:30e5/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::4ad9:fabb:aad4:9468/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:a2:a4:1d brd ff:ff:ff:ff:ff:ff
inet 10.0.9.10/8 brd 10.255.255.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
inet6 fe80::c466:3844:d978:b3d8/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::61d0:9d9f:ccc3:9f2e/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::9a5f:c1fc:8228:8d16/64 scope link tentative noprefixroute dadfailed
valid_lft forever preferred_lft forever
4: eth0.100@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:03:66:58 brd ff:ff:ff:ff:ff:ff
inet 192.168.133.41/24 brd 192.168.133.255 scope global noprefixroute eth0.100
valid_lft forever preferred_lft forever
inet6 fe80::497:afcc:dfdd:bafb/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@ceph-osd0 ~]# ping 192.168.133.40
PING 192.168.133.40 (192.168.133.40) 56(84) bytes of data.
64 bytes from 192.168.133.40: icmp_seq=1 ttl=64 time=1.05 ms
64 bytes from 192.168.133.40: icmp_seq=2 ttl=64 time=0.543 ms
64 bytes from 192.168.133.40: icmp_seq=3 ttl=64 time=0.577 ms
^C
— 192.168.133.40 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.543/0.724/1.052/0.232 ms
[root@ceph-osd0 ~]#
I also created a VLAN interface to the vlan-1 VLAN on my laptop (ubuntu):
Note the interface name is derived from KVM as we are in a KVM virtualized environment. The parent interface is virbr0 and this is the 192.168.122.0 connection to the cluster on KVM from the laptop.
The VLAN interface “piggybacks” via virbr0 as virbr0.100 with subnet 192.168.133.0
(there is no KVM defined subnet for the 192.168.133.0 – it is purely VLAN virtual)
root@asus:/home/kevin#
root@asus:/home/kevin# nmcli con add type vlan con-name vlan-1 ifname virbr0.100 dev virbr0 id 100 ip4 192.168.133.1/24
Connection ‘vlan-1’ (e2f09575-95d1-4028-b99b-eb49300bf8b2) successfully added.
root@asus:/home/kevin# nmcli con
NAME UUID TYPE DEVICE
vlan-1 e2f09575-95d1-4028-b99b-eb49300bf8b2 vlan virbr0.100
root@asus:/etc/netplan# ip add show | grep virbr0
3: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
9: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr0 state UNKNOWN group default qlen 1000
11: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr0 state UNKNOWN group default qlen 1000
13: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr0 state UNKNOWN group default qlen 1000
17: vnet6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr0 state UNKNOWN group default qlen 1000
23: virbr0.100@virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.133.1/24 brd 192.168.133.255 scope global noprefixroute virbr0.100
root@asus:/etc/netplan#
root@asus:/etc/netplan# ls /proc/net/vlan
config virbr0.100
root@asus:/etc/netplan#