Proxmox part 2: Networking

Updated November 23, 2023 16 minutes

In part 1, we installed a fresh Proxmox server, configured SSH, updated repositories, and configured a basic node firewall.

In this post, we will continue setting up our Proxmox server’s network.

Use bridge networking for private networks

By default, Proxmox uses bridge networking, which is very simple to setup, assuming you already have a LAN and an existing DHCP server and gateway on it. With bridge networking, you can use a single network interface with all of the virtual machines accessible through it. Each VM will have a unique virtual MAC address, and each will receive a unique IP address from your DHCP server. All the VMs become discoverable on the network, just like any other machine on your LAN.

By default, Proxmox creates a single bridge network, named vmbr0, and this is connected to the management interface, and it is assumed you will connect this to your LAN.

The default Bridge network, vmbr0
The default Bridge network, vmbr0

If you are able to use bridge networking, great! Nothing more to do here. If you have run out of IP addresses though, read on!

Use Network Address Translation (NAT) if you have limited IP addresses

If you are deploying to the internet, you likely have only a finite number of IP addresses, or possibly, only one.

When you have limited IP addresses, you can use Network Address Translation (NAT) to let several virtual machines access the network using the same IP address. Source NAT (SNAT) or IP Masquerading provides private VMs outbound/egress to the WAN/internet. Inbound/ingress “port forwarding” is called destintation NAT (DNAT), and with this you have multiple servers all on one IP address, but each using a unique port number from the host, forwarded directly to the private service.

Proxmox has no builtin support, through the dashboard, for configuring either kinds of NAT. However, because Proxmox is based on Debian Linux, we can configure the NAT rules with iptables through the command line.

If you want to enable NAT, here are the steps:

  • Connect to the pve node through SSH, logging in as the root user.
  • Download the proxmox_nat.sh configuraton script, and make it executable:
wget https://raw.githubusercontent.com/EnigmaCurry/blog.rymcg.tech/master/src/proxmox/proxmox_nat.sh

chmod +x proxmox_nat.sh
  • The script has a menu interface:
NAT bridge tool:
 * Type `i` or `interfaces` to list the bridge interfaces.
 * Type `c` or `create` to create a new NAT bridge.
 * Type `l` or `list` to list the NAT rules.
 * Type `n` or `new` to create some new NAT rules.
 * Type `d` or `delete` to delete some existing NAT rules.
 * Type `?` or `help` to see this help message again.
 * Type `q` or `quit` to quit.
  • Type i (and press Enter) to list all the bridge interfaces (you can see I have created some additional NAT bridges already):
Enter command (for help, enter `?`)
: i

Currently configured bridges:
BRIDGE  NETWORK         COMMENT
vmbr0   10.13.13.11/24
vmbr1   10.99.0.2/24
vmbr50  172.16.13.2/24  pfsense MGMT only
vmbr55  10.55.0.1/24    NAT 10.55.0.1/24 bridged to vmbr0
  • Type c (and press Enter) to create a new interface:
Enter command (for help, enter `?`)
: c

Configuring new NAT bridge ...
Enter the existing bridge to NAT from
: vmbr0
Enter a unique number for the new bridge (dont write the vmbr prefix)
: 56

Configuring new interface: vmbr56
Enter the static IP address and network prefix in CIDR notation for vmbr56:
: 10.56.0.1/24

## DEBUG: IP_CIDR=10.56.0.1/24

Enter the description/comment for this interface
: NAT 10.56.0.1/24 bridged to vmbr0
Wrote /etc/network/interfaces
Activated vmbr56
  • Type i again and see the new interface vmbr56 has been created:
Enter command (for help, enter `?`)
: i

Currently configured bridges:
BRIDGE  NETWORK         COMMENT
vmbr0   10.13.13.11/24
vmbr1   10.99.0.2/24
vmbr50  172.16.13.2/24  pfsense MGMT only
vmbr55  10.55.0.1/24    NAT 10.55.0.1/24 bridged to vmbr0
vmbr56   10.56.0.1/24   NAT 10.56.0.1/24 bridged to vmbr0
  • Type n to create create a new NAT rule:
Enter command (for help, enter `?`)
: n

Defining new port forward rule:
Enter the inbound interface
: vmbr0
Enter the protocol (tcp, udp)
: tcp
Enter the inbound Port number
: 2222
Enter the destination IP address
: 10.56.0.2
Enter the destination Port number
: 22
INTERFACE  PROTOCOL  IN_PORT  DEST_IP    DEST_PORT
vmbr0      tcp       2222     10.56.0.2  22
? Is this rule correct? (Y/n): y

? Would you like to define more port forwarding rules now? (y/N): n
Wrote /etc/network/my-iptables-rules.sh
Systemd unit already enabled: my-iptables-rules
NAT rules applied: /etc/network/my-iptables-rules.sh

## Existing inbound port forwarding (DNAT) rules:
INTERFACE  PROTOCOL  IN_PORT  DEST_IP    DEST_PORT
vmbr0      tcp       2222     10.56.0.2  22
  • Type l to list the current NAT rules, showing the rule you just added:
Enter command (for help, enter `?`)
: l

## Existing inbound port forwarding (DNAT) rules:
INTERFACE  PROTOCOL  IN_PORT  DEST_IP    DEST_PORT
vmbr0      tcp       2222     10.56.0.2  22
  • Type d to delete an existing NAT rule:
Enter command (for help, enter `?`)
: d

LINE#  INTERFACE  PROTOCOL  IN_PORT  DEST_IP    DEST_PORT
1      vmbr0      tcp       2222     10.56.0.2  22
Enter the line number for the rule you wish to delete (type `q` or blank for none)
: 1
Wrote /etc/network/my-iptables-rules.sh
Systemd unit already enabled: my-iptables-rules
NAT rules applied: /etc/network/my-iptables-rules.sh
No inbound port forwarding (DNAT) rules have been created yet.
  • Type e to enable or disable the systemd service that manages these rules:
Enter command (for help, enter `?`)
: e

The systemd unit is named: my-iptables-rules
The systemd unit is currently: enabled
? Would you like to enable the systemd unit on boot? (Y/n): y
Systemd unit enabled: my-iptables-rules
NAT rules applied: /etc/network/my-iptables-rules.sh

The script

#!/bin/bash

## Setup NAT (IP Masquerading egress + Port Forwarding ingress) on Proxmox
## See https://blog.rymcg.tech/blog/proxmox/02-networking/

SYSTEMD_UNIT="my-iptables-rules"
SYSTEMD_SERVICE="/etc/systemd/system/${SYSTEMD_UNIT}.service"
IPTABLES_RULES_SCRIPT="/etc/network/${SYSTEMD_UNIT}.sh"

## Default network address is for a /24 based on the the bridge number:
## (Change the prefix [10.1] per install, to create unique addresses):
DEFAULT_NETWORK_PATTERN="10.1.BRIDGE.1/24"

set -eo pipefail
stderr(){ echo "$@" >/dev/stderr; }
error(){ stderr "Error: $@"; }
cancel(){ stderr "Canceled."; exit 2; }
fault(){ test -n "$1" && error $1; stderr "Exiting."; exit 1; }
print_array(){ printf '%s\n' "$@"; }
trim_trailing_whitespace() { sed -e 's/[[:space:]]*$//'; }
trim_leading_whitespace() { sed -e 's/^[[:space:]]*//'; }
trim_whitespace() { trim_leading_whitespace | trim_trailing_whitespace; }
confirm() {
    ## Confirm with the user.
    local default=$1; local prompt=$2; local question=${3:-". Proceed?"}
    if [[ $default == "y" || $default == "yes" || $default == "ok" ]]; then
        dflt="Y/n"
    else
        dflt="y/N"
    fi
    read -e -p $'\e[32m?\e[0m '"${prompt}${question} (${dflt}): " answer
    answer=${answer:-${default}}
    if [[ ${answer,,} == "y" || ${answer,,} == "yes" || ${answer,,} == "ok" ]]; then
        return 0
    else
        return 1
    fi
}
ask() {
    local __prompt="${1}"; local __var="${2}"; local __default="${3}"
    while true; do
        read -e -p "${__prompt}"$'\x0a\e[32m:\e[0m ' -i "${__default}" ${__var}
        export ${__var}
        [[ -z "${!__var}" ]] || break
    done
}
ask_allow_blank() {
    local __prompt="${1}"; local __var="${2}"; local __default="${3}"
    read -e -p "${__prompt}"$'\x0a\e[32m:\e[0m ' -i "${__default}" ${__var}
    export ${__var}
}
check_var(){
    local __missing=false
    local __vars="$@"
    for __var in ${__vars}; do
        if [[ -z "${!__var}" ]]; then
            error "${__var} variable is missing."
            __missing=true
        fi
    done
    if [[ ${__missing} == true ]]; then
        fault
    fi
}
check_num(){
    local var=$1
    check_var var
    if ! [[ ${!var} =~ ^[0-9]+$ ]] ; then
        fault "${var} is not a number: '${!var}'"
    fi
}
element_in_array () {
  local e match="$1"; shift;
  for e; do [[ "$e" == "$match" ]] && return 0; done
  return 1
}
get_bridges() {
    readarray -t INTERFACES < <(cat /etc/network/interfaces | grep -Po "^iface \K(vmbr[0-9]*)")
    stderr ""
    stderr "Currently configured bridges:"
    (
        echo "BRIDGE|NETWORK|COMMENT
"
        for i in "${INTERFACES[@]}"; do
            local COMMENT="$(get_interface_comment ${i})"
            echo "${i}|$(get_interface_network ${i})|$(get_interface_comment ${i})"
        done
    ) | column -t -s '|' | trim_trailing_whitespace
}
prefix_to_netmask () {
    #thanks https://forum.archive.openwrt.org/viewtopic.php?id=47986&p=1#p220781
    set -- $(( 5 - ($1 / 8) )) 255 255 255 255 $(( (255 << (8 - ($1 % 8))) & 255 )) 0 0 0
    [ $1 -gt 1 ] && shift $1 || shift
    echo ${1-0}.${2-0}.${3-0}.${4-0}
}
validate_ip_address () {
    #thanks https://stackoverflow.com/a/21961938
    echo "$@" | grep -o -E  '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)' >/dev/null
}
validate_ip_network() {
    #thanks https://stackoverflow.com/a/21961938
    PREFIX=$(echo "$@" | grep -o -P "/\K[[:digit:]]+$")
    if [[ "${PREFIX}" -ge 0 ]] && [[ "${PREFIX}" -le 32 ]]; then
        echo "$@" | grep -o -E  '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/[[:digit:]]+' >/dev/null
    else
        return 1
    fi
}
debug_var() {
    local var=$1
    check_var var
    echo "## DEBUG: ${var}=${!var}" > /dev/stderr
}

new_interface() {
    local INTERFACE IP_CIDR IP_ADDRESS COMMENT OTHER_BRIDGE DEFAULT_IP_CIDR
    set -e
    echo
    echo "Configuring new NAT bridge ..."
    ask "Enter the existing bridge to NAT from" OTHER_BRIDGE vmbr0
    if ! element_in_array "$OTHER_BRIDGE" "${INTERFACES[@]}"; then
        fault "Sorry, ${OTHER_BRIDGE} is not a valid bridge (it does not exist)"
    fi
    ask "Enter a unique number for the new bridge (don't write the vmbr prefix)" BRIDGE_NUMBER
    check_num BRIDGE_NUMBER
    INTERFACE="vmbr${BRIDGE_NUMBER}"

    if element_in_array "$INTERFACE" "${INTERFACES[@]}"; then
        error "Sorry, ${INTERFACE} already exists."
        echo
        return
    fi
    echo
    echo "Configuring new interface: ${INTERFACE}"
    if [[ "${BRIDGE_NUMBER}" -ge 0 ]] && [[ "${BRIDGE_NUMBER}" -le 255 ]]; then
        DEFAULT_IP_CIDR=$(echo "${DEFAULT_NETWORK_PATTERN}" | sed "s/BRIDGE/${BRIDGE_NUMBER}/")
    else
        DEFAULT_IP_CIDR=""
    fi
    ask "Enter the static IP address and network prefix in CIDR notation for ${INTERFACE}:" IP_CIDR "${DEFAULT_IP_CIDR}"
    if ! validate_ip_network "${IP_CIDR}"; then
        fault "Bad IP address/network prefix (use the format eg. 192.168.1.1/24)"
    fi
    echo
    debug_var IP_CIDR
    IP_ADDRESS=$(echo "$IP_CIDR" | cut -d "/" -f 1)
    NET_PREFIX="$(echo "$IP_CIDR" | cut -d "/" -f 2)"
    if ! validate_ip_address "${IP_ADDRESS}"; then
        fault "Bad IP address: ${IP_ADDRESS}"
    fi
    echo
    ask "Enter the description/comment for this interface" COMMENT "NAT ${IP_CIDR} bridged to ${OTHER_BRIDGE}"
    cat <<EOF >> /etc/network/interfaces

auto ${INTERFACE}
iface ${INTERFACE} inet static
        address  ${IP_ADDRESS}/${NET_PREFIX}
        bridge_ports none
        bridge_stp off
        bridge_fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '${IP_CIDR}' -o ${OTHER_BRIDGE} -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '${IP_CIDR}' -o ${OTHER_BRIDGE} -j MASQUERADE
#${COMMENT}

EOF
    echo "Wrote /etc/network/interfaces"
    ifup "${INTERFACE}"
    echo "Activated ${INTERFACE}"
}

get_interface_comment() { awk "/^iface ${1} /,/^$/" /etc/network/interfaces | grep -v -e '^$' | grep -e '^#' | tail -1 | tr -d '#'; }

get_interface_network() { awk "/^iface ${1} /,/^$/" /etc/network/interfaces | grep -o -P "^\W+address \K(.*)"; }


activate_iptables_rules() {
    if [[ ! -f ${IPTABLES_RULES_SCRIPT} ]]; then
        fault "iptables script not found: ${IPTABLES_RULES_SCRIPT}"
    fi
    if [[ ! -f ${SYSTEMD_SERVICE} ]]; then
        cat <<EOF > ${SYSTEMD_SERVICE}
[Unit]
Description=Load iptables ruleset from ${IPTABLES_RULES_SCRIPT}
ConditionFileIsExecutable=${IPTABLES_RULES_SCRIPT}
After=network-online.target

[Service]
Type=forking
ExecStart=${IPTABLES_RULES_SCRIPT}
TimeoutSec=0
RemainAfterExit=yes
GuessMainPID=no

[Install]
WantedBy=network-online.target
EOF
    fi
    systemctl daemon-reload
    if [[ "$(systemctl is-enabled ${SYSTEMD_UNIT})" != "enabled" ]]; then
        enable_service
    else
        echo "Systemd unit already enabled: ${SYSTEMD_UNIT}"
        systemctl restart ${SYSTEMD_UNIT} && echo "NAT rules applied: ${IPTABLES_RULES_SCRIPT}"
    fi
}

get_port_forward_rules() {
    # Retrieve the PORT_FORWARD_RULES array from the iptables script:
    if [[ ! -f "${IPTABLES_RULES_SCRIPT}" ]]; then
        return
    fi
    IFS=' ' read -ra rule_parts < <(grep -P -o "^PORT_FORWARD_RULES=\(\K(.*)\)$" ${IPTABLES_RULES_SCRIPT} | tr -d '()' | tail -1)
    for part in "${rule_parts[@]}"; do
        echo "${part}"
    done
}

create_iptables_rules() {
    readarray -t PORT_FORWARD_RULES <<< "$@"
    if [[ "${#PORT_FORWARD_RULES[@]}" -eq "0" ]]; then
        fault "PORT_FORWARD_RULES array is empty!"
    fi
    cat <<'EOF' > ${IPTABLES_RULES_SCRIPT}
#!/bin/bash
## Script to configure the DNAT port forwarding rules:
## This script should not be edited by hand, it is generated from proxmox_nat.sh

error(){ echo "Error: $@"; }
fault(){ test -n "$1" && error $1; echo "Exiting." >/dev/stderr; exit 1; }
check_var(){
    local __missing=false
    local __vars="$@"
    for __var in ${__vars}; do
        if [[ -z "${!__var}" ]]; then
            error "${__var} variable is missing."
            __missing=true
        fi
    done
    if [[ ${__missing} == true ]]; then
        fault
    fi
}
purge_port_forward_rules() {
    iptables-save | grep -v "Added by ${BASH_SOURCE}" | iptables-restore
}
apply_port_forward_rules() {
    ## Validate all the rules:
    set -e
    if [[ "${#PORT_FORWARD_RULES[@]}" -le 1 ]] && [[ "${PORT_FORWARD_RULES[0]}" == "" ]]; then
        error "PORT_FORWARD_RULES array is empty!"
        exit 0
    fi
    for rule in "${PORT_FORWARD_RULES[@]}"; do
        echo "debug: ${rule}"
        IFS=':' read -ra rule_parts <<< "$rule"
        if [[ "${#rule_parts[@]}" != "5" ]]; then
            fault "Invalid rule (there should be 5 parts): ${rule}"
        fi
    done
    ## Apply all the rules:
    for rule in "${PORT_FORWARD_RULES[@]}"; do
        IFS=':' read -ra rule_parts <<< "$rule"
        local INTERFACE PROTOCOL IN_PORT DEST_IP DEST_PORT
        INTERFACE="${rule_parts[0]}"
        PROTOCOL="${rule_parts[1]}"
        IN_PORT="${rule_parts[2]}"
        DEST_IP="${rule_parts[3]}"
        DEST_PORT="${rule_parts[4]}"
        check_var INTERFACE PROTOCOL IN_PORT DEST_IP DEST_PORT
        iptables -t nat -A PREROUTING -i ${INTERFACE} -p ${PROTOCOL} \
            --dport ${IN_PORT} -j DNAT --to ${DEST_IP}:${DEST_PORT} \
            -m comment --comment "Added by ${BASH_SOURCE}"
    done
}
EOF
    cat <<EOF >> ${IPTABLES_RULES_SCRIPT}
## PORT_FORWARD_RULES is an array of port forwarding rules,
## each item in the array contains five elements separated by colon:
## INTERFACE:PROTOCOL:OUTSIDE_PORT:IP_ADDRESS:DEST_PORT

## * IMPORTANT: PORT_FORWARD_RULES should all be on ONE LINE with no line breaks.

## Here is an example with two rules (commented out), and explained:
##  * For any TCP packet on port 2222 coming from vmbr0, forward to 10.15.0.2 on port 22
##  * For any UDP packet on port 5353 coming from vmbr0, forward to 10.15.0.3 on port 53
## PORT_FORWARD_RULES=(vmbr0:tcp:2222:10.15.0.2:22 vmbr0:udp:5353:10.15.0.3:53)

PORT_FORWARD_RULES=(${PORT_FORWARD_RULES[@]})

### Apply all the rules:
purge_port_forward_rules
apply_port_forward_rules
EOF
    chmod a+x "${IPTABLES_RULES_SCRIPT}"
    echo "Wrote ${IPTABLES_RULES_SCRIPT}"
}

print_port_forward_rule() {
    IFS=':' read -ra rule_parts <<< "$@"
    local INTERFACE PROTOCOL IN_PORT DEST_IP DEST_PORT
    INTERFACE="${rule_parts[0]}"
    PROTOCOL="${rule_parts[1]}"
    IN_PORT="${rule_parts[2]}"
    DEST_IP="${rule_parts[3]}"
    DEST_PORT="${rule_parts[4]}"
    check_var INTERFACE PROTOCOL IN_PORT DEST_IP DEST_PORT
    echo "${INTERFACE}|${PROTOCOL}|${IN_PORT}|${DEST_IP}|${DEST_PORT}"
}

print_port_forward_rules() {
    readarray -t PORT_FORWARD_RULES < <(get_port_forward_rules)
    if [[ "${#PORT_FORWARD_RULES[@]}" -le 1 ]] && [[ "${PORT_FORWARD_RULES[0]}" == "" ]]; then
        echo "No inbound port forwarding (DNAT) rules have been created yet."
    else
        echo "## Existing inbound port forwarding (DNAT) rules:"
        (
            echo "INTERFACE|PROTOCOL|IN_PORT|DEST_IP|DEST_PORT"
            for rule in "${PORT_FORWARD_RULES[@]}"; do
                print_port_forward_rule "${rule}"
            done
        ) | column -t -s '|'
    fi
}

define_port_forwarding_rules() {
    readarray -t PORT_FORWARD_RULES < <(get_port_forward_rules)
    while true; do
        echo "Defining new port forward rule:"
        ask "Enter the inbound interface" INTERFACE vmbr0
        ask "Enter the protocol (tcp, udp)" PROTOCOL tcp
        ask "Enter the inbound Port number" IN_PORT
        check_num IN_PORT
        ask "Enter the destination IP address" DEST_IP
        validate_ip_address "${DEST_IP}" || fault "Invalid ip address: ${DEST_IP}"
        ask "Enter the destination Port number" DEST_PORT
        check_num DEST_PORT
        check_var INTERFACE PROTOCOL IN_PORT DEST_IP DEST_PORT
        local RULE="${INTERFACE}:${PROTOCOL}:${IN_PORT}:${DEST_IP}:${DEST_PORT}"
        (
            echo "INTERFACE|PROTOCOL|IN_PORT|DEST_IP|DEST_PORT"
            print_port_forward_rule "${RULE}"
        ) | column -t -s '|'
        confirm yes "Is this rule correct" "?" || return
        PORT_FORWARD_RULES+=("$RULE")
        echo
        confirm no "Would you like to define more port forwarding rules now" "?" || break
    done
    create_iptables_rules "${PORT_FORWARD_RULES[@]}"
    activate_iptables_rules
    echo
    print_port_forward_rules
    echo
}

delete_port_forwarding_rules() {
    readarray -t PORT_FORWARD_RULES < <(get_port_forward_rules)
    while true; do
        if [[ "${PORT_FORWARD_RULES[@]}" == "" ]]; then
            print_port_forward_rules
            break
        fi
        echo
        (
            echo "LINE# INTERFACE PROTOCOL IN_PORT DEST_IP DEST_PORT"
            print_port_forward_rules 2>/dev/null | grep -v "#" | tail -n +2 | cat -n | trim_whitespace
        ) | column -t | trim_whitespace
        ask_allow_blank 'Enter the line number for the rule you wish to delete (type `q` or blank for none)' RULE_TO_DELETE
        if [[ -z "${RULE_TO_DELETE}" ]] || [[ "${RULE_TO_DELETE}" == "q" ]]; then
            break
        fi
        RULE_TO_DELETE=$((${RULE_TO_DELETE} - 1))
        if [[ "${RULE_TO_DELETE}" -lt 0 ]] || \
               [[ "${RULE_TO_DELETE}" -gt "${#PORT_FORWARD_RULES[@]}" ]]; then
            error "Invalid rule number"
            break
        fi
        local to_delete="${PORT_FORWARD_RULES[${RULE_TO_DELETE}]}"
        PORT_FORWARD_RULES=("${PORT_FORWARD_RULES[@]/${to_delete}}")
        create_iptables_rules "${PORT_FORWARD_RULES[@]}"
        activate_iptables_rules
    done
    echo
}

enable_service() {
    echo "The systemd unit is named: ${SYSTEMD_UNIT}"
    echo "The systemd unit is currently: $(systemctl is-enabled ${SYSTEMD_UNIT})"
    if confirm yes "Would you like to enable the systemd unit on boot" "?"; then
        systemctl enable ${SYSTEMD_UNIT}
        echo "Systemd unit enabled: ${SYSTEMD_UNIT}"
        systemctl restart ${SYSTEMD_UNIT}
        echo "NAT rules applied: ${IPTABLES_RULES_SCRIPT}"
    else
        systemctl disable ${SYSTEMD_UNIT}
        echo "Systemd unit is disabled on next boot: ${SYSTEMD_UNIT}"
    fi
}

print_help() {
    echo "NAT bridge tool:"
    echo ' * Type `i` or `interfaces` to list the bridge interfaces.'
    echo ' * Type `c` or `create` to create a new NAT bridge.'
    echo ' * Type `l` or `list` to list the NAT rules.'
    echo ' * Type `n` or `new` to create some new NAT rules.'
    echo ' * Type `d` or `delete` to delete some existing NAT rules.'
    echo ' * Type `e` or `enable` to enable or disable adding the rules on boot.'
    echo ' * Type `?` or `help` to see this help message again.'
    echo ' * Type `q` or `quit` to quit.'
}

main() {
    echo
    get_bridges
    echo
    while :
    do
        print_help
        echo
        ask_allow_blank 'Enter command (for help, enter `?`)' COMMAND
        echo
        if [[ "$COMMAND" == 'q' ]] || [[ "$COMMAND" == 'quit' ]]; then
            if [[ "$(systemctl is-enabled ${SYSTEMD_UNIT})" != "enabled" ]]; then
                enable_service
            fi
            echo "goodbye"
            exit 0
        elif [[ $COMMAND == '?' || $COMMAND == "help" ]]; then
            print_help
        elif [[ $COMMAND == "i" || $COMMAND == "interfaces" ]]; then
            get_bridges
        elif [[ $COMMAND == "c" || $COMMAND == "create" ]]; then
            get_bridges
            new_interface || true
        elif [[ $COMMAND == "l" || $COMMAND == "list" ]]; then
            print_port_forward_rules
        elif [[ $COMMAND == "n" || $COMMAND == "new" ]]; then
            define_port_forwarding_rules
        elif [[ $COMMAND == "d" || $COMMAND == "delete" ]]; then
            delete_port_forwarding_rules
        elif [[ $COMMAND == "e" || $COMMAND == "enable" ]]; then
            enable_service
        fi
        echo
    done
}

main

Systemd unit to manage NAT rules

The script includes a systemd unit that is setup to add the DNAT (ingress) rules on every system boot.

Here are the relevant files, whose paths are all declared at the top of the script:

SYSTEMD_UNIT="my-iptables-rules"
SYSTEMD_SERVICE="/etc/systemd/system/${SYSTEMD_UNIT}.service"
IPTABLES_RULES_SCRIPT="/etc/network/${SYSTEMD_UNIT}.sh"
  • SYSTEMD_UNIT is the name of the systemd service that is started. You can interact with it with systemctl:
$ systemctl status my-iptables-rules
● my-iptables-rules.service - Load iptables ruleset from /etc/network/my-iptables-rules.sh
     Loaded: loaded (/etc/systemd/system/my-iptables-rules.service; enabled; preset: enabled)
     Active: active (exited) since Wed 2023-11-22 16:05:48 MST; 6h ago
    Process: 469722 ExecStart=/etc/network/my-iptables-rules.sh (code=exited, status=0/SUCCESS)
        CPU: 8ms

Nov 22 16:05:48 pve systemd[1]: Starting my-iptables-rules.service - Load iptables ruleset from /…s.sh...Nov 22 16:05:48 pve my-iptables-rules.sh[469722]: Error: PORT_FORWARD_RULES array is empty!
Nov 22 16:05:48 pve systemd[1]: Started my-iptables-rules.service - Load iptables ruleset from /e…les.sh.Hint: Some lines were ellipsized, use -l to show in full.

The output Error: PORT_FORWARD_RULES array is empty! is normal when you have not yet defined any DNAT rules (ie. all ports are blocked). If you need to see the full log output, use journalctl:

journalctl --unit my-iptables-rules
  • SYSTEMD_SERVICE is the full path to the systemd service config file, (and which is automatically created by the script).
## You don't need to copy this, this is just an example of what
## the script automatically creates for you:
[Unit]
Description=Load iptables ruleset from /etc/network/my-iptables-rules
ConditionFileIsExecutable=/etc/network/my-iptables-rules
After=network-online.target

[Service]
Type=forking
ExecStart=/etc/network/my-iptables-rules
TimeoutSec=0
RemainAfterExit=yes
GuessMainPID=no

[Install]
WantedBy=network-online.target

The WantedBy config will ensure the service is started on boot.

  • IPTABLES_RULES_SCRIPT is the path to the NAT rules configuration/script, (and which is automatically created by the script). The systemd service calls this script to add the NAT rules, on boot. You can also call the script yourself anytime. When the script is executed, all of the existing DNAT rules are purged (they are all tagged Added by ${IPTABLES_RULES_SCRIPT}, and so are deleted based on this same tag.). New rules are then created based on the PORT_FORWARD_RULES variable in the current IPTABLES_RULES_SCRIPT.

Manage bridges from the dashboard

Although you cannot manage the NAT rules from the dashboard, you can add or remove the bridges:

Proxmox dashboard shows all the NAT bridges, you can easily delete them from here
Proxmox dashboard shows all the NAT bridges, you can easily delete them from here

Creating VMs with NAT

With bridge networking, VMs can query a DHCP server to get an IP address. When using a private NAT network, there is no DHCP server (by default) that can be contacted. Therefore, you will need to configure a static IP address and gateway for each VM:

VM cloud-init setting a static IP address and gateway
VM cloud-init setting a static IP address and gateway

Configure the virtual network card to connect to the correct bridge:

A VM selecting the bridge to connect the virtual ethernet cable to
A VM selecting the bridge to connect the virtual ethernet cable to

Adding a DHCP server

If you want to add a DHCP server to your NAT bridge, you can use something simple like dnsmasq. You’ll need to create a VM with a static IP.

For example, suppose:

  • 10.1.1.1 is the Proxmox host on vmbr1 (this is the gateway)
  • 10.1.1.2 is a VM with a static IP, tasked with running dnsmasq (this is the DHCP server)
# Run this on the VM running on 10.1.1.2:
systemctl stop systemd-resolved
systemctl mask systemd-resolved
apt update
apt install dnsmasq

Edit the /etc/dnsmasq.conf config file on the VM:

## Example /etc/dnsmasq.conf file for running a DHCP server:
### 10.1.1.1 is the gateway (the proxmox host)
### 10.1.1.2 is this VM, the DHCP server

# Run on the main VM interface:
interface=eth0
except-interface=lo

# Set an appropriate domain:
domain=vm1
bind-interfaces

# Listen on the static IP address for this VM (10.1.1.2):
listen-address=10.1.1.2
server=::1
server=127.0.0.1

# Serve the DHCP range from .10 to .250:
dhcp-range=10.1.1.10,10.1.1.250,255.255.255.0,1h
# Add the default route through the proxmox host (10.1.1.1):
dhcp-option=3,10.1.1.1
dhcp-option=6,10.1.1.1

Save the file, and restart dnsmasq:

systemctl enable --now dnsmasq
systemctl restart dnsmasq

If you now boot another VM on the same bridge (vmbr1), and if its configured for DHCP, it should get an ip from the dnsmasq server, and create a route through the proxmox host 10.1.1.1. Now you won’t have to set any more static IPs.



You can discuss this blog on Matrix (Element): #blog-rymcg-tech:enigmacurry.com

This blog is copyright EnigmaCurry and dual-licensed CC-BY-SA and MIT. The source is on github: enigmacurry/blog.rymcg.tech and PRs are welcome. ❤️