Traditional point-to-point site-to-site VPN protocols require extensive setup in certain use cases. For example, if you want shortcuts between branch offices rather than let every packet go through the HQ, then you need to set up a cartesian product of tunnels by hand, which is time-consuming and error prone. So, people want something better, something easy to set up and maintain, and dynamic enough. While there are already a lot dynamic point-to-multipoint or full-mesh site-to-site VPN implementations (e.g. Tinc VPN, ZeroTier) on Linux, you don’t have many choices on these commercial black boxes.

Cisco DMVPN (Cisco Dynamic Multipoint VPN) is one solution to this. Huawei also had their DMVPN-compatible solution called DSVPN (Dynamic Smart VPN). Since the protocol is largely compatible, I’ll just reference it as DMVPN.

On the DMVPN Protocol

Strictly speaking, DMVPN is not one protocol, but is stitched from 4 protocols:

Multipoint GRE (mGRE)
Next Hop Resolution Protocol (NHRP) inside mGRE
(Optional) IPSec encryption outside mGRE
(Optional) another routing protocol, typically OSPF

Multipoint GRE

Multipoint GRE is just plain GRE but with a mapping of inside destination IPs to outside destination IPs. It doesn’t have a single remote tunnel endpoint IP, instead we need to have some external protocol to fill its IP mapping (like the ARP protocol to the Ethernet).

Next Hop Resolution Protocol

NHRP is a protocol which can automatically fill the mGRE tunnel’s IP mapping, it’s the ARP for mGRE. It works in a straightforward way: some (usually 1 or 2) devices are designated as hubs (NHS, Next Hop Server), while others are all spokes. Hubs does not have any mapping information while NHRP daemon is started, and spokes have statically configured mapping of all hubs' IP mapping information. When spokes' NHRP daemon is started, the daemon sends registration packets to the hubs, announcing its existence, its own external and internal IPs and optionally some other routes. When spokes try to connect to each other, they query the packet destination’s external IP addresses with the hub and try to directly connect them with GRE. If the other end is not reachable, packet is routed to the hub then to the destination.

IPSec

IPSec is a protocol mainly used to transparently encrypt IP packets. I’m not going to talk about it since it is not the point of this article.

Routing Protocol

NHRP is not suitable for distributing a lot routes with it. For mid-sized or larger networks, we usually only use NHRP to resolve tunnel endpoint IPs, and use another routing protocol (RIP, OSPF or BGP) for routing the client networks.

Security

Since DMVPN can run without IPSec, if the NHRP daemon implementation is flawed, there is possibility that one plain unprotected tunnel can register successfully thus resulting in a downgrade attack.

Lab Setup

There are already example setups on the wild Internet using a Cisco router as the hub and other Cisco/Huawei devices as the spokes (see the references section). Today we try using Huawei USG firewall as the hub. All configurations are provided in 2 parts, one for plaintext mGRE/NHRP, and one for adding IPSec protection to the existing tunnel.

Security warnings first:

DO NOT use the plaintext config in production! If you want to capture NHRP packets in a lab environment, use this config. It is not secure for production traffic.
DO NOT blindly copy the IPSec ciphers for production. Use strong GCM ciphers and enable PFS if your devices support them.

And some notes on cross-vendor compatibility:

Huawei devices send L3MTU as OSPF MTU, and other vendors send L2MTU, so we ignore MTU by default in OSPF config
The maximum MTU of Huawei USG firewalls is 1500, so we just use 1500 for every device

Hub - Huawei USG

Test gear: Huawei USG6320, software version V500R005C00SPC200

If you config DSVPN from the USG web interface, here is a quick lookup table for you:

Huawei (English)	Huawei (中文)	Other Vendors
Reverse Routes from Branches	总部从分支学习反向路由	send static routes with NHRP
Mutual Route Learning	分支节点相互学习路由	no shortcut
Route Aggregation to HQ	分支节点路由汇聚到总部	shortcut

Plain Text

interface Tunnel0
 description hub
 ip address 192.168.1.1 255.255.255.0
 tunnel-protocol gre p2mp
 source GigabitEthernet0/0/0
 ospf network-type broadcast
 ospf dr-priority 2
 alias dsvpn-hub-test
 service-manage ping permit
 nhrp network-id 1000
 nhrp authentication plain 114514
 nhrp redirect
 nhrp entry multicast dynamic
 undo nhrp hub reverse-route enable
ospf 1
 area 0.0.0.0
  network 192.168.1.0 0.0.0.255

With IPSec

ipsec sha2 compatible enable
ike dpd type periodic
ipsec proposal proposal_1
 encapsulation-mode auto
 esp authentication-algorithm sha2-256
 esp encryption-algorithm aes-256
ike proposal 1
 encryption-algorithm aes-256
 dh group14
 authentication-algorithm sha2-256
 authentication-method pre-share
 integrity-algorithm hmac-sha2-256
 prf hmac-sha2-256
ike peer peer_1
 exchange-mode auto
 pre-shared-key 114514
 ike-proposal 1
 dpd type periodic
ipsec profile profile_1
 ike-peer peer_1
 proposal proposal_1
 sa duration traffic-based 5242880
 sa duration time-based 3600
interface Tunnel0
 ipsec profile profile_1

Security

Even if the DMVPN tunnel is configured to use IPSec, a cleartext NHRP request will still result in a NHRP neighbor entry. You are not able to send any packets through the tunnel, though. (This might lead to buffer overflow vulnerabilities from unsolicited NHRP peers.)

Caveats

NHRP redirect & multicast config is only available in CLI
OSPF network type can only be broadcast (non-configurable)
OSPF MTU ignore is not user-configurable
MTU is not user-configurable (1500 hardcoded)
Huawei never officially said DSVPN is compatible with DMVPN, so the compatibility is not guaranteed
If an IP address of the same subnet as the one configured on the DMVPN interface doesn’t have a NHRP entry, an USG spoke will send it to the HUB

Huawei extensions to NHRP protocol (use undo nhrp identity enable to disable):

1024 (0x0400): 0-byte padding
1025 (0x0401): device identity (string, length 1-31 bytes, default value is hostname)
1026 (0x0402): 0-byte padding
1027 (0x0403): device serial number (string, 20 bytes)

BTW, one free advice for you: if you have two DSVPN instances mapped to one OSPF process from the CLI, do not try to edit DSVPN config from the web; this will lead to unexpected results.

Spoke 1 - Cisco IOS XE

Test gear: Cisco CSR1000v, software IOS XE 16.9.4

Plain Text

interface Tunnel0
 ip address 192.168.1.2 255.255.255.0
 no ip redirects
 ip mtu 1500
 ip nhrp authentication 114514
 ip nhrp network-id 1000
 ip nhrp holdtime 30
 ip nhrp nhs 192.168.1.1 nbma 10.0.0.1 multicast
 ip nhrp registration no-unique
 ip ospf network broadcast
 ip ospf priority 0
 ip ospf mtu-ignore
 qos pre-classify
 tunnel source GigabitEthernet1
 tunnel mode gre multipoint
router ospf 1
 network 192.168.1.0 0.0.0.255 area 0

With IPSec

crypto isakmp policy 10
 encr aes 256
 hash sha256
 authentication pre-share
 group 14
crypto isakmp key 114514 address 0.0.0.0 no-xauth
crypto isakmp keepalive 30 10 periodic
crypto isakmp diagnose error
crypto ipsec transform-set dsvpn esp-aes 256 esp-sha256-hmac
 mode tunnel
crypto ipsec profile dsvpn
 set transform-set dsvpn
interface Tunnel0
 tunnel protection ipsec profile dsvpn

Security

Not tested.

Caveats

NHRP network-id or NHRP domain is just a process ID; it is only used locally, but if you don’t configure it, NHRP process will be down forever.

The tunnel’s L2MTU will be hardcoded to 9976 and you cannot configure it. (Note that the transport MTU will be the correct value i.e. 1476, and you can still manually set the L3MTU.) So we explicitly set OSPF to ignore MTU mismatches.

On older devices, the config line

ip nhrp nhs 192.168.1.1 nbma 10.0.0.1 multicast

should be replaced with the old grammar

ip nhrp map 192.168.1.1 10.0.0.1
ip nhrp map multicast 10.0.0.1
ip nhrp nhs 192.168.1.1

Spoke 2 - VyOS

Test gear: VyOS 1.2.6 (with patch) / VyOS 1.3 nightly (official)

Plain Text

top
edit interface tunnel tun0
set address '192.168.1.3/24'
set encapsulation 'gre'
set ip ospf network broadcast
set local-ip '10.0.0.3'
set mtu 1512
set multicast 'enable'

top
edit protocols nhrp tunnel tun0
set cisco-authentication '114514'
set holding-time '10'
set map 192.168.1.1/24 nbma-address '10.0.0.1'
set map 192.168.1.1/24 register
set map 192.168.1.1/24 cisco
set multicast 'dynamic'
set redirect

With IPSec

top
edit vpn ipsec esp-group dmvpn
set compression 'disable'
set lifetime '3600'
set mode 'tunnel'
set pfs 'enable'
set proposal 1 encryption 'aes256'
set proposal 1 hash 'sha256'

top
edit vpn ipsec ike-group dmvpn
set close-action 'restart'
set dead-peer-detection action 'restart'
set dead-peer-detection interval '15'
set dead-peer-detection timeout '30'
set key-exchange 'ikev1'
set lifetime '86400'
set proposal 1 dh-group '14'
set proposal 1 encryption 'aes256'
set proposal 1 hash 'sha256'

top
edit vpn ipsec profile dmvpn
set authentication mode 'pre-shared-secret'
set authentication pre-shared-secret '114514'
set bind tunnel tun0
set esp-group 'dmvpn'
set ike-group 'dmvpn'

top
set vpn ipsec ipsec-interfaces interface 'eth0'

Security

Current OpenNHRP’s IPSec configuration works by first generating a IPSec config without local/remote IPs to StrongSwan, then after a neighbor is discovered, use swanctl with local/remote IP override to create a dynamic IPSec peer. If the IPSec tunnel is down for some reason, it won’t be restarted, resulting in packets being sent in cleartext. Due to this reason, I do not recommend using VyOS for DMVPN in production networks.

Caveats

OpenNHRP implementation from VyOS 1.2.6 is not compatible with Huawei’s NHRP implementation. If you really want to use VyOS together with Huawei devices, you need to apply this patch. First we need to compile it. On a Debian 8 (jessie):

apt install git build-essential debhelper libc-ares-dev pkg-config
mkdir opennhrp
cd opennhrp
git clone https://github.com/Jamesits/vyos-opennhrp.git
cd vyos-opennhrp
dpkg-buildpackage -us -uc

Then install it onto VyOS:

dpkg -i vyos-opennhrp_0.14.1-1+vyos3+equuleus1_amd64.deb
reboot

The patch will be available for the current running VyOS image.

Spoke 3 - Huawei AR Router

Test gear: Huawei AR6121-S, VRP 5.170 (V300R019C10SPC300)

Plain Text

interface Tunnel0/0/1
 description DSVPN
 ip address 192.168.1.4 255.255.255.0
 tunnel-protocol gre p2mp
 source GigabitEthernet0/0/0
 ospf network-type broadcast
 ospf dr-priority 0
 nhrp authentication cipher 114514
 nhrp tunnel-if-state related
 nhrp registration no-unique
 nhrp registration interval 30
 nhrp entry 192.168.1.1 10.0.0.1 register preference 10

With IPSec

ipsec authentication sha2 compatible enable
ipsec proposal dsvpn
 esp authentication-algorithm sha2-256
 esp encryption-algorithm aes-256
ike proposal 1
 encryption-algorithm aes-256
 dh group14
 authentication-algorithm sha2-256
 authentication-method pre-share
 integrity-algorithm hmac-sha2-256
 prf hmac-sha2-256
ike peer dsvpn
 exchange-mode auto
 pre-shared-key cipher 114514
 ike-proposal 1
 dpd type periodic
 dpd idle-time 12
 dpd retransmit-interval 5
 rsa encryption-padding oaep
 rsa signature-padding pss
 ikev2 authentication sign-hash sha2-256
ipsec profile dsvpn 1
 ike-peer dsvpn
 proposal dsvpn
 sa duration traffic-based 5242880
 sa duration time-based 3600
interface Tunnel0/0/1
 ipsec profile dsvpn shared

Security

Not tested.

Caveats

For an AR router to connect to other routers successfully with IPSec, ipsec authentication sha2 compatible enable might be required.

If IPSec tunnel mode is required (e.g. spoke is behind NAT), manually switch to tunnel mode:

ipsec proposal dsvpn
 encapsulation-mode tunnel

Unlike an USG, an AR does not support encapsulation mode auto negotiation.

If an IP address of the same subnet as the one configured on the DMVPN interface doesn’t have a NHRP entry, an AR spoke will drop it, which would cause some problems in certain topologies.

Notes

References:

Drown in Codes

I was sixteen with an open heart…

Interoperating Cisco DMVPN, Huawei DSVPN and OpenNHRP

On the DMVPN Protocol

Multipoint GRE

Next Hop Resolution Protocol

IPSec

Routing Protocol

Security

Lab Setup

Hub - Huawei USG

Plain Text

With IPSec

Security

Caveats

Spoke 1 - Cisco IOS XE

Plain Text

With IPSec

Security

Caveats

Spoke 2 - VyOS

Plain Text

With IPSec

Security

Caveats

Spoke 3 - Huawei AR Router

Plain Text

With IPSec

Security

Caveats

Notes