Anycast with multiple BNGs

On a previous post we saw an example of a network access topology running anycast default gateways.

The idea is to save IPv4 addresses, without other methods than standard routing protocols. Just plain BGP that can be implemented on most vendors, either via hardaware appliances or virtualized network devices.

For the following examples, I’ll simulate a public /24 prefix using the 198.51.100.0/24 which is reserved by IANA as TEST-NET-2 for documentation.

Lab Network

This lab network comprises 2 AGG/BNG routers, a single core router (which will perform additional aggregation), and a single edge router. All devices are MikroTik RouterOS 6.48.6 CHR instances.

Lab network

edge-01

/interface bridge
add name=lo0

/ip address
add address=10.0.1.1 interface=lo0 network=10.0.1.1
add address=10.0.0.1/30 interface=ether1 network=10.0.0.0

/routing ospf instance
set [ find default=yes ] router-id=10.0.1.1
/routing ospf interface
add passive=yes
add interface=ether1 network-type=point-to-point
/routing ospf network
add area=backbone network=10.0.0.0/30
add area=backbone network=10.0.1.1/32

/routing bgp instance
set default as=65000 router-id=10.0.1.1
/routing bgp peer
add default-originate=always in-filter=core-01-in name=core-01 out-filter=core-01-out remote-address=10.1.1.1 remote-as=65000 update-source=lo0
/routing filter
add action=accept chain=core-01-in prefix=198.51.100.0/24
add action=discard chain=core-01-in
add action=accept chain=core-01-out prefix=0.0.0.0/0
add action=discard chain=core-01-out

/system identity
set name=edge-01

The “edge” router is peering with the “core” through their loopbacks, and just advertising a default to it (or cores, on a future stage), and accepting the entire 198.51.100.0/24.

core-01

/interface bridge
add name=lo0

/ip address
add address=10.1.1.1 interface=lo0 network=10.1.1.1
add address=10.0.0.2/30 interface=ether1 network=10.0.0.0

add address=10.255.255.1/30 interface=ether2 network=10.255.255.0
add address=10.255.255.5/30 interface=ether3 network=10.255.255.4

/routing bgp instance
set default as=65000 cluster-id=10.1.1.1 router-id=10.1.1.1
/routing bgp peer
add in-filter=edge-01-in name=edge-01 out-filter=edge-01-out remote-address=10.0.1.1 remote-as=65000 update-source=lo0
/routing bgp peer
add in-filter=edge-01-in name=edge-01 out-filter=edge-01-out remote-address=10.0.1.1 remote-as=65000 update-source=lo0
add default-originate=if-installed in-filter=agg-bng-in name=agg-bng-01 out-filter=agg-bng-out remote-address=10.10.1.1 remote-as=65000 route-reflect=yes update-source=lo0
add default-originate=if-installed in-filter=agg-bng-in name=agg-bng-02 out-filter=agg-bng-out remote-address=10.10.1.2 remote-as=65000 route-reflect=yes update-source=lo0

/routing filter
add action=accept chain=edge-01-in prefix=0.0.0.0/0
add action=discard chain=edge-01-in
add action=accept chain=edge-01-out prefix=198.51.100.0/24
add action=discard chain=edge-01-out

/routing filter
add action=accept chain=agg-bng-in prefix=198.51.100.0/24 prefix-length=24-29
add action=discard chain=agg-bng-in
add action=accept chain=agg-bng-out prefix=0.0.0.0/0
add action=discard chain=agg-bng-out

/routing ospf instance
set [ find default=yes ] router-id=10.1.1.1
/routing ospf interface
add passive=yes
add interface=ether1 network-type=point-to-point
add interface=ether2 network-type=point-to-point
add interface=ether3 network-type=point-to-point
/routing ospf network
add area=backbone network=10.0.0.0/30
add area=backbone network=10.1.1.1/32
add area=backbone network=10.255.255.0/30
add area=backbone network=10.255.255.4/30

/system identity
set name=core-01

The “core” is peering with the “edge” of course, and also with two BNGs named as agg-bng-xx. This core is advertising its default to them, and accepting all prefixes within 198.51.100.254/24, with a prefix length up to /29.

If you come from a IOS land, this syntax would be something like this.

ip prefix-list BNG
 permit 5 198.51.100.254/24 ge 24 le 29

agg-bng-01

/interface bridge
add name=lo0

/ip address
add address=10.10.1.1 interface=lo0 network=10.10.1.1
add address=10.255.255.2/30 interface=ether1 network=10.255.255.0
add address=198.51.100.254/24 interface=ether2 network=198.51.100.0

/routing ospf instance
set [ find default=yes ] router-id=10.10.1.1
/routing ospf interface
add passive=yes
add interface=ether1 network-type=point-to-point
/routing ospf network
add area=backbone network=10.255.255.0/30
add area=backbone network=10.10.1.1/32

/routing bgp instance
set default as=65000 router-id=10.10.1.1
/routing bgp peer
add in-filter=core-01-in name=core-01 out-filter=core-01-out remote-address=10.1.1.1 remote-as=65000 route-reflect=yes update-source=lo0
/routing filter
add action=accept chain=core-01-in prefix=0.0.0.0/0
add action=discard chain=core-01-in
add action=accept chain=core-01-out prefix=198.51.100.0/24 prefix-length=24-29
add action=discard chain=core-01-out

/system identity
set name=agg-bng-01

agg-bng-02

/interface bridge
add name=lo0

/ip address
add address=10.10.1.2 interface=lo0 network=10.10.1.2
add address=10.255.255.6/30 interface=ether1 network=10.255.255.0
add address=198.51.100.254/24 interface=ether2 network=198.51.100.0

/routing ospf instance
set [ find default=yes ] router-id=10.10.1.2
/routing ospf interface
add passive=yes
add interface=ether1 network-type=point-to-point
/routing ospf network
add area=backbone network=10.255.255.4/30
add area=backbone network=10.10.1.2/32

/routing bgp instance
set default as=65000 router-id=10.10.1.2
/routing bgp peer
add in-filter=core-01-in name=core-01 out-filter=core-01-out remote-address=10.1.1.1 remote-as=65000 route-reflect=yes update-source=lo0
/routing filter
add action=accept chain=core-01-in prefix=0.0.0.0/0
add action=discard chain=core-01-in
add action=accept chain=core-01-out prefix=198.51.100.0/24 prefix-length=24-29
add action=discard chain=core-01-out

/system identity
set name=agg-bng-02

Finally, the BNGs are peering with the core, accepting a default, and allowing any advertisements from 198.51.100.254/24 from /24 to /29.

Both routers have 198.51.100.254/24 as the anycast default gateway.

If you wonder hor this works, this lab network is similar to the one on the previous post which you can check here.

BNG PPPoE and DHCP

We will start by setting up PPPoE services on our BNGs.

At this point we will work it with local secrets and keeping all the AAA process in the same router, with RADIUS as a future option.

Be aware that RouterOS by default will try its best to adjust the TCP MSS to match the PPPoE interface MTU.

Also, this being PPPoE, we have no restrictions on using the first address on the network as the PPP connection will not care about it being a network address. However, this will have the obvious restrictions and behavior if we run DHCP.

We will also skip the .254 address on the address pool as this is assigned to the ether2 interface on both routers as our anycast default gateway.

agg-bng-01

/ip pool
add name=internet ranges=198.51.100.0-198.51.100.127

/ppp profile
add local-address=198.51.100.254 name=internet remote-address=internet

/ppp secret
add name=sub1 password=sub1 profile=internet
add name=sub2 password=sub2 profile=internet

/interface pppoe-server server
add default-profile=internet interface=ether2 disabled=no

agg-bng-02

/ip pool
add name=internet ranges=198.51.100.128-198.51.100.253

/ppp profile
add local-address=198.51.100.254 name=internet remote-address=internet

/ppp secret
add name=sub3 password=sub3 profile=internet
add name=sub4 password=sub4 profile=internet

/interface pppoe-server server
add default-profile=internet interface=ether2 disabled=no

For DHCP, we will reuse the same previously created address pool. The following config applies to both routers.

/ip dhcp-server
add address-pool=internet disabled=no interface=ether2 name=dhcp1
/ip dhcp-server network
add address=198.51.100.0/24 gateway=198.51.100.254 netmask=24

Finally, we will add some test subscribers. A dumb switch will act as the last-mile technology which could be xPON, wireless, DSL, you name it. All the subs are running RouterOS 6.48.6, and this is just to have something capable to talk PPPoE. There is also a VPCS 0.8.2 which comes by default with GNS3.

GNS3 Topology

Address me, father

Starting with sub03 VPCS, we will ask DHCP to the BNG.

Welcome to Virtual PC Simulator, version 0.8.2
Dedicated to Daling.
Build time: Aug 23 2021 11:15:00
Copyright (c) 2007-2015, Paul Meng (mirnshi@gmail.com)
All rights reserved.

VPCS is free software, distributed under the terms of the "BSD" licence.
Source code and license can be found at vpcs.sf.net.
For more information, please visit wiki.freecode.com.cn.

Press '?' to get help.

Executing the startup file


sub03> ip dhcp
DORA IP 198.51.100.252/24 GW 198.51.100.254

sub03> ping 198.51.100.254

84 bytes from 198.51.100.254 icmp_seq=1 ttl=64 time=1.278 ms
84 bytes from 198.51.100.254 icmp_seq=2 ttl=64 time=1.234 ms
84 bytes from 198.51.100.254 icmp_seq=3 ttl=64 time=0.946 ms
^C

If you pay attention, we did get the .252 address, instead of the .253.

sub4 had probably requested this one before, as RouterOS by default comes with a DHCP client on ether1. Is this the case?

[admin@RouterOS] > /ip ad pr
Flags: X - disabled, I - invalid, D - dynamic
 #   ADDRESS            NETWORK         INTERFACE                                                                                                                                                                                         
 0 D 198.51.100.253/24  198.51.100.0    ether1                 

Indeed, both are running DHCP. And just for reference, this is how it looks from the BNG.

[admin@agg-bng-02] /ip dhcp-server> lease pr
Flags: X - disabled, R - radius, D - dynamic, B - blocked
 #   ADDRESS                                       MAC-ADDRESS       HOST-NAME                             SERVER                             RATE-LIMIT                             STATUS  LAST-SEEN
 0 D 198.51.100.253                                0C:04:49:87:00:00 RouterOS                              dhcp1                                                                     bound   2m34s
 1 D 198.51.100.252                                00:50:79:66:68:01 sub03                                 dhcp1                                                                     bound   3m52s

Same is happening with sub01 and sub02, however we’ll remove the DHCP client and add a PPPoE client.

[admin@RouterOS] /interface pppoe-client> add interface=ether1 user=sub1 password=sub1 add-default-route=yes
[admin@RouterOS] /interface pppoe-client> pr
Flags: X - disabled, I - invalid, R - running
 0 X  name="pppoe-out2" max-mtu=auto max-mru=auto mrru=disabled interface=ether1 user="sub1" password="sub1" profile=default keepalive-timeout=10 service-name="" ac-name="" add-default-route=yes default-route-distance=1
      dial-on-demand=no use-peer-dns=no allow=pap,chap,mschap1,mschap2
[admin@RouterOS] /interface pppoe-client> ena 0
[admin@RouterOS] /interface pppoe-client> pr
Flags: X - disabled, I - invalid, R - running
 0  R name="pppoe-out2" max-mtu=auto max-mru=auto mrru=disabled interface=ether1 user="sub1" password="sub1" profile=default keepalive-timeout=10 service-name="" ac-name="" add-default-route=yes default-route-distance=1
      dial-on-demand=no use-peer-dns=no allow=pap,chap,mschap1,mschap2
[admin@RouterOS] /interface pppoe-client> /ip ad pr
Flags: X - disabled, I - invalid, D - dynamic
 #   ADDRESS            NETWORK         INTERFACE
 0 D 198.51.100.125/32  198.51.100.254  pppoe-out2
[admin@RouterOS] /interface pppoe-client>

This config looks as follows on sub02.

[admin@RouterOS] >
[admin@RouterOS] > /ip dhcp-client
[admin@RouterOS] /ip dhcp-client> remove [find]
[admin@RouterOS] /ip dhcp-client> /
[admin@RouterOS] > /interface pppoe-client
[admin@RouterOS] /interface pppoe-client> add add-default-route=yes disabled=no interface=ether1 name=pppoe-out2 password=sub1 user=sub1
[admin@RouterOS] /interface pppoe-client> pr
Flags: X - disabled, I - invalid, R - running
 0  R name="pppoe-out2" max-mtu=auto max-mru=auto mrru=disabled interface=ether1 user="sub1" password="sub1" profile=default keepalive-timeout=10 service-name="" ac-name="" add-default-route=yes default-route-distance=1
      dial-on-demand=no use-peer-dns=no allow=pap,chap,mschap1,mschap2
[admin@RouterOS] /interface pppoe-client> /ip ad pr
Flags: X - disabled, I - invalid, D - dynamic
 #   ADDRESS            NETWORK         INTERFACE
 0 D 198.51.100.124/32  198.51.100.254  pppoe-out2
[admin@RouterOS] /interface pppoe-client> /ip ro pr
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          pppoe-out2                1
 1 ADC  198.51.100.254/32  198.51.100.124  pppoe-out2                0
[admin@RouterOS] /interface pppoe-client>

Alright, we can ping the gateway from both. Can we get beyond it?

sub03> trace 1.1.1.1
trace to 1.1.1.1, 8 hops max, press Ctrl+C to stop
 1   198.51.100.254   2.362 ms  1.396 ms  0.958 ms
 2     *  *  *
 3     *  *  *
^C 4

The BNGs is aware of this subscriber, however, we are not advertising anything to the core- yet.

We like connected things

If you recall, on the AGGs, there was a precise out filter on the peering to the core.

Well, the idea is to let the core know about some parts of the subnet, covered by this filter. And the easiest way is to have BGP to

  • Know there are some hosts running DHCP, probably via static routes pointing to the local interface.
  • PPPoE subs will already have a dynamic and connected route on the routing table.
  • Have BGP redistribute connected and statics, in case there are no PPPoE subscribers and we only have DHCP subscribers.
  • Aggretate all PPPoE interfaces into a supernet, because we are allowing up to /29
    This supernet will be a /25, because we created our internet pool from 198.51.100.1-198.51.100.127. Same concept applies for agg02, with the consideration that the aggregate will be 198.51.100.128/25
[admin@agg-bng-01] /routing bgp aggregate> add prefix=198.51.100.0/25 instance=default
[admin@agg-bng-01] /routing bgp aggregate> pr
Flags: X - disabled, A - active
 #   PREFIX               INSTANCE
 0   198.51.100.0/25      default
[admin@agg-bng-01] /routing bgp aggregate> set include-igp=yes
[admin@agg-bng-01] /routing bgp aggregate> ..instance
[admin@agg-bng-01] /routing bgp instance> set redistribute-connected=yes redistribute-static=yes
[admin@agg-bng-01] /routing bgp> advertisements pr
PEER     PREFIX               NEXTHOP          AS-PATH                                                                                                                                                                  ORIGIN     LOCAL-PREF
core-01  198.51.100.0/25      10.10.1.1                            

The include-igp setting will match all IGP routes, like connected routes and iBGP routes.

You can see that the core is aware of a part of the /24 exists on this BNG.

[admin@core-01] > /ip ro pr
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  0.0.0.0/0                          10.0.1.1                200
 1 ADC  10.0.0.0/30        10.0.0.2        ether1                    0
 2 ADo  10.0.1.1/32                        10.0.0.1                110
 3 ADC  10.1.1.1/32        10.1.1.1        lo0                       0
 4 ADo  10.10.1.1/32                       10.255.255.2            110
 5 ADo  10.10.1.2/32                       10.255.255.6            110
 6 ADC  10.255.255.0/30    10.255.255.1    ether2                    0
 7 ADC  10.255.255.4/30    10.255.255.5    ether3                    0
 8 ADb  198.51.100.0/24                    10.10.1.1               200
 9 ADb  198.51.100.0/25                    10.10.1.1               200

However, with this setup, we are still advertising the entire /24 to the core. Let’s adjust the filters on both routers to advertise only anything longer than 24.

[admin@agg-bng-01] /routing filter> pr
Flags: X - disabled
 0   chain=core-01-in prefix=0.0.0.0/0 invert-match=no action=accept set-bgp-prepend-path=""

 1   chain=core-01-in invert-match=no action=discard set-bgp-prepend-path=""

 2   chain=core-01-out prefix=198.51.100.0/24 prefix-length=24-29 invert-match=no action=accept set-bgp-prepend-path=""

 3   chain=core-01-out invert-match=no action=discard set-bgp-prepend-path=""
[admin@agg-bng-01] /routing filter> set prefix-length=25-29 2

[admin@agg-bng-01] /routing filter> pr
Flags: X - disabled
 0   chain=core-01-in prefix=0.0.0.0/0 invert-match=no action=accept set-bgp-prepend-path=""

 1   chain=core-01-in invert-match=no action=discard set-bgp-prepend-path=""

 2   chain=core-01-out prefix=198.51.100.0/24 prefix-length=25-29 invert-match=no action=accept set-bgp-prepend-path=""

 3   chain=core-01-out invert-match=no action=discard set-bgp-prepend-path=""
[admin@agg-bng-01] /routing filter> ..
[admin@agg-bng-01] /routing> bgp ad pr
PEER     PREFIX               NEXTHOP          AS-PATH                                                                                                                                                                  ORIGIN     LOCAL-PREF
core-01  198.51.100.0/25      10.10.1.1                                                                                                                                                                                 

And now, the routing table on our core looks as follows.

[admin@core-01] > ip ro pr
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  0.0.0.0/0                          10.0.1.1                200
 1 ADC  10.0.0.0/30        10.0.0.2        ether1                    0
 2 ADo  10.0.1.1/32                        10.0.0.1                110
 3 ADC  10.1.1.1/32        10.1.1.1        lo0                       0
 4 ADo  10.10.1.1/32                       10.255.255.2            110
 5 ADo  10.10.1.2/32                       10.255.255.6            110
 6 ADC  10.255.255.0/30    10.255.255.1    ether2                    0
 7 ADC  10.255.255.4/30    10.255.255.5    ether3                    0
 8 ADb  198.51.100.0/25                    10.10.1.1               200
 9 ADb  198.51.100.128/25                  10.10.1.2               200
[admin@core-01] >

Going out and beyond

For the sake of examples, I’m adding a lo1 interface on the edge with 1.1.1.1/32 to simulate an external host.

[admin@edge-01] > /inte bridge add name=lo1
[admin@edge-01] > /ip address add address=1.1.1.1/32 interface=lo1

And now, from sub4 for example, let’s run a traceroute to it.

[admin@RouterOS] /tool> traceroute 1.1.1.1
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 198.51.100.254                     0%    2   1.6ms     2.6     1.6     3.6       1
 2 10.255.255.5                       0%    2   3.2ms     3.8     3.2     4.3     0.6
 3 1.1.1.1                            0%    2   4.6ms     4.7     4.6     4.8     0.1

There is a special consideration here if you still haven’t noticed it.

How does agg1 knows about what’s happening on agg2. For example, if from agg1 we try to reach hosts on the 198.51.100.128/25 network, the immediate next hop is agg1 itself, because we have a DAC route pointing to 198.51.100.0/24

[admin@agg-bng-01] > /ip ro pr
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADb  0.0.0.0/0                          10.0.1.1                200
 1 ADo  10.0.0.0/30                        10.255.255.1            110
 2 ADo  10.0.1.1/32                        10.255.255.1            110
 3 ADo  10.1.1.1/32                        10.255.255.1            110
 4 ADC  10.10.1.1/32       10.10.1.1       lo0                       0
 5 ADo  10.10.1.2/32                       10.255.255.1            110
 6 ADC  10.255.255.0/30    10.255.255.2    ether1                    0
 7 ADo  10.255.255.4/30                    10.255.255.1            110
 8 ADC  198.51.100.0/24    198.51.100.254  ether2                    0
 9 ADbU 198.51.100.0/25                                             20
10 ADC  198.51.100.126/32  198.51.100.254  <pppoe-sub1-1>            0
11 ADC  198.51.100.127/32  198.51.100.254  <pppoe-sub1>              0

However, we already have all rhe routing info we need on the core.

The fix is simple and involves of course, filtering, but we’ll cover that on the next post.

The case for Anycast default gateways

Or to deliver public IPv4 addressing, without breaking the bank and your routing tables.

Years passed by, technology became cheaper and more powerful, making the Internet bigger and bigger. The cloud emerged, and we saw the rise and fall of many service providers all around the globe. Now your CPE had a public IP address, maybe your printer too, someone thought it was a good idea that your fridge could have a public address to, and maybe your home cameras and your microwave and your mobile and your cat and your toilet and so on.

The network became so big, that we saw an enormous increase on the number service providers and AS numbers. Everyone needed public addresses, maybe a /18, or a /20. Some poor bastards were only delivered a /22.

At this point, people started talking about IPv4 exhaustion.

Maybe you had a small network, emailed ARIN/LACNIC asking for moar addresses!, and they told you no.

Maybe you asked your upstream for a /26 in your DIA, and they told you no.

Maybe your customers asked for public addressing, and you realized that it was just wasteful to assign a /30 to anyone. Who can afford such a waste!?

IP addresses brokers are making a shitload of money out of this. Small operators need more addresses in order to achieve sustainable growth. Most RIRs will just tell you “Dude, there is nothing left!”

On this post we are not considering standard routed customers or where we can provision a a /30 or similar into the PE equipment, and let a routing protocol do its thing.

The question is, how do you provision an interface to your residential customers, in such a way that they can have a routable public IP address on their CPE, while keeping separate broadcast domains or VLANs for customer blocks, and doing all of this without wasting addresses in the process.

You probably have a router somewhere, with an IP address which serves as a default gateway for the entire segment. Maybe this router also serves DHCP, or acts a a PPPoE server, or any IP addresses provisioning method. How do achieve an efficient IP addressing schema, efficient route aggregation, and efficient layer 2 segmentation.

Yeah it would be easy to take a /21 of publics and putting them all on a VLAN. We’ll cover this option on next articles.

Many roads to the same place

For us small and medium operators, most typical efforts in IP addresses saving involves some sort of layer 2 extension, or subnetting into smaller blocks. Let’s look at some of these alternatives.

Big subnet, VLAN your way out, single access router

Simple enough. Put an entire /24, you will lose 3 addresses on the network, broadcast, and default gateway. Extend your customer VLAN over all the required switches in between.

The good part about this is that you will make an efficient usage of your addresses by only losing 3 out of 255, which I guess is a decent tradeoff.

Of course, this is an administration nightmare. Huge broadcast domain, VLANs that split over multiple switches in several locations, a strong requirement for DHCP snooping to prevent rouge servers, and a big STP tree to take care of.

Small subnets, VLANs everywhere, multiple access router

The obvious alternative is the exact opposite to the one we just saw. Let’s take a /24, split it into decent chunks, and put them into their own VLAN, which will be targeted into a separate access router.

This approach allows to segment the huge broadcast domain into several smaller VLANs, enabling us to keep possible broadcasts isolated into their own domain. You can also run MST (or PVST maybe) on top of it, to isolate loops into single instances of STP, instead of having a big spanning tree covering everything.

Even if this looks better, there is an obvious tradeoff. For every subnet, we still lose 3 addresses.

3 in 255 is not so much. If you split that into, let’s say /26s, you will need 3 for every network, broadcast and gateway address in every subnet. 3 addresses for 4 /26s subnet, is 12 wasted addresses. And 12 in 255 is not a minor thing. Specially once you remember that you have to pay a fee to your RIR for every address, every year.

Big subnet, VPLS your way out, single access router

This is a similar approach as the first one, where we used a big single layer 2 domain. We can make this layer 2 segment over VPLS tunnels, which will extend layer 2 using a MPLS overlay.

The IP addressing here can be the same used on the first scenario. A big /24 with a default gateway, thus losing 3 addresses out of 25.

This is a commonly used alternative which works, although I am not a fan of it. On this topology can run everything over a single (and different) transit VLANs between every router, but there are some requirements.

You will need to put layer 3 capable CPEs on every customer. Those CPEs will need to talk extended MTUs, be able to run MPLS/VPLS, and a routing protocol to readvertise their loopbacks into the aggregator router, to be able to terminate VPLS tunnels. And of course as we are making a big layer 2 domain, you still have to consider DHCP snooping, and possible loops.

An alternative without reinventing the wheel

The main focus is to avoid wasting IP addresses. Readers of this humble blog are medium size operators, and every penny spent on IP addressing can make a huge difference on a long term. Hopefully, proving that we are capable of making a difference on making a smart usage of addresses, can help us to present a business case to ARIN/LACNIC/whoever, to successfully request and get additional blocks of public addresses.

MPLS/VPLS solutions can work on top of this, but many operators do not have gear capable of talking such protocols, either because they are offloading their access layer into feature-limited software solutions, or because their hardware needs to be licensed to be MPLS capable. Also, many operators don’t have skilled MPLS engineers to design and support such network topologies.

Is there a way we can accomplish this in an easier way?

Anycast at rescue

Anycast is a network addressing and routing methodology in which a single destination IP address is shared by devices in multiple locations. Anycast usually comes to mind when we think about CDNs, DNS servers, and any destination that has to be present on multiple locations at the same time, on a same IP address.

For our purposes, our destination will be our default gateway.

An anycast IP address is not related at all with the kind of service present on the upper layers, so we can easily provide network services over an anycast address.

Surely you are familiar with CloudFlare 1.1.1.1 or Google 8.8.8.8, which are anycast DNS services. The 8.8.8.8 I ping from home is probably not the same 8.8.8.8 you can ping from your end.

Think twice about this. Anycast means we will have duplicate IP addresses in our network, by design. This is not VRRP or any kind of HSRP where you can have active/passive addresses. This is in fact, using the same IP address over many devices, on active interfaces.

Our new topology

On this scenario, we will share a default gateway on multiple devices.

The 10.10.10.254/24 address will be present and active on two routers, which face different layer 2 segments on the inside. For this example, I’m using a PE simulating DIA services on one side, and a BNG or PPPoE concentrator, to simulate residential services like FTTH or unlicensed wireless access.

Our objective here is to able to handoff /24 addressing to customers, so they can use the 10.10.10.254 address as their default gateway.

This address won’t be installed on any routing protocols, and instead, we are looking to advertise /32 (or bigger summaries) v4 prefixes over BGP, install them on the core router, and be able to advertise a single /24 summary from the core to the rest of the network. This core will be in fact, an aggregator too.

By doing this, the entire /24 can be subnetted up /32 advertisements for single hosts, or any other subnet like /25 or /26 for access subnets, while all the addresses usable inside those subnets. We can think of them as pools of addresses instead of subnets.

For example, 10.10.10.0-10.10.10.61 is the first /24, but customers inside will use a /24 submask, to be able to reach 10.10.10.254 on the AGG/BNG.

I will illustrate examples on both IOS and MikroTik platforms on the following steps.

GNS3 Topology

Our lab topology looks as follows. CORE, AGG and PE are all Cisco CSR10000v 16.12.03, and the BNG is Mikrotik CHR 6.48.6.

Core router

This will act as BGP RR for AS 65000. For sake of simplicity, all interfaces are on OSPF area 0 to reditribute loopbacks.

interface Loopback0
 ip address 10.1.1.1 255.255.255.255
!
interface GigabitEthernet1
 ip address 10.255.255.5 255.255.255.252
 negotiation auto
 no mop enabled
 no mop sysid
!
interface GigabitEthernet2
 ip address 10.255.255.1 255.255.255.252
 negotiation auto
 no mop enabled
 no mop sysid
!
router ospf 1
 router-id 10.1.1.1
 passive-interface default
 no passive-interface GigabitEthernet1
 no passive-interface GigabitEthernet2
 network 10.1.1.1 0.0.0.0 area 0
 network 10.255.255.0 0.0.0.3 area 0
 network 10.255.255.4 0.0.0.3 area 0
!
router bgp 65000
 bgp router-id 10.1.1.1
 bgp log-neighbor-changes
 neighbor 10.10.1.2 remote-as 65000
 neighbor 10.10.1.2 update-source Loopback0
 neighbor 10.10.1.3 remote-as 65000
 neighbor 10.10.1.3 update-source Loopback0
!

BNG Aggregation Router

This node will act as BGP RR client for AS 65000. As with the core, we are peering between loopbacks. I highlighted the default 10.10.10.254 gateway on ether2, which faces the VPCS host behind.

/interface bridge
add name=lo0
/routing bgp instance
set default as=65000 redistribute-static=yes
/routing ospf instance
set [ find default=yes ] router-id=10.10.10.2
/ip address
add address=10.10.1.3 interface=lo0 network=10.10.1.3
add address=10.255.255.2/30 interface=ether1 network=10.255.255.0
add address=10.10.10.254/24 interface=ether2 network=10.10.10.0
/routing bgp peer
add name=peer1 remote-address=10.1.1.1 remote-as=65000 update-source=lo0
/routing ospf interface
add passive=yes
add interface=ether1
/routing ospf network
add area=backbone network=10.255.255.0/30
add area=backbone network=10.10.1.3/32

Verifiying routing protocols

At this point, we should have a full neighborship relation on OSPF, sucessful loopbacks redistribution, and established peerings between CORE and BNG.

CORE#sh ip ospf neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface
10.10.1.3        1   FULL/BDR        00:00:34    10.255.255.2    GigabitEthernet2
10.10.1.2         1   FULL/BDR        00:00:37    10.255.255.6    GigabitEthernet1
CORE#sh ip bgp summ
CORE#sh ip bgp summary
BGP router identifier 10.1.1.1, local AS number 65000
BGP table version is 2, main routing table version 2
1 network entries using 248 bytes of memory
1 path entries using 136 bytes of memory
1/1 BGP path/bestpath attribute entries using 288 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 672 total bytes of memory
BGP activity 1/0 prefixes, 1/0 paths, scan interval 60 secs
1 networks peaked at 23:11:52 Aug 20 2022 UTC (00:09:10.360 ago)

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.10.1.2       4        65000     107     108        2    0    0 01:33:52        0
10.10.1.3       4        65000     100      96        2    0    0 01:25:23        1

CORE#sh ip ro
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
       n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       H - NHRP, G - NHRP registered, g - NHRP registration summary
       o - ODR, P - periodic downloaded static route, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 8 subnets, 2 masks
C        10.1.1.1/32 is directly connected, Loopback0
O        10.10.1.2/32 [110/2] via 10.255.255.6, 01:34:00, GigabitEthernet1
O        10.10.1.3/32 [110/11] via 10.255.255.2, 01:25:28, GigabitEthernet2
B        10.10.10.2/32 [200/0] via 10.10.1.3, 00:09:13
C        10.255.255.0/30 is directly connected, GigabitEthernet2
L        10.255.255.1/32 is directly connected, GigabitEthernet2
C        10.255.255.4/30 is directly connected, GigabitEthernet1
L        10.255.255.5/32 is directly connected, GigabitEthernet1

BNG Route Advertisements

We will handle advertisements with static routes using the interface as the gateway for the desired hosts. Other methods are valid, like redistributing connected routes for the case of PPPoE interface sessions.

/routing bgp instance
set default as=65000 redistribute-static=yes

/ip route
add distance=1 dst-address=10.10.10.2/32 gateway=ether2

On the CORE side, this static route advertisement will look like this.

CORE#sh ip bgp
BGP table version is 4, local router ID is 10.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
              t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i  10.10.10.2/32    10.10.1.3                     100      0 ?

PC2 VPCS Config

I’ll assign 10.10.10.2/24 to this host.

PC2> show ip

NAME        : PC2[1]
IP/MASK     : 10.10.10.2/24
GATEWAY     : 10.10.10.254
DNS         :
MAC         : 00:50:79:66:68:01
LPORT       : 20088
RHOST:PORT  : 127.0.0.1:20089
MTU         : 1500

Do we have a sucessful routing to the outside at this point? Let’s run a traceroute.

PC2> trace 10.1.1.1 -P 1
trace to 10.1.1.1, 8 hops max (ICMP), press Ctrl+C to stop
 1   10.10.10.254   0.941 ms  0.750 ms  0.828 ms
 2   10.1.1.1   2.662 ms  1.719 ms  1.811 ms

Awesome! At this point we have built a sucessful routed network – altough – this is nothing out of the ordinary.

Reusing default gateways

The VPCS host used 10.10.10.2/24 on a single layer 2 segment. Let’s consider the PE scenario, where we will asign 10.10.10.1/24, to another host, behind a PE, behing a totally different aggregator router. We want to keep 10.10.10.254/24 as the default gateway here.

AGG Config

The AGG router config is almost the same as the BNG. We are adding a vlan77 to manage the CPE behind the AGG.

interface Loopback0
 ip address 10.10.1.2 255.255.255.0
!
interface GigabitEthernet1
 ip address 10.255.255.6 255.255.255.252
 negotiation auto
!

router ospf 1
 router-id 10.10.1.2
 passive-interface default
 no passive-interface GigabitEthernet1
 network 10.10.1.2 0.0.0.0 area 0
 network 10.255.255.4 0.0.0.3 area 0
!
router bgp 65000
 bgp router-id 10.10.1.2
 bgp log-neighbor-changes
 neighbor 10.1.1.1 remote-as 65000
 neighbor 10.1.1.1 update-source Loopback0
!

PE Config

PE configuration is dead simple. An address on vlan16 for management, and a bridge-domain gi1 and gi4.

bridge-domain 1
 member GigabitEthernet1 service-instance 1
 member GigabitEthernet4 service-instance 1
!
interface GigabitEthernet1
 no ip address
 service instance 1 ethernet
  encapsulation untagged
 !
!
interface GigabitEthernet1.16
 encapsulation dot1Q 16
 ip address 172.16.100.100 255.255.0.0
!
interface GigabitEthernet4
 no ip address
 service instance 1 ethernet
  encapsulation untagged
 !
!

AGG BGP Advertisements

Here we are doing the same we did on the BNG; adding a static route for the destination host, and static redistribution under BGP.

ip route 10.10.10.1 255.255.255.255 GigabitEthernet2
!
router bgp 65000
 redistribute static

Verifying

At this point, we should see sucessful routing from the AGG to the CORE.

CORE#sh ip os neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface
10.10.1.3         1   FULL/BDR        00:00:33    10.255.255.2    GigabitEthernet2
10.10.1.2         1   FULL/BDR        00:00:37    10.255.255.6    GigabitEthernet1
CORE#sh ip bgp summ
CORE#sh ip bgp summary
BGP router identifier 10.1.1.1, local AS number 65000
BGP table version is 5, main routing table version 5
2 network entries using 496 bytes of memory
2 path entries using 272 bytes of memory
2/2 BGP path/bestpath attribute entries using 576 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1344 total bytes of memory
BGP activity 2/0 prefixes, 2/0 paths, scan interval 60 secs
2 networks peaked at 19:14:11 Aug 21 2022 UTC (00:05:04.304 ago)

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.10.1.2       4        65000    1427    1425        5    0    0 21:32:05        1
10.10.1.3       4        65000    1465    1416        5    0    0 21:23:36        1

CORE#sh ip bgp nei
CORE#sh ip bgp neighbors 10.10.1.2 re
CORE#sh ip bgp neighbors 10.10.1.2 rou
CORE#sh ip bgp neighbors 10.10.1.2 routes
BGP table version is 5, local router ID is 10.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
              t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i  10.10.10.1/32    10.10.1.2                0    100      0 ?

Total number of prefixes 1
CORE#sh ip ro
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
       n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       H - NHRP, G - NHRP registered, g - NHRP registration summary
       o - ODR, P - periodic downloaded static route, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 9 subnets, 2 masks
C        10.1.1.1/32 is directly connected, Loopback0
O        10.10.1.2/32 [110/2] via 10.255.255.6, 21:32:25, GigabitEthernet1
O        10.10.1.3/32 [110/11] via 10.255.255.2, 19:55:51, GigabitEthernet2
B        10.10.10.1/32 [200/0] via 10.10.1.2, 00:05:19
B        10.10.10.2/32 [200/0] via 10.10.1.3, 19:55:46
C        10.255.255.0/30 is directly connected, GigabitEthernet2
L        10.255.255.1/32 is directly connected, GigabitEthernet2
C        10.255.255.4/30 is directly connected, GigabitEthernet1
L        10.255.255.5/32 is directly connected, GigabitEthernet1

Ok, routes are there, how about reachability?

CORE#traceroute 10.10.10.1 source lo0
Type escape sequence to abort.
Tracing the route to 10.10.10.1
VRF info: (vrf in name/id, vrf out name/id)
1 10.255.255.6 4 msec 2 msec 1 msec
2 10.10.10.1 14 msec 4 msec 3 msec
CORE#

Looks good, and from the end host?

PC1> trace 10.1.1.1 -P 1
trace to 10.1.1.1, 8 hops max (ICMP), press Ctrl+C to stop
1 10.10.10.254 2.303 ms 1.750 ms 1.438 ms
2 10.1.1.1 2.495 ms 2.000 ms 1.838 ms

The first hop in path is the same as before, 10.10.10.254. We are resuing the same address on different routers, while keeping routing intact.

Stay tuned for the upcoming post where the BNG will act as a proper PPPoE termination point, and doing all of this dynamically without operator intervention.

MikroTik VPN with Windows NPS RADIUS

With the advance of cheap MikroTik routers and ready to use CHR instances, setting up a VPN concentrator for remote access has become an easy task. Moving even further, a single router could provide VPN access and dynamic routing to integrate remote networks to the backbone.

I have started a gig as a consultant and sysadmin for a logistics insurance company, and one of my first proposals was to improve the network access for road warriors and remote workers.

The past

There was a Proxmox hypervisor, with some Windows 2012 R2 servers, providing Terminal Services, to execute a locally installed client for an ERP system. Proxmox was also using iptables on its the Debian backend of the to masquerade the VM networks with a public IP address, for Internet connectivity, dstnat rules for a NGINX reverse proxy, and RDP for the Windows servers..

I guess we all know having internet-exposed RDP is not a good idea, even if it is running in a non default port, so the former sysadmin transitioned to a SSH tunnel system, where the users connected to the hypervisor via SSH to establish tunnel to the desired server.

This solution, which I considered not elegant, was the only available at the moment due to networking constraints of the VPS provider, so really it was the best they were able to do, and it worked fine for them.

Over the Proxmox hypervisor, they also had a MikroTik CHR instance, with a P1 license, which was used to make a L2TP tunnel to a RB2011UiAS-rm located on their HQ.

Networks behind the tunnel endpoints were routed with static routers, so I configured a quick multi-area OSPF routing system, with the directly connected networks on area 0, along with the /30 network of the tunnel. I added an additional area on both ends, for the future VPN networks. Once OSPF was working as expected, I remove the static routes.

Securing the tunnels

This interconnection via the L2TP tunnel was just plain ol’ L2TP, without IPsec. This is no bueno, and could be improved. Fortunately, IPsec configuration on MikroTik is trivial. Just select “Use IPsec” on both ends, and use the same IPsec pre-shared key.

Configuring via Winbox

This can of course be configured via CLI. Would you like some RouterOS configuration Ansible on next posts? Let me know in the comments.

/interface l2tp-server server
set authentication=mschap1,mschap2 default-profile=VPN enabled=yes ipsec-secret="PUT_A_SECRET_HERE" use-ipsec=yes

VPN profiles

It’s always a good idea to copy the default-encryption profile, and create a new one based on that template. I set up a local address which was of course, part of the networks announced in a separate area by the OSPF process. I also added a IP pool to be able to provide dynamic addresses for the VPN users.

PPP profile

Maybe you are aware that in the Cisco world, you have to use tcp adjust-mss to adjust the maximum TCP segment size, to advoid fragmentation of packets over the tunnel. Fortunately, this is configured by default on RouterOS.

We don’t want any fragmentation

Finally, to be able to redirect the dial-in to a RADIUS server, we need to instruct the PPP AAA system to use RADIUS, as shown next.

Setting up RADIUS authentication

RADIUS servers are very simple to set up on RouterOS.

Under the RADIUS submenu, add a new server for PPP service, and configure the following parameters.

  • IP address of the radius server
  • RADIUS secret
  • Authentication and accounting ports, usually 1812 and 1813. Some servers use 1645 for accounting. Those are all UDP.
  • REALM if your server supports that extension
  • Which source address should the router use for its NAS-IP-Address
Configuring RADIUS

Using Windows NPS as a RADIUS server

NPS can work without a Certificate Authority but if you are working in an Active Directory environment, you’ll save a lot of headaches by installing the CA role.

Installing NPS and CA
Selecting Roles

In my particular scenario, the server was not part of a domain so the certificate generation and association was skipped.

Once the roles have been configured, I headed to the NPS service configuration, and add new RADIUS client.

Make sure to match the RADIUS secret and the source IP address as you configured on the MikroTik side.

New RADIUS Client

Next, the network access policies. I wanted to match the NAS IPv4 address, and the authentication types. If you are not familiar with the RADIUS lang, NAS stands for Network Access Server, which in this case, is the MikroTik router which provides the VPN service.

MikroTik source address

I had to use a CHAP fallback due to some legacy devices withuout MSCHAP support.

Authentication methods

Next, I added a new Access condition, matched the NAS address once again, and selected the local server as point of authentication.

Authenticate on this server

Once everything was properly configured, I set up the VPN client on my side, which looks like as follows. The idea of using NPS as RADIUS was to be able to use my Windows account credentials for the VPN.

VPN client on Windows 10

I verified the successful authentication on the router logs, and the VPN was sucessfully connected.

Contributing to Netbox devices library

The Netbox community has launched a repository for standard devices.

https://github.com/netbox-community/devicetype-library

This comes handy for new and existing Netbox installations, because now you can populate your database with predefined device models for the most common networking gear manufacturers.

I have contributed to the repository and created the entire set of MikroTik routers and switches, updated to December 2019.

https://github.com/netbox-community/devicetype-library/tree/master/device-types/MikroTik

Feel free to use it and contribute!
And as always, you can find me on Github at https://github.com/baldoarturo

Upgrading a MikroTik CHR Cluster

I upgraded a CHR cluster with the main objectives of reduce costs, improve network redundancy and provide an easy administration for CHR instances. As explained in previous posts, CHR can be run on many popular hypervisors, and most users are having great success using Hyper-V Failover clusters or vSphere HA to provide highly available routers without depending on VRRP or other gateway redundancy protocols.

These virtual routers currently provide two main services besides routing for ISP customers. They act as PPPoE concentrator for FTTH users, and provide traffic shaping and policing depending on the customer service plan.

Server Hardware

For this node, I will use a 32 core Dell R730, with 32 GB of RAM, and a 500 GB RAID 10 storage. On future post, new hosts will be added to the cluster.

Unracking the server

Network Conectivity

This server comes with a 4 port Gigabit Ethernet NIC, which could be used without any issues with the ixgbe driver.

First idea was to use two ports in a LACP bundle, and the other two in separate port groups.

I had previous Netflow analysis where I saw a predictable traffic behavior, where most of the bandwidth usage was going from and to a CDN peer of the ISP network. Customers had a mix of public and private addresses of the Class B segment, and they were being moved to CG-NAT ranges. In other words, traffic from a specific set of addresses were going from and to a specific set of addresses.

Why not configure two port-channels, instead of using separate port groups? I tested and due to the nature of the IP addressing on the customer side of the routers, none of the available hashing modes for LACP allowed to achieve a decent distribution on both links of the port-channel.

So, for the purposes of this cluster, I added an Intel X520 dual SFP+ card, providing 20 Gbps conectivity to the CHR instances. Peak bandwidth usage was around 4200 Mbps, so this card is more than enough to allow for future grow.

Installing the Intel X520 NIC

The Intel X520 only supports Intel branded SFP modules, and this behavior can be tuned configuring the kernel module. However, for this particular scenario, where both ports will be connected to a top of rack Dell Force10 S4048-ON switch, I choosed to use DAC cables to keep things simple.

DAC cables on the switch
Connecting the server

The server is using ESXi 6.5 for the hypervisor. After booting, I noticed the NICs were being recognized as vmnic5 and vmnic6, but they were using the ixgbe driver and only establishing links at 1 Gbps.

I downloaded the ixgben driver which is provided by VMware itself here and uploaded it to ESXi via SFTP.

For all my SFTP needs, my tool of choice always is Bitwise SSH client.

Once uploaded, I installed the offline bundle with the following command line.

[root@esxi] esxcli software vib install -d "/complete/path/to/the/driver/bundle"

Then I followed the KB article to disable the native ixgbe driver and use the new one. First, I placed the host on maintenance mode, and then I executedthe following to disable the driver.

[root@esxi] esxcli system module set --enabled=false --module=ixgbe

After a reboot, the new ixgben driver was loaded, and the NICs were establishing links at 10 Gbps.

I added the new NICs to the previously created virtual switches, checked the correct assignments of port groups, and then migrated the VMs to this host.

Building a network on Entre Ríos

It is always nice to fly. I took two flights, the first one with a little stop at Aeroparque (AEP), and then a short one to Paraná city (PRA). The skies were just beautiful.

Travelling MDZ to AEP

My current company is establishing operations on Entre Rios province, where a we are initiating a brand-new ISP service for the towns of Crespo, Libertador San Martin, and Paraná City. This was the main task among another small consulting and assistance.

My first time seeing the mighty Paraná river

Connecting People

Service is provided with two providers, and BGP sessions must be established with both to announce a /24 prefix of our AS, and probably receive just a default route from the upstream. There is no need to use the full table- yet. Both providers has approximately the same AS-PATH.

We’ll use a MikroTik CCR1036-8G-2S+ as the border router. Although it has SFP+ ports to allow 10 Gbps operation, at the moment the links will be negotiated using SFP modules at 1000 Mbps.

Main customer will be directly connected to this router using copper at 1 Gbps. They are using a MikroTik CRS326-24G-2S+ for their edge router, which will be enough for their 100 Mbps service. They provide us co-location too, so I installed the core router on their shelter, which is backed up by dual A/C systems and dual UPS-rectifier systems.

The new router racked and powered up
We’ll have some mate while waiting for the upstream provider port to go into no shutdown

Once the upstream was go, I was able to see they were in fact sending us the full BGP table, which we don’t need yet, so a couple route filters were configured to use put only a default gateway on the main routing table. As the default route was configured as a static one, the route filter policy was as easy as discard all BGP inputs.

[rootmin@ROUTER-EDUC.AR-PARANA] > routing filter export 
# jun/18/2019 16:24:37 by RouterOS 6.42.6
#
/routing filter
add action=discard chain=dynamic-in protocol=bgp

On this site there was also an Ubiquiti AirFiber 11X wireless link to reach Libertador San Martín town. Both radios were previously installed but not configured, so I connected to the radio and the site and configured it as Master. We traveled to the remote end, configured the radio as Slave and it worked just fine. Ubiquiti is getting up to date with their firmwares and UI, and it has became pretty straight forward to get a link working, even for someone with little or none networking skills.

¿Do you think this ease-of-use is making the job easier for us, or is it the start point of a madness of wireless spectrum usage?

From this node at Libertador, we installed two single-mode fiber lines, one to connect the town Hospital and another for the town University. MikroTik CRS326-24G-2S+ switches were installed at each site to be used as CPEs.

All monitoring, reporting and backup systems were previously configured at our NOC, so that was ll for us on the site.

Watching cars go by

I also assisted a brand new urban surveillance camera installation on the entrance of the Raúl Uranga – Carlos Sylvestre Begnis Subfluvial Tunnel. The objective was to read license plates on this strategic points, which is one of the few exits outside the Paraná river, and the one which has the most vehicle traffic.

Previously we had selected a Hikvision DS-2CD4A26FWD-IZHS8/P (yep, that’s the model name) camera which was already installed by Policía of Entre Ríos technician. This camera was specifically designed for licence-plate recognition (LPR). It supports OCR on hardware and works in very low light conditions, as low as 0.0027 lux.

Traces of Paraná City

I stayed at Hotel Howard Johnson Plaza Resort & Casino Mayorazgo, and I encourage you to visit it. The rooms are lovely and the staff is excellent.

My view from the hotel room

Be sure to schedule time to walk on the Paraná river borders, visit the Martiniano Leguizamón historic town museum and enjoy yourself. This is a beautiful city.

Blue skies at Crespo, Entre Ríos

Using The Dude on MikroTik CHR

The Dude network monitor is RouterOS package intended to manage a network environment. It automatically scan all devices within specified subnets, draw and layout a network maps, monitor services, and alert you in case of problems.

Previous versions of The Dude were developed as Windows x86 software, but later versions went through a full rebuild, and now it is distributed as a RouterOS package. This comes handy as the same RouterOS instance can be linked to the network, eliminating the the need for additional VPNs on servers or gateways. Instead, all tunneling can be done inside the CHR instance.

The Windows versions also had a web GUI which was, awful. For all the new editions, you’ll need a software client available on https://download.mikrotik.com/routeros/6.43.14/dude-install-6.43.14.exe

It will update itself whenever you connect to a newer RouterOS version. Just be sure to run it as administrator on W10.

Installing

Get the CHR package from https://download.mikrotik.com/routeros/6.43.14/dude-6.43.14.npk.

Once downloaded, upload it to the CHR instance via Winbox drag-and-drop, FTP client, or just download it from inside chr:

Downloading from CHR

Reboot the CHR instance, and you will find the new Dude menu inside Winbox.

New Dude menu

Head to Dude > Settings and tick Enabled to enable the server. A few folders will be created on the filesystem, and the server will be ready to accept connections on port 8291. The previous x86 based versions of Dude used port TCP/2210 or TCP/2211, but on this new integrated RouterOS package, all the management is handled on the same port as Winbox.

If you still don’t have the client, get it on https://download.mikrotik.com/routeros/6.43.14/dude-install-6.43.14.exe.

One you connect, the following window should appear by default. You can run a discover for multiple networks and let Dude map your network for you, but it will only disconver layer 3 adyancencies. In order to have complete control over the monitoring, I suggest to build your backbone manually and let the autodiscovery handle your management VLANs/VRFs.

Introduction to MikroTik CHR

MikroTik Cloud Hosted Router (CHR) is a RouterOS version intended to be used as a virtual machine instance.

It runs on x86-64-bit architecture and can be deployed on most hypervisors such as:

  • VMWare, ESXi, Player and Workstation
  • Microsoft Hyper-V
  • Oracle VirtualBox
  • KVM‌
  • And others, like Xen, but I haven’t tested it yet

Some special requeriments apply depending on the subyacent hypervisor.

ESXi

Network adapters must be vmxnet3 or E1000‌. Just use vmxnet3 to get the most. Disks must be IDE, VMware paravirtual SCSI, LSI Logic SAS or LSI Logic Parallel.

Hyper-V

Network adapters must be Network adapter or Legacy Network adapter .Disks IDE or SCSI.

Qemu/KVM

Virtio, E1000 or vmxnet3 NICs. IDE, Sata or Virtio disks.

VirtualBox

Networking using E1000 or rtl8193, and disks with IDE, SATA, SCSI or SAS interfaces.

Licensing

The CHR images have full RouterOS features enabled by default, but they use a different licensing model than other RouterOS versions.

Paid licenses

p1

p1 (perpetual-1), which allows CHR to run indefinitely. It comes with a limit of 1Gbps upload per interface. All the rest of the features provided by CHR are available without restrictions. It can be upgraded p1 to p10 or p-unlimited.

p10

p10 (perpetual-10), which also allows CHR to run indefinitely, with a 10Gbps upload limit per interface. All features are available without restrictions. It can be upgraded to p-unlimited.

p-unlimited (really?)

The p-unlimited (perpetual-unlimited) license level allows CHR to run indefinitely. It is the highest tier license and it has no enforced limitations.

Free licenses (yay!)

There are two ways to use and try CHR free of charge.

free

The free license level allows CHR to run indefinitely, with a limit of 1Mbps upload per interface. All the rest of the features have no restrictions. This level comes activated by default on all images.

60-day trial

Th p1/p10/pU licenses can be tested with a 60 days trial.

Cool. How can i try it?

The easiest way to spin up a working instance of CHR is using the OVA appliance provided by MikroTik.

https://download2.mikrotik.com/routeros/6.43.14/chr-6.43.14.ova

Deployment on ESXi

Once downloaded, the OVA can be used to deploy a new instance. I’ll be using ESXi on this example. The OVA comes preconfigured with a single network adapter, but more interfaces can be added on a later stage.

Creating new VM from OVA template
Setting VM name, and uploading OVA file
I’ll use local storage for it
Thin provisioned disks, and a previously configured VM network
Review everything, and deploy

Initial Configuration

After the VM boots, log in via CLI with the default credentials:

  • Username: admin
  • Password: none

CHR comes with a free licence‌ by default, limited to 1Mbps upload limit. This is handy for lab purposes, or low traffic scenarios like stand-alone DHCP servers.

A DHCP client is enabled by default on the single existing ether1 interface. Use any of the following methods to find out the adquired address.

/ip dhcp-client print
/ip address print

Let’s get a trial licence. You will need the credentials for your MikroTik account. If you don’t have a MikroTik account, get one here.

The CHR instance will also need Internet access, so be sure to connect the virtual NIC to a VM network where it can make its way to the outside.

[admin@CHR] > sys license renew account=your@account.com password=yourpassword level=

Level ::= p-unlimited | p1 | p10

Once you request a trial license, check the status with

[admin@CHR] > sys lic print
        system-id: 0ywIRMYrtGA
            level: p1
  next-renewal-at: may/05/2019 17:59:59
      deadline-at: jun/04/2019 17:59:59

We’ll install The Dude on the next post, and configure it for some custom monitoring.