On a previous post we saw an example of a network access topology running anycast default gateways.
The idea is to save IPv4 addresses, without other methods than standard routing protocols. Just plain BGP that can be implemented on most vendors, either via hardaware appliances or virtualized network devices.
For the following examples, I’ll simulate a public /24 prefix using the 198.51.100.0/24 which is reserved by IANA as TEST-NET-2 for documentation.
Lab Network
This lab network comprises 2 AGG/BNG routers, a single core router (which will perform additional aggregation), and a single edge router. All devices are MikroTik RouterOS 6.48.6 CHR instances.
edge-01
/interface bridge
add name=lo0
/ip address
add address=10.0.1.1 interface=lo0 network=10.0.1.1
add address=10.0.0.1/30 interface=ether1 network=10.0.0.0
/routing ospf instance
set [ find default=yes ] router-id=10.0.1.1
/routing ospf interface
add passive=yes
add interface=ether1 network-type=point-to-point
/routing ospf network
add area=backbone network=10.0.0.0/30
add area=backbone network=10.0.1.1/32
/routing bgp instance
set default as=65000 router-id=10.0.1.1
/routing bgp peer
add default-originate=always in-filter=core-01-in name=core-01 out-filter=core-01-out remote-address=10.1.1.1 remote-as=65000 update-source=lo0
/routing filter
add action=accept chain=core-01-in prefix=198.51.100.0/24
add action=discard chain=core-01-in
add action=accept chain=core-01-out prefix=0.0.0.0/0
add action=discard chain=core-01-out
/system identity
set name=edge-01The “edge” router is peering with the “core” through their loopbacks, and just advertising a default to it (or cores, on a future stage), and accepting the entire 198.51.100.0/24.
core-01
/interface bridge
add name=lo0
/ip address
add address=10.1.1.1 interface=lo0 network=10.1.1.1
add address=10.0.0.2/30 interface=ether1 network=10.0.0.0
add address=10.255.255.1/30 interface=ether2 network=10.255.255.0
add address=10.255.255.5/30 interface=ether3 network=10.255.255.4
/routing bgp instance
set default as=65000 cluster-id=10.1.1.1 router-id=10.1.1.1
/routing bgp peer
add in-filter=edge-01-in name=edge-01 out-filter=edge-01-out remote-address=10.0.1.1 remote-as=65000 update-source=lo0
/routing bgp peer
add in-filter=edge-01-in name=edge-01 out-filter=edge-01-out remote-address=10.0.1.1 remote-as=65000 update-source=lo0
add default-originate=if-installed in-filter=agg-bng-in name=agg-bng-01 out-filter=agg-bng-out remote-address=10.10.1.1 remote-as=65000 route-reflect=yes update-source=lo0
add default-originate=if-installed in-filter=agg-bng-in name=agg-bng-02 out-filter=agg-bng-out remote-address=10.10.1.2 remote-as=65000 route-reflect=yes update-source=lo0
/routing filter
add action=accept chain=edge-01-in prefix=0.0.0.0/0
add action=discard chain=edge-01-in
add action=accept chain=edge-01-out prefix=198.51.100.0/24
add action=discard chain=edge-01-out
/routing filter
add action=accept chain=agg-bng-in prefix=198.51.100.0/24 prefix-length=24-29
add action=discard chain=agg-bng-in
add action=accept chain=agg-bng-out prefix=0.0.0.0/0
add action=discard chain=agg-bng-out
/routing ospf instance
set [ find default=yes ] router-id=10.1.1.1
/routing ospf interface
add passive=yes
add interface=ether1 network-type=point-to-point
add interface=ether2 network-type=point-to-point
add interface=ether3 network-type=point-to-point
/routing ospf network
add area=backbone network=10.0.0.0/30
add area=backbone network=10.1.1.1/32
add area=backbone network=10.255.255.0/30
add area=backbone network=10.255.255.4/30
/system identity
set name=core-01The “core” is peering with the “edge” of course, and also with two BNGs named as agg-bng-xx. This core is advertising its default to them, and accepting all prefixes within 198.51.100.254/24, with a prefix length up to /29.
If you come from a IOS land, this syntax would be something like this.
ip prefix-list BNG
permit 5 198.51.100.254/24 ge 24 le 29agg-bng-01
/interface bridge
add name=lo0
/ip address
add address=10.10.1.1 interface=lo0 network=10.10.1.1
add address=10.255.255.2/30 interface=ether1 network=10.255.255.0
add address=198.51.100.254/24 interface=ether2 network=198.51.100.0
/routing ospf instance
set [ find default=yes ] router-id=10.10.1.1
/routing ospf interface
add passive=yes
add interface=ether1 network-type=point-to-point
/routing ospf network
add area=backbone network=10.255.255.0/30
add area=backbone network=10.10.1.1/32
/routing bgp instance
set default as=65000 router-id=10.10.1.1
/routing bgp peer
add in-filter=core-01-in name=core-01 out-filter=core-01-out remote-address=10.1.1.1 remote-as=65000 route-reflect=yes update-source=lo0
/routing filter
add action=accept chain=core-01-in prefix=0.0.0.0/0
add action=discard chain=core-01-in
add action=accept chain=core-01-out prefix=198.51.100.0/24 prefix-length=24-29
add action=discard chain=core-01-out
/system identity
set name=agg-bng-01agg-bng-02
/interface bridge
add name=lo0
/ip address
add address=10.10.1.2 interface=lo0 network=10.10.1.2
add address=10.255.255.6/30 interface=ether1 network=10.255.255.0
add address=198.51.100.254/24 interface=ether2 network=198.51.100.0
/routing ospf instance
set [ find default=yes ] router-id=10.10.1.2
/routing ospf interface
add passive=yes
add interface=ether1 network-type=point-to-point
/routing ospf network
add area=backbone network=10.255.255.4/30
add area=backbone network=10.10.1.2/32
/routing bgp instance
set default as=65000 router-id=10.10.1.2
/routing bgp peer
add in-filter=core-01-in name=core-01 out-filter=core-01-out remote-address=10.1.1.1 remote-as=65000 route-reflect=yes update-source=lo0
/routing filter
add action=accept chain=core-01-in prefix=0.0.0.0/0
add action=discard chain=core-01-in
add action=accept chain=core-01-out prefix=198.51.100.0/24 prefix-length=24-29
add action=discard chain=core-01-out
/system identity
set name=agg-bng-02Finally, the BNGs are peering with the core, accepting a default, and allowing any advertisements from 198.51.100.254/24 from /24 to /29.
Both routers have 198.51.100.254/24 as the anycast default gateway.
If you wonder hor this works, this lab network is similar to the one on the previous post which you can check here.
BNG PPPoE and DHCP
We will start by setting up PPPoE services on our BNGs.
At this point we will work it with local secrets and keeping all the AAA process in the same router, with RADIUS as a future option.
Be aware that RouterOS by default will try its best to adjust the TCP MSS to match the PPPoE interface MTU.
Also, this being PPPoE, we have no restrictions on using the first address on the network as the PPP connection will not care about it being a network address. However, this will have the obvious restrictions and behavior if we run DHCP.
We will also skip the .254 address on the address pool as this is assigned to the ether2 interface on both routers as our anycast default gateway.
agg-bng-01
/ip pool
add name=internet ranges=198.51.100.0-198.51.100.127
/ppp profile
add local-address=198.51.100.254 name=internet remote-address=internet
/ppp secret
add name=sub1 password=sub1 profile=internet
add name=sub2 password=sub2 profile=internet
/interface pppoe-server server
add default-profile=internet interface=ether2 disabled=no
agg-bng-02
/ip pool
add name=internet ranges=198.51.100.128-198.51.100.253
/ppp profile
add local-address=198.51.100.254 name=internet remote-address=internet
/ppp secret
add name=sub3 password=sub3 profile=internet
add name=sub4 password=sub4 profile=internet
/interface pppoe-server server
add default-profile=internet interface=ether2 disabled=no
For DHCP, we will reuse the same previously created address pool. The following config applies to both routers.
/ip dhcp-server
add address-pool=internet disabled=no interface=ether2 name=dhcp1
/ip dhcp-server network
add address=198.51.100.0/24 gateway=198.51.100.254 netmask=24Finally, we will add some test subscribers. A dumb switch will act as the last-mile technology which could be xPON, wireless, DSL, you name it. All the subs are running RouterOS 6.48.6, and this is just to have something capable to talk PPPoE. There is also a VPCS 0.8.2 which comes by default with GNS3.
Address me, father
Starting with sub03 VPCS, we will ask DHCP to the BNG.
Welcome to Virtual PC Simulator, version 0.8.2
Dedicated to Daling.
Build time: Aug 23 2021 11:15:00
Copyright (c) 2007-2015, Paul Meng ([email protected])
All rights reserved.
VPCS is free software, distributed under the terms of the "BSD" licence.
Source code and license can be found at vpcs.sf.net.
For more information, please visit wiki.freecode.com.cn.
Press '?' to get help.
Executing the startup file
sub03> ip dhcp
DORA IP 198.51.100.252/24 GW 198.51.100.254
sub03> ping 198.51.100.254
84 bytes from 198.51.100.254 icmp_seq=1 ttl=64 time=1.278 ms
84 bytes from 198.51.100.254 icmp_seq=2 ttl=64 time=1.234 ms
84 bytes from 198.51.100.254 icmp_seq=3 ttl=64 time=0.946 ms
^C
If you pay attention, we did get the .252 address, instead of the .253.
sub4 had probably requested this one before, as RouterOS by default comes with a DHCP client on ether1. Is this the case?
[admin@RouterOS] > /ip ad pr
Flags: X - disabled, I - invalid, D - dynamic
# ADDRESS NETWORK INTERFACE
0 D 198.51.100.253/24 198.51.100.0 ether1 Indeed, both are running DHCP. And just for reference, this is how it looks from the BNG.
[admin@agg-bng-02] /ip dhcp-server> lease pr
Flags: X - disabled, R - radius, D - dynamic, B - blocked
# ADDRESS MAC-ADDRESS HOST-NAME SERVER RATE-LIMIT STATUS LAST-SEEN
0 D 198.51.100.253 0C:04:49:87:00:00 RouterOS dhcp1 bound 2m34s
1 D 198.51.100.252 00:50:79:66:68:01 sub03 dhcp1 bound 3m52sSame is happening with sub01 and sub02, however we’ll remove the DHCP client and add a PPPoE client.
[admin@RouterOS] /interface pppoe-client> add interface=ether1 user=sub1 password=sub1 add-default-route=yes
[admin@RouterOS] /interface pppoe-client> pr
Flags: X - disabled, I - invalid, R - running
0 X name="pppoe-out2" max-mtu=auto max-mru=auto mrru=disabled interface=ether1 user="sub1" password="sub1" profile=default keepalive-timeout=10 service-name="" ac-name="" add-default-route=yes default-route-distance=1
dial-on-demand=no use-peer-dns=no allow=pap,chap,mschap1,mschap2
[admin@RouterOS] /interface pppoe-client> ena 0
[admin@RouterOS] /interface pppoe-client> pr
Flags: X - disabled, I - invalid, R - running
0 R name="pppoe-out2" max-mtu=auto max-mru=auto mrru=disabled interface=ether1 user="sub1" password="sub1" profile=default keepalive-timeout=10 service-name="" ac-name="" add-default-route=yes default-route-distance=1
dial-on-demand=no use-peer-dns=no allow=pap,chap,mschap1,mschap2
[admin@RouterOS] /interface pppoe-client> /ip ad pr
Flags: X - disabled, I - invalid, D - dynamic
# ADDRESS NETWORK INTERFACE
0 D 198.51.100.125/32 198.51.100.254 pppoe-out2
[admin@RouterOS] /interface pppoe-client>
This config looks as follows on sub02.
[admin@RouterOS] >
[admin@RouterOS] > /ip dhcp-client
[admin@RouterOS] /ip dhcp-client> remove [find]
[admin@RouterOS] /ip dhcp-client> /
[admin@RouterOS] > /interface pppoe-client
[admin@RouterOS] /interface pppoe-client> add add-default-route=yes disabled=no interface=ether1 name=pppoe-out2 password=sub1 user=sub1
[admin@RouterOS] /interface pppoe-client> pr
Flags: X - disabled, I - invalid, R - running
0 R name="pppoe-out2" max-mtu=auto max-mru=auto mrru=disabled interface=ether1 user="sub1" password="sub1" profile=default keepalive-timeout=10 service-name="" ac-name="" add-default-route=yes default-route-distance=1
dial-on-demand=no use-peer-dns=no allow=pap,chap,mschap1,mschap2
[admin@RouterOS] /interface pppoe-client> /ip ad pr
Flags: X - disabled, I - invalid, D - dynamic
# ADDRESS NETWORK INTERFACE
0 D 198.51.100.124/32 198.51.100.254 pppoe-out2
[admin@RouterOS] /interface pppoe-client> /ip ro pr
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADS 0.0.0.0/0 pppoe-out2 1
1 ADC 198.51.100.254/32 198.51.100.124 pppoe-out2 0
[admin@RouterOS] /interface pppoe-client>
Alright, we can ping the gateway from both. Can we get beyond it?
sub03> trace 1.1.1.1
trace to 1.1.1.1, 8 hops max, press Ctrl+C to stop
1 198.51.100.254 2.362 ms 1.396 ms 0.958 ms
2 * * *
3 * * *
^C 4
The BNGs is aware of this subscriber, however, we are not advertising anything to the core- yet.
We like connected things
If you recall, on the AGGs, there was a precise out filter on the peering to the core.
Well, the idea is to let the core know about some parts of the subnet, covered by this filter. And the easiest way is to have BGP to
- Know there are some hosts running DHCP, probably via static routes pointing to the local interface.
- PPPoE subs will already have a dynamic and connected route on the routing table.
- Have BGP redistribute connected and statics, in case there are no PPPoE subscribers and we only have DHCP subscribers.
- Aggretate all PPPoE interfaces into a supernet, because we are allowing up to /29
This supernet will be a /25, because we created our internet pool from 198.51.100.1-198.51.100.127. Same concept applies for agg02, with the consideration that the aggregate will be 198.51.100.128/25
[admin@agg-bng-01] /routing bgp aggregate> add prefix=198.51.100.0/25 instance=default
[admin@agg-bng-01] /routing bgp aggregate> pr
Flags: X - disabled, A - active
# PREFIX INSTANCE
0 198.51.100.0/25 default
[admin@agg-bng-01] /routing bgp aggregate> set include-igp=yes
[admin@agg-bng-01] /routing bgp aggregate> ..instance
[admin@agg-bng-01] /routing bgp instance> set redistribute-connected=yes redistribute-static=yes
[admin@agg-bng-01] /routing bgp> advertisements pr
PEER PREFIX NEXTHOP AS-PATH ORIGIN LOCAL-PREF
core-01 198.51.100.0/25 10.10.1.1 The include-igp setting will match all IGP routes, like connected routes and iBGP routes.
You can see that the core is aware of a part of the /24 exists on this BNG.
[admin@core-01] > /ip ro pr
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADb 0.0.0.0/0 10.0.1.1 200
1 ADC 10.0.0.0/30 10.0.0.2 ether1 0
2 ADo 10.0.1.1/32 10.0.0.1 110
3 ADC 10.1.1.1/32 10.1.1.1 lo0 0
4 ADo 10.10.1.1/32 10.255.255.2 110
5 ADo 10.10.1.2/32 10.255.255.6 110
6 ADC 10.255.255.0/30 10.255.255.1 ether2 0
7 ADC 10.255.255.4/30 10.255.255.5 ether3 0
8 ADb 198.51.100.0/24 10.10.1.1 200
9 ADb 198.51.100.0/25 10.10.1.1 200
However, with this setup, we are still advertising the entire /24 to the core. Let’s adjust the filters on both routers to advertise only anything longer than 24.
[admin@agg-bng-01] /routing filter> pr
Flags: X - disabled
0 chain=core-01-in prefix=0.0.0.0/0 invert-match=no action=accept set-bgp-prepend-path=""
1 chain=core-01-in invert-match=no action=discard set-bgp-prepend-path=""
2 chain=core-01-out prefix=198.51.100.0/24 prefix-length=24-29 invert-match=no action=accept set-bgp-prepend-path=""
3 chain=core-01-out invert-match=no action=discard set-bgp-prepend-path=""
[admin@agg-bng-01] /routing filter> set prefix-length=25-29 2
[admin@agg-bng-01] /routing filter> pr
Flags: X - disabled
0 chain=core-01-in prefix=0.0.0.0/0 invert-match=no action=accept set-bgp-prepend-path=""
1 chain=core-01-in invert-match=no action=discard set-bgp-prepend-path=""
2 chain=core-01-out prefix=198.51.100.0/24 prefix-length=25-29 invert-match=no action=accept set-bgp-prepend-path=""
3 chain=core-01-out invert-match=no action=discard set-bgp-prepend-path=""
[admin@agg-bng-01] /routing filter> ..
[admin@agg-bng-01] /routing> bgp ad pr
PEER PREFIX NEXTHOP AS-PATH ORIGIN LOCAL-PREF
core-01 198.51.100.0/25 10.10.1.1 And now, the routing table on our core looks as follows.
[admin@core-01] > ip ro pr
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADb 0.0.0.0/0 10.0.1.1 200
1 ADC 10.0.0.0/30 10.0.0.2 ether1 0
2 ADo 10.0.1.1/32 10.0.0.1 110
3 ADC 10.1.1.1/32 10.1.1.1 lo0 0
4 ADo 10.10.1.1/32 10.255.255.2 110
5 ADo 10.10.1.2/32 10.255.255.6 110
6 ADC 10.255.255.0/30 10.255.255.1 ether2 0
7 ADC 10.255.255.4/30 10.255.255.5 ether3 0
8 ADb 198.51.100.0/25 10.10.1.1 200
9 ADb 198.51.100.128/25 10.10.1.2 200
[admin@core-01] >
Going out and beyond
For the sake of examples, I’m adding a lo1 interface on the edge with 1.1.1.1/32 to simulate an external host.
[admin@edge-01] > /inte bridge add name=lo1
[admin@edge-01] > /ip address add address=1.1.1.1/32 interface=lo1
And now, from sub4 for example, let’s run a traceroute to it.
[admin@RouterOS] /tool> traceroute 1.1.1.1
# ADDRESS LOSS SENT LAST AVG BEST WORST STD-DEV STATUS
1 198.51.100.254 0% 2 1.6ms 2.6 1.6 3.6 1
2 10.255.255.5 0% 2 3.2ms 3.8 3.2 4.3 0.6
3 1.1.1.1 0% 2 4.6ms 4.7 4.6 4.8 0.1
There is a special consideration here if you still haven’t noticed it.
How does agg1 knows about what’s happening on agg2. For example, if from agg1 we try to reach hosts on the 198.51.100.128/25 network, the immediate next hop is agg1 itself, because we have a DAC route pointing to 198.51.100.0/24
[admin@agg-bng-01] > /ip ro pr
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADb 0.0.0.0/0 10.0.1.1 200
1 ADo 10.0.0.0/30 10.255.255.1 110
2 ADo 10.0.1.1/32 10.255.255.1 110
3 ADo 10.1.1.1/32 10.255.255.1 110
4 ADC 10.10.1.1/32 10.10.1.1 lo0 0
5 ADo 10.10.1.2/32 10.255.255.1 110
6 ADC 10.255.255.0/30 10.255.255.2 ether1 0
7 ADo 10.255.255.4/30 10.255.255.1 110
8 ADC 198.51.100.0/24 198.51.100.254 ether2 0
9 ADbU 198.51.100.0/25 20
10 ADC 198.51.100.126/32 198.51.100.254 <pppoe-sub1-1> 0
11 ADC 198.51.100.127/32 198.51.100.254 <pppoe-sub1> 0
However, we already have all rhe routing info we need on the core.
The fix is simple and involves of course, filtering, but we’ll cover that on the next post.
