I upgraded a CHR cluster with the main objectives of reduce costs, improve network redundancy and provide an easy administration for CHR instances. As explained in previous posts, CHR can be run on many popular hypervisors, and most users are having great success using Hyper-V Failover clusters or vSphere HA to provide highly available routers without depending on VRRP or other gateway redundancy protocols.
These virtual routers currently provide two main services besides routing for ISP customers. They act as PPPoE concentrator for FTTH users, and provide traffic shaping and policing depending on the customer service plan.
For this node, I will use a 32 core Dell R730, with 32 GB of RAM, and a 500 GB RAID 10 storage. On future post, new hosts will be added to the cluster.
This server comes with a 4 port Gigabit Ethernet NIC, which could be used without any issues with the ixgbe driver.
First idea was to use two ports in a LACP bundle, and the other two in separate port groups.
I had previous Netflow analysis where I saw a predictable traffic behavior, where most of the bandwidth usage was going from and to a CDN peer of the ISP network. Customers had a mix of public and private addresses of the Class B segment, and they were being moved to CG-NAT ranges. In other words, traffic from a specific set of addresses were going from and to a specific set of addresses.
Why not configure two port-channels, instead of using separate port groups? I tested and due to the nature of the IP addressing on the customer side of the routers, none of the available hashing modes for LACP allowed to achieve a decent distribution on both links of the port-channel.
So, for the purposes of this cluster, I added an Intel X520 dual SFP+ card, providing 20 Gbps conectivity to the CHR instances. Peak bandwidth usage was around 4200 Mbps, so this card is more than enough to allow for future grow.
The Intel X520 only supports Intel branded SFP modules, and this behavior can be tuned configuring the kernel module. However, for this particular scenario, where both ports will be connected to a top of rack Dell Force10 S4048-ON switch, I choosed to use DAC cables to keep things simple.
The server is using ESXi 6.5 for the hypervisor. After booting, I noticed the NICs were being recognized as vmnic5 and vmnic6, but they were using the ixgbe driver and only establishing links at 1 Gbps.
I downloaded the ixgben driver which is provided by VMware itself here and uploaded it to ESXi via SFTP.
It is always nice to fly. I took two flights, the first one with a little stop at Aeroparque (AEP), and then a short one to Paraná city (PRA). The skies were just beautiful.
My current company is establishing operations on Entre Rios province, where a we are initiating a brand-new ISP service for the towns of Crespo, Libertador San Martin, and Paraná City. This was the main task among another small consulting and assistance.
Service is provided with two providers, and BGP sessions must be established with both to announce a /24 prefix of our AS, and probably receive just a default route from the upstream. There is no need to use the full table- yet. Both providers has approximately the same AS-PATH.
We’ll use a MikroTik CCR1036-8G-2S+ as the border router. Although it has SFP+ ports to allow 10 Gbps operation, at the moment the links will be negotiated using SFP modules at 1000 Mbps.
Main customer will be directly connected to this router using copper at 1 Gbps. They are using a MikroTik CRS326-24G-2S+ for their edge router, which will be enough for their 100 Mbps service. They provide us co-location too, so I installed the core router on their shelter, which is backed up by dual A/C systems and dual UPS-rectifier systems.
Once the upstream was go, I was able to see they were in fact sending us the full BGP table, which we don’t need yet, so a couple route filters were configured to use put only a default gateway on the main routing table. As the default route was configured as a static one, the route filter policy was as easy as discard all BGP inputs.
On this site there was also an Ubiquiti AirFiber 11X wireless link to reach Libertador San Martín town. Both radios were previously installed but not configured, so I connected to the radio and the site and configured it as Master. We traveled to the remote end, configured the radio as Slave and it worked just fine. Ubiquiti is getting up to date with their firmwares and UI, and it has became pretty straight forward to get a link working, even for someone with little or none networking skills.
¿Do you think this ease-of-use is making the job easier for us, or is it the start point of a madness of wireless spectrum usage?
From this node at Libertador, we installed two single-mode fiber lines, one to connect the town Hospital and another for the town University. MikroTik CRS326-24G-2S+ switches were installed at each site to be used as CPEs.
All monitoring, reporting and backup systems were previously configured at our NOC, so that was ll for us on the site.
Watching cars go by
I also assisted a brand new urban surveillance camera installation on the entrance of the Raúl Uranga – Carlos Sylvestre Begnis Subfluvial Tunnel. The objective was to read license plates on this strategic points, which is one of the few exits outside the Paraná river, and the one which has the most vehicle traffic.
Previously we had selected a Hikvision DS-2CD4A26FWD-IZHS8/P (yep, that’s the model name) camera which was already installed by Policía of Entre Ríos technician. This camera was specifically designed for licence-plate recognition (LPR). It supports OCR on hardware and works in very low light conditions, as low as 0.0027 lux.
The Dude network monitor is RouterOS package intended to manage a network environment. It automatically scan all devices within specified subnets, draw and layout a network maps, monitor services, and alert you in case of problems.
Previous versions of The Dude were developed as Windows x86 software, but later versions went through a full rebuild, and now it is distributed as a RouterOS package. This comes handy as the same RouterOS instance can be linked to the network, eliminating the the need for additional VPNs on servers or gateways. Instead, all tunneling can be done inside the CHR instance.
Once downloaded, upload it to the CHR instance via Winbox drag-and-drop, FTP client, or just download it from inside chr:
Reboot the CHR instance, and you will find the new Dude menu inside Winbox.
Head to Dude > Settings and tick Enabled to enable the server. A few folders will be created on the filesystem, and the server will be ready to accept connections on port 8291. The previous x86 based versions of Dude used port TCP/2210 or TCP/2211, but on this new integrated RouterOS package, all the management is handled on the same port as Winbox.
One you connect, the following window should appear by default. You can run a discover for multiple networks and let Dude map your network for you, but it will only disconver layer 3 adyancencies. In order to have complete control over the monitoring, I suggest to build your backbone manually and let the autodiscovery handle your management VLANs/VRFs.
MikroTik Cloud Hosted Router (CHR) is a RouterOS version intended to be used as a virtual machine instance.
It runs on x86-64-bit architecture and can be deployed on most hypervisors such as:
VMWare, ESXi, Player and Workstation
And others, like Xen, but I haven’t tested it yet
Some special requeriments apply depending on the subyacent hypervisor.
Network adapters must be vmxnet3 or E1000. Just use vmxnet3 to get the most. Disks must be IDE, VMware paravirtual SCSI, LSI Logic SAS or LSI Logic Parallel.
Network adapters must be Network adapter or Legacy Network adapter .Disks IDE or SCSI.
Virtio, E1000 or vmxnet3 NICs. IDE, Sata or Virtio disks.
Networking using E1000 or rtl8193, and disks with IDE, SATA, SCSI or SAS interfaces.
The CHR images have full RouterOS features enabled by default, but they use a different licensing model than other RouterOS versions.
p1 (perpetual-1), which allows CHR to run indefinitely. It comes with a limit of 1Gbps upload per interface. All the rest of the features provided by CHR are available without restrictions. It can be upgraded p1 to p10 or p-unlimited.
p10 (perpetual-10), which also allows CHR to run indefinitely, with a 10Gbps upload limit per interface. All features are available without restrictions. It can be upgraded to p-unlimited.
The p-unlimited (perpetual-unlimited) license level allows CHR to run indefinitely. It is the highest tier license and it has no enforced limitations.
Free licenses (yay!)
There are two ways to use and try CHR free of charge.
The free license level allows CHR to run indefinitely, with a limit of 1Mbps upload per interface. All the rest of the features have no restrictions. This level comes activated by default on all images.
Th p1/p10/pU licenses can be tested with a 60 days trial.
Cool. How can i try it?
The easiest way to spin up a working instance of CHR is using the OVA appliance provided by MikroTik.
Once downloaded, the OVA can be used to deploy a new instance. I’ll be using ESXi on this example. The OVA comes preconfigured with a single network adapter, but more interfaces can be added on a later stage.
After the VM boots, log in via CLI with the default credentials:
CHR comes with a free licence by default, limited to 1Mbps upload limit. This is handy for lab purposes, or low traffic scenarios like stand-alone DHCP servers.
A DHCP client is enabled by default on the single existing ether1 interface. Use any of the following methods to find out the adquired address.
/ip dhcp-client print
/ip address print
Let’s get a trial licence. You will need the credentials for your MikroTik account. If you don’t have a MikroTik account, get one here.
The CHR instance will also need Internet access, so be sure to connect the virtual NIC to a VM network where it can make its way to the outside.