MikroTik Cloud Hosted Router (CHR) is a RouterOS version intended to be used as a virtual machine instance.
It runs on x86-64-bit architecture and can be deployed on most hypervisors such as:
VMWare, ESXi, Player and Workstation
Microsoft Hyper-V
Oracle VirtualBox
KVM
And others, like Xen, but I haven’t tested it yet
Some special requeriments apply depending on the subyacent hypervisor.
ESXi
Network adapters must be vmxnet3 or E1000. Just use vmxnet3 to get the most. Disks must be IDE, VMware paravirtual SCSI, LSI Logic SAS or LSI Logic Parallel.
Hyper-V
Network adapters must be Network adapter or Legacy Network adapter .Disks IDE or SCSI.
Qemu/KVM
Virtio, E1000 or vmxnet3 NICs. IDE, Sata or Virtio disks.
VirtualBox
Networking using E1000 or rtl8193, and disks with IDE, SATA, SCSI or SAS interfaces.
Licensing
The CHR images have full RouterOS features enabled by default, but they use a different licensing model than other RouterOS versions.
Paid licenses
p1
p1 (perpetual-1), which allows CHR to run indefinitely. It comes with a limit of 1Gbps upload per interface. All the rest of the features provided by CHR are available without restrictions. It can be upgraded p1 to p10 or p-unlimited.
p10
p10 (perpetual-10), which also allows CHR to run indefinitely, with a 10Gbps upload limit per interface. All features are available without restrictions. It can be upgraded to p-unlimited.
p-unlimited (really?)
The p-unlimited (perpetual-unlimited) license level allows CHR to run indefinitely. It is the highest tier license and it has no enforced limitations.
Free licenses (yay!)
There are two ways to use and try CHR free of charge.
free
The free license level allows CHR to run indefinitely, with a limit of 1Mbps upload per interface. All the rest of the features have no restrictions. This level comes activated by default on all images.
60-day trial
Th p1/p10/pU licenses can be tested with a 60 days trial.
Cool. How can i try it?
The easiest way to spin up a working instance of CHR is using the OVA appliance provided by MikroTik.
Once downloaded, the OVA can be used to deploy a new instance. I’ll be using ESXi on this example. The OVA comes preconfigured with a single network adapter, but more interfaces can be added on a later stage.
Creating new VM from OVA templateSetting VM name, and uploading OVA fileI’ll use local storage for itThin provisioned disks, and a previously configured VM networkReview everything, and deploy
Initial Configuration
After the VM boots, log in via CLI with the default credentials:
Username: admin
Password: none
CHR comes with a free licence by default, limited to 1Mbps upload limit. This is handy for lab purposes, or low traffic scenarios like stand-alone DHCP servers.
A DHCP client is enabled by default on the single existing ether1 interface. Use any of the following methods to find out the adquired address.
/ip dhcp-client print
/ip address print
Let’s get a trial licence. You will need the credentials for your MikroTik account. If you don’t have a MikroTik account, get one here.
The CHR instance will also need Internet access, so be sure to connect the virtual NIC to a VM network where it can make its way to the outside.
Last post talked about an introduction to Machine Learning and how outcomes can be predicted using sklearn’s LogisticReggression.
Sometimes, the input data could require additional processing to prefer certain classes of information, that it considered more valuable or more representative to the outcome.
The LogisticRegression model allows to set the preference, or weight, at the time of being created, or later when being fitted.
The data used on the previous entry had four main classes: DRAFT, ACT, SLAST and FLAST. Once it is encoded and fitted, it can be selected by its index. I prefer to initialize some mnemonics selectors to ease the coding and make the entire code more human friendly.
Machine Learning is the science and art of programming computers so they can learn from data.
For example, your spam filter is a Machine Learning program that can learn to flag spam given examples of spam emails (flagged by users, detected by other methods) and examples of regular (non-spam, also called “ham”) emails.
The examples that the system uses to learn are called the training set. The new ingested data is called the test set. The performance measure of the prediction model is called accuracy and it’s the objetive of this project.
The tools
To tackle this, Python (version 3) will be used, among the package scikit-learn. You can find more info about this package on the official page.
In general, a learning problem considers a set of n samples of data and then tries to predict properties of unknown data. If each sample is more than a single number and, for instance, a multi-dimensional entry (aka multivariate data), it is said to have several attributes or features.
Supervised learning consists in learning the link between two datasets: the observed data X and an external variable y that we are trying to predict, usually called “target” or “labels”. Most often, y is a 1D array of length n_samples.
All supervised estimators in scikit-learn implement a fit(X, y) method to fit the model and a predict(X) method that, given unlabeled observations X, returns the predicted labels y.
If the prediction task is to classify the observations in a set of finite labels, in other words to “name” the objects observed, the task is said to be a classification task. On the other hand, if the goal is to predict a continuous target variable, it is said to be a regression task.
When doing classification in scikit-learn, y is a vector of integers or strings.
The Models
LinearRegression, in its simplest form, fits a linear model to the data set by adjusting a set of parameters in order to make the sum of the squared residuals of the model as small as possible.
LogisticRegression, which has a very counter-intuitive model, is a better choice when linear regression is not the right approach as it will give too much weight to data far from the decision frontier. A linear approach is to fit a sigmoid function or logistic function.
The Data
Data is presented on a CSV file. It has around 2500 rows, with 5 columns. Correct formatting and integrity of values cannot be assured, so additional processing will be needed. The sample file is like this.
The Code
We need three main libraries to start:
numpy, which basically is a N-dimensional array object. It also has tools for linear algebra, Fourier transforms and random numbers. It can be used as an efficient multi-dimensional container of generic data, where arbitrary data-types can be defined.
pandas, which provides high-performance and easy-to-use data structures and data analysis tools simple and efficient tools for data mining and data analysis
sklearn, the main machine learning library. It has capabilities for classification, regression, clustering, dimensionality reduction, model selection and data preprocessing.
A non essential, but useful library is matplotlib, to plot sets of data.
In order to provide data for sklearnmodels to work, it has to be encoded first. As the sample data has strings, or labels, a LabelEncoder is needed. Next, the prediction model is declared, where a LogisticRegression model is used.
The input data file path is also declared, in order to be loaded with pandas.read_csv().
import pandas as pd
import numpy as np
import matplotlib.pyplot as pyplot
from sklearn.preprocessing import LabelEncoder
from sklearn.linear_model import LogisticRegression
encoder = LabelEncoder()
model = LogisticRegression(
solver='lbfgs', multi_class='multinomial', max_iter=5000)
# Input dataset
file = "sample_data.csv"
The CSV file can be loaded into a pandas dataframe in a single line. The library also provides a convenient method to remove any rows with missing values.
# Use pandas to load csv. Pandas can eat mixed data with numbers and strings
data = pd.read_csv(file, header=0, error_bad_lines=False)
# Remove missing values
data = data.dropna()
print("Valid data items : %s" % len(data))
Once loaded, the data needs to be encoded in order to be fitted into the prediction model. This is handled by the previously declared LabelEncoder. Once encoded, the x and y datasets are selected. The pandas library provides a way to drop entire labels from a dataframe, which allows to easily select data.
encoded_data = data.apply(encoder.fit_transform)
x = encoded_data.drop(columns=['PREDICTION'])
y = encoded_data.drop(columns=['DRAFT', 'ACT', 'SLAST', 'FLAST'])
The main objective is to test against different lengths of train and test data, to find out how much data provides the best accuracy. The lengths of data will be incremented in steps of 100 to get a broad variety of results.
length = 100
scores = []
lenghts = []
while length < len(x):
x_train = x[:length]
y_train = y[:length]
x_test = x.sample(n=length)
y_test = y.sample(n=length)
print("Fitting model for %s training values" % length)
trained = model.fit(x_train, y_train.values.ravel())
score = model.score(x_test, y_test)
print("Score for %s training values is %0.6f" % (length, score))
length = length + 100
scores.append(score)
lenghts.append(length)
Zabbix is an open source monitoring tool for diverse IT components, including networks, servers, virtual machines (VMs) and cloud services. It provides monitoring metrics, among others network utilization, CPU load and disk space consumption. Data can be collected in a agent-less fashion using SNMP, ICMP, or with an multi-platform agent, available for most operating systems.
Even when it is considered one of the best NMS on the market, its reporting capabilities are very limited. For example, this is an availability report created with PRTG.
And this is a Zabbix Report. There is no graphs, no data tables, and it is difficult to establish a defined time span for the data collection.
My client required an executive report with the following information.
Host / Service Name
Minimum SLA for ICMP echo request monitoring
Achieved SLA for ICMP echo request monitoring
Memory usage graph, if host is being SNMP-monitored
Main network interface graph, if host is being SNMP-monitored
And storage usage graph, also if the host is being SNMP-monitored
Using the Zabbix API
To do call the API, we need to send HTTP POST requests to the api_jsonrpc.php file located in the frontend directory. For example, if the Zabbix frontend is installed under http://company.com/zabbix, the HTTP request to call the apiinfo.version method may look like this:
An object has to be created to initialize the client. I prefer to set url, username, and password dynamically, with data provided by the end user, so no credentials are stored here.
server = new $.jqzabbix({
url: url, // URL of Zabbix API
username: user, // Zabbix login user name
password: pass, // Zabbix login password
basicauth: false, // If you use basic authentication, set true for this option
busername: '', // User name for basic authentication
bpassword: '', // Password for basic authentication
timeout: 5000, // Request timeout (milli second)
limit: 1000, // Max data number for one request
});
As told before, the first step is to authenticate with the API, and save the authorization token. This is handled by the jqzabbix library by first making a request to get the API version, and then authenticating.
server.getApiVersion();
server.userLogin();
If the authentication procedure is completed properly, the API version and authentication ID are stored as properties of the server object. The userlogin() method allows to set callbacks for both success and error.
var success = function() { console.log('Success!'); }
var error = function() { console.error('Error!'); }
server.userLogin(null, success, error)
Once authenticated, the Zabbix API methods are called in the following fashion with the sendAjaxRequest method.
I set a global array hosts to store the hosts information. Another global array called SEARCH_GROUPS is used to define which hosts groups should considered on the API request. By setting the selectHosts parameter to true, the hosts on the host groups are retrieved too on the response.
On success, the result is stored on the hosts array, and the get_graphs function is called. If there is an error, the default error callback is fired.
Previously, user defined graphs were configured on Zabbix, to match the client requeriments of specific information. All names for the graphs that should be included on the report were terminated the ” – Report” suffix.
This function retrieves all those graphs, and by setting the selectHosts to true, the hosts linked to each graph are retrieved too.
On success, the result is stored on the graphs array, and the render function is called. If there is an error, the default error callback is fired.
By this time you should have noticed that the Zabbix API allows to retrieve values for the graphs, but no images. An additional PHP file will be stored with the HTML and JS files, as a helper to call the web interface by using php_curl.
You can get it on https://zabbix.org/wiki/Get_Graph_Image_PHP. I made a couple modifications to it in order to pass username and password on the URL query, with parameters for the graph ID, the timespan, and the image dimensions.
NetBox is an IP address management (IPAM) and data center infrastructure management (DCIM) tool. Initially conceived by the network engineering team at DigitalOcean, NetBox was developed specifically to address the needs of network and infrastructure engineers.
When I started using NetBox on my daily job, I planned to use it as a replacement for all the spreadsheets I had for switch configurations, IP address management, secrets, and VLAN assignments. NetBox can handle all of this and more, but the interface didn’t suit my needs.
NetBox is built using the Python Django framework, which I have used for another projects. I used Visual Studio Code to clone the repository and debug, as it has native support for the Django template language.
I keep a copy of the repository on my local machine for ease of modifications. Prior, I have set DEBUG=TRUE on netbox/configuration.py, and allowed localhostand my local network to access the development server. Also, I set the correct settings to connect to the existing postgresql database.
Connecting the existing DB to my local development server
This environment works for test purposes, but the best you can do is to set up separated development and production environments, and commit your changes to production once everything is tested.
Using VSCode to debug Django
The URL definition for the single device view is around line #147 of the netbox/dcim/urls.py file, and it looks like this.
Heading to the DeviceView view, I put a breakpoint on the interfaces QuerySet of the view definition, and launched the debugger. The default location is at http://localhost:8000.
Setting up the debuggerBreakpoints
I headed to http://localhost:8000/dcim/devices/570/, where I had defined a switch with several VLANs, to hit the breakpoint and find out if the QuerySet had information about the VLANs, or if they were queried in a per-interface basis, on the interface view.
QuerySet returns this
Lucky me, the QuerySet recovered all the information I needed, and it is passed to the template via a render() call.
All the information I want is rendered on this table. This is the power of the Django framework. I added line #513 as an additional header for the VLANs column.
This table has a for loop which iterates for each interface of the device, so I edited the included template file at dcim/inc/interface.html.
Both tagged and untagged VLANs groups have a bolded title, and the VID and VLAN name is shown after it. I used the dictsort filter, which is part of the Django framework, to sort all the VLANs by their VID.
dcim/inc/interface.html
The final result looks like the following image, and it allows to keep track of all the VLANs on all ports, at first sight. This is easier and more user friendly than getting that information interface per interface, or making a new custom view.
NetBox is an IP address management (IPAM) and data center infrastructure management (DCIM) tool. Initially conceived by the network engineering team at DigitalOcean, NetBox was developed specifically to address the needs of network and infrastructure engineers.
$ git clone -b master https://github.com/ninech/netbox-docker.git
$ cd netbox-docker
Once cloned, I used docker-compose to pull the images
$ docker-compose pull
And then I started the stack with
$ docker-compose up -d
The service will be up and running after a few minutes. Once ready, you need to find where to connect to with
$ docker-compose port nginx 8080
Or use this snippet
$ echo "http://$(docker-compose port nginx 8080)/"
Here I use Portainer as a gui to manage Docker, and Traefik as a reverse proxy to enable FQDN access to the services behind. I added an entry on my DNS to route netbox.arturo.local to the Docker IP address, on the exposed port for Nginx.
Andrew Foster at Topland Communications reached me via Upwork looking to customize and fine tune a existing Spiceworks installation.
After a quick inspection, I decided to tackle the project by compacting the DB in first place. Spiceworks keeps a lot of logs regarding the system activity, which are located on C:\Program Files\Spiceworks\Log. In order to clean them, the first step is to stop Spiceworks service.
Logs are stored in two main locations:
C:\Program Files\Spiceworks\Log, for the Spiceworks service
C:\Program Files\Spiceworks\httpd\log\, where the Apache server keeps them
Once the logs are cleaned, I compacted the DB to increase the performance, and I started the service again.
Ticket rules were configured to auto assign support tickets, thus saving time to the support operators.
And the user portal was customized to match the company colors and logo.
A security company contacted me thorough Upwork, searching for support on a brand new installation of an Axis Camera Station System on a educational institution. This company, Coyote Cabling from New Mexico, US, was on charge of a 52 camera installation, with an option to add 32 existing cameras on a later stage.
After a research, they decided to use Axis S1148 servers, which really are re-branded Dell servers. The S1148 comes with a ready to use Windows Server 2012 OS, and with the Axis Camera Station preinstalled. This vendor supported hardware allowed to reduce licensing costs, because they are included on the server price, and avoid any incompatibilities
Client wanted to set up a ISPConfig 3 Control Panel onto a Digital Ocean droplet.
Digital Ocean works best for this kind of services, because they provision the public addresses directly on the server. The configuration is easier to build and mantain, thanks to the Digital Ocean integrated firewall.
ISPConfig allows to manage servers and hosting plans from a friendly GUI.
After a couple of successful jobs with my client Visual Link Internet LLC, they reached me to set up a service similar to dyndns.net. I had already developed another value added services for their customers, like web filtering and firewalling, so I found this project very interesting and fun to do.
Cool, but what is DNS ?
DNS stands for Domain Name Systems. Yep, domains like google.com.
It is based on a distributed database that takes some time to update globally. When DNS was first introduced, the database was small and could be easily maintained by hand. As the system grew this task became difficult for any one site to handle, and a new management structure was introduced to spread out the updates among many domain name registrars.
Due to the distributed nature of the DNS systems and its registrars, updates to the global DNS system may take hours to distribute. Thus DNS is only suitable for services that do not change their IP address very often, but not for servers being run with dynamic addresses, which are likely to change their IP address over very short periods of time.
Ok, but my ISP gives dynamic addresses, and I want to access services on my network. What can i do?
Dynamic DNS is a system that addresses the problem of rapid updates. The term is used in two ways, which, while technically similar, have very different purposes and user populations. The first is “standards-based DNS updates”, which uses an extension of the DNS protocol to ask for an update. The second is usually a web-based protocol, normally a single HTTP fetch with username and password which then updates some DNS records (by some unspecified method).
Many providers offer commercial or free Dynamic DNS service for this scenario. The automatic reconfiguration is generally implemented in the user’s router or computer, which runs software to update the DDNS service. The communication between the user’s equipment and the provider is not standardized, although a few standard web-based methods of updating have emerged over time.
Yeah, but those free services are now paid, and some have even disappeared
I know, I know. But this service can be built in-house. Using open source software, there are no fees, and the company domain name can be used to keep things professional.
This is what my client wanted, so I deployed a solution on a that allowed to offer added value services to customers, and provide easy remote access. Using a open source solution based on PHP (https://github.com/nicokaiser/Dyndns) and some custom Bash scripts I was able to deliver a stable system in a short amount of time.
The main techonologies I used are Apache 2 and PHP 7 for the HTTP requests and update system, and BIND9 for the DNS service.
The solution used the standard URL schema of DynDNS, so it is compatible with any device with support for it. Also, because most CPEs of the client’s network were MikroTik based, I also wrote a RouterOS script to call the update.
