Tag: python

Categories
Ansible Netbox Projects Python

j2ipaddr

Jinja2 filters for IP addresses, the easy way

Why

On networking and network automation, we need to extract info about IP addresses as a combination of two values:

  • a host address
  • a subnet mask

For 10.10.10.5/24, the host address is 10.10.10.5 and the subnet mask is 255.255.255.0, and its prefix length is 24.

There is additional information we can infer from this single item, as its network address, broadcast address.

Useful data for network engineers are wildcards or hostmasks, network size, class, type, and so on.

Jinja2 provides several integrated filters to work with, however it can be complicated to use complex data types.

Ansible provides a way to work this on its ansible.utils.ipaddr collection.

However, probably you won’t need the entire Ansible package just to be able to use it.

This package intends to provide a set of filters and handler to the Python 3 netaddr module, on a way that is hopefully easy and lightweight to use.

What

Included filters are the following:

ip_address(addr)

Returns an IP address for a combination of IP address and subnet mask

ip_address('10.10.10.5/24')
> 10.10.10.5
{{ '10.10.10.5/24 | ip_address }}
> 10.10.10.5

ip_prefixlen(addr)

Returns a prefix length for a combination of IP address and subnet mask

ip_prefixlen('10.10.10.5/24')
> 24
{{ '10.10.10.5/24 | ip_prefixlen }}
> 24

ip_netmask(addr)

Returns a subnet mask for a combination of IP address and subnet mask

ip_netmask('10.10.10.5/24')
> 255.255.255.0
{{ '10.10.10.5/24 | ip_netmask }}
> 255.255.255.0

ip_hostmask(addr)

Returns a wilcard or hostmask for a combination of IP address and subnet mask

ip_hostmask('10.10.10.5/24')
> 0.0.0.255
{{ '10.10.10.5/24 | ip_hostmask }}
> 0.0.0.255

ip_wildcard(addr)

Alias for ip_hostmask(addr)

ip_wildcard('10.10.10.5/24')
> 0.0.0.255
{{ '10.10.10.5/24 | ip_wildcard }}
> 0.0.0.255

ip_network(addr)

Returns a network address for a combination of IP address and subnet mask

ip_network('10.10.10.5/24')
> 10.10.10.0
{{ '10.10.10.5/24 | ip_network_hosts_size }}
> 10.10.10.0

ip_broadcast(addr)

Returns a broadcast address for a combination of IP address and subnet mask

ip_broadcast('10.10.10.5/24')
> 10.10.10.255
{{ '10.10.10.5/24 | ip_broadcast }}
> 10.10.10.255

ip_network_hosts_size(addr)

Returns the size of the subnet for a combination of IP address and subnet mask

ip_network_hosts_size('10.10.10.5/24')
> 255
{{ '10.10.10.5/24 | ip_network_hosts_size }}
> 255

ip_network_first(addr)

Returns the first usable address in network address for a combination of IP address and subnet mask

ip_network('10.10.10.5/24')
> 10.10.10.1
{{ '10.10.10.5/24 | ip_network_hosts_size }}
> 10.10.10.1

ip_network_last(addr)

Returns the last usable address in network address for a combination of IP address and subnet mask

ip_network('10.10.10.5/24')
> 10.10.10.254
{{ '10.10.10.5/24 | ip_network_hosts_size }}
> 10.10.10.254

How

Simply install with pip.

$ pip install j2ipaddr

To insert the filters on your Jinja2 processor, simply use the following syntax. The filter name can be changed by adjusting the dict key name.

import jinja2
import j2ipaddr.filters
jinja2.filters.FILTERS['ip_prefixlen'] = filters.ip_prefixlen

Or, probably an easier way, use the following one-liner to load all the filters into your Jinja2 filters

import jinja2
import j2ipaddr.filters
jinja2.filters.FILTERS = {**jinja2.filters.FILTERS, **filters.load_all()}

On your templates, you can do this as an example:

Variables

host:
  interfaces:
    Te1/0/1:
      ipv4_addresses:
        - 10.10.10.5/24

Template

router ospf 10
  network {{host.interfaces.Te1/0/1.ipv4_addresses[0] | ip_network }} {{host.interfaces.Te1/0/1.ipv4_addresses[0] | ip_wildcard  }} area 0.0.0.0

The output would looks like this:

router ospf 10
  network 10.0.0.0 0.0.0.255 area 0.0.0.0

Where

You can find this project on

Categories
Networking Projects Ubiquiti

UFiber v4 Python Client

Yeah, I finally remembered to make a post about this. I know it will like as a copy-paste of the previous one, because, in fact it is.

Ok, if you have been following the series, you should already know that I equally love and hate UFiber OLTs. They are affordable, deliver a lot of bang for the buck, and have an awful GUI.

Well, the GUI is lovely on v4.

Python in the middle

I wrote a quick and dirty client which acts as a sort of middleware between the HTTP inteface of the OLT and you.

It allows to provision non existing ONUs, GPON profiles, WiFi profiles, retrieve active ONU status and general configuration.

Take a look to it on https://github.com/baldoarturo/ufiber-client-4, and feel free to contribute if you want to.

How to help

It would be awesome to have docs šŸ˜€

Are you a pydoc master? Let’a add docstrings.

Do you have an OLT for me to test? Ping me and we can set up a VPN.

olt.py

This is the core of the project. It uses the OLTClient class to provide a middleware between you and the HTTP interface of the OLT.

Initialize a new OLTClient instance with:

client = OLTClient(host='192.168.1.1', username='ubnt', password='ubnt', debug_level=logging.DEBUG)

Required params are only host, and credentials.

The initialization will handle the login for you, altough you can call the login() method manually.

If the OLT is network reachable, and you have provided the right credentials, and the OLT GUI is alive and well, you should be ready to start.

What changes on v4

Well, UBNT got rid of the GPON profiles. šŸ™

This software is intented to give you an alternative by keeping profiles as JSON in the ./profiles folder.

You can copy theĀ template.jsonĀ file and make your way using it as a starting point. It should be self descriptive.

There is anĀ schema.jsonĀ which validates your profile before pushing changes into the OLT.

Categories
Networking Projects Ubiquiti

UFiber Python Client

Ok, if you have been following the series, you should already know that I equally love and hate UFiber OLTs. They are affordable, deliver a lot of bang for the buck, and have an awful GUI.

Please, be aware that this can change for better or worse in the future, and at the time I’m writing this the latest firmware is v3.1.3. I trust in you UBNT, hope you can sort out this and give us a better product. I’ll keep my fingers crossed.

Python in the middle

I wrote a quick and dirty client which acts as a sort of middleware between the HTTP inteface of the OLT and you.

It allows to provision non existing ONUs, GPON profiles, WiFi profiles, retrieve active ONU status and general configuration.

Take a look to it on https://github.com/baldoarturo/ufiber-client, and feel free to contribute if you want to.

Edited on Aug 15 2020: I did the same for firmware version 4, which is cleaner and fixes a lot of bugs. Stay tuned!

ufiber-client

This is a quick dirty project built to provide a quick dirty client for Ubiquiti UFiber OLTs, using firmware version 3.x

There is also a CLI attempt, but I couldn’t find any ready to use packages to build a decent CLI.

More info about what am I doing this is on the following entries:

olt.py

This is the core of the project. It uses the OLTCLient class to provide a middleware between you and the HTTP interface of the olt.

Initialize a new OLTClient instance with:

client = olt.OLTClient(host, username, password)

The initialization will handle the login for you, altough you can call the login() method manually.

If the OLT is network reacheable, and you have provided the right credentials, and the OLT WEB GUI is alive and well, you should be ready to start.

You can also connect using cli.py:

$ /cli.py
UFiber Client for fw version 3.1.3
UFiber> help

Documented commands (type help <topic>):
========================================
connect  help  onu  quit  show

UFiber> connect 10.20.0.101
Username:admin
Password:
Logging to 10.20.0.101 ...
Connection OK
UFiber>
Categories
Networking Projects Ubiquiti

UFiber OLT API

In a previous post we took a quick look to the Ubiquiti UFiber OLT. As always, UBNT tries to offer a non expensive solution to provide last-mile conectivity for end users. I am using non-expensive because UBNT gear is not cheap. Yeah, it can be affordable, but you only get what you pay for.

We saw that the command line is very limited, even when the software is a fork of Vyatta. There is no way to get ONUs provisioned from the command line, so forgot about Ansible (we love Ansible), netmiko, and other SSH clients tools to ease your life.

UBNT wants you to use the web GUI, period. They offer a dockerized management system called UNMS, which really comes handy after you have provisioned your customers.

Both you and me, as network operators, know that provisioning customers is one of the more boring tasks, but is is still a critical one. Fast and precise provisioning translates in more customers, more stability, faster troubleshooting, and peace of mind.

Yeah, SONAR exists, but not all operators can work with their pricing and technology supports. And don’t even think to integrate billing if you are using electronic invoicing with AFIP in Argentina.

If you are still here, don’t give up. If there is a will, there is a way.

Under the hood

The OLT has a web GUI served by HTTPS, with a self-signed certificate, on port 443. There is no easy way to use a proper certificate here, but well, it’s something.

If you are not authenticated, this is what waits for you in the URL root.

Looks like a login screen

I want to know if this is a standard HTML form. And indeed it is.

<form id="LoginForm" method="post" class="ui-form">
    <input id="Username" name="username" class="text-input" type="text" placeholder="Username" autocapitalize="off" autocorrection="off">
    <input id="Password" name="password" class="text-input" type="password" placeholder="Password">
    <input id="LoginButton" class="submit-input ui-button ui-widget ui-state-default ui-corner-all" type="submit" value="Login" role="button" aria-disabled="false">
</form>

What happens when we log in? I’m using Chrome version 81 and something, let’s open devtools to see the network activity.

General

Request URL:
https://x.x.x.x/
Request Method:
POST
Status Code:
303 See Other
Remote Address:
x.x.x.x:443

Request Headers

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: es,en;q=0.9
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 32
Content-Type: application/x-www-form-urlencoded
Cookie: PHPSESSID=5da99950e9f74ad8b727f219c9e41d76; X-CSRF-TOKEN=9f0c78e2ea8994b39834e0241466c21b68a28df59bf98364ece91dcd183bdab5; beaker.session.id=29fdb5243db8446f81f75587c9c2a722
DNT: 1
Host: x.x.x.x
Origin: https://x.x.x.x
Referer: https://x.x.x.x/
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36

Response

Content-Length: 0
Content-Type: text/html; charset=UTF-8
Date: Fri, 02 Jan 2015 08:54:30 GMT
Location: https://x.x.x.x/
Server: Server
Set-Cookie: PHPSESSID=a24b5cbbd6874a1eb09c2d086a93efc6
Set-Cookie: X-CSRF-TOKEN=6f13035a0b7aa4b375e6798c7c60f12e805ecea8c74a3306da81c710e6a3701b
Set-Cookie: beaker.session.id=a24b5cbbd6874a1eb09c2d086a93efc6; httponly; Path=/; secure

Form Data

username: ubnt
password: ubnt

So, this is a standard POST. And we got a cookie.

This can be translated to Python by using the request module.

host = 'olt.ubnt'
url = 'https://{host}'.format(host=host)

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'

HEADER_FORM_URLENCODED = {
    'Content-Type': 'application/x-www-form-urlencoded',
    'User-Agent': USER_AGENT,
}

form_data = {
    'username': username,
    'password': password,
}
response = requests.post(
    verify=False,
    url=url,
    headers=HEADER_FORM_URLENCODED,
    data=form_data
)

Good enough for me. Of course this should be into a try/catch structure, but ymmv.

Shut up and take my ONUs

So, we are logged in. What about ONU configuration? The GUI allows to update firware, but this is done automatically starting from software version 3.1.3. We can also set many parameters of the ONU configuration, and provision them via OMCI, but the ONU should already exist in the ONU list.

Configuring existing ONUs

There is no way to add non-existing ONUs to the configuration, which makes pre-provisioning impossible.

What happens when we click on save?

A POST request is made, passing an interesting payload to the /api/edge/batch.json endpoint. Let’s see how does it looks like.

This POST puts a payload with all the ONU configuration, and a bit more. It uses a JSON structure which looks like this.

{
    "SET": {
        "onu-list": {
            "UBNTffffffff": {
                "disable": "false",
                "profile": "profile-2",
                "name": "ARTURO TEST",
                "wifi": {
                    "provisioned": false,
                    "enabled": true,
                    "ssid": "UBNT-ONU",
                    "hide-ssid": false,
                    "auth-mode": "wpa2psk",
                    "wpapsk": "",
                    "channel": "auto",
                    "channel-width": "20/40",
                    "tx-power": "100"
                },
                "pppoe-mode": "auto",
                "pppoe-user": "ARTURO",
                "pppoe-password": "ARTURO",
                "wan-address": "null",
                "port-forwards": []
            }
        }
    }
}

Of course this is a fake ONU with a UBNTffffffff serial number. Yeah, we can fool the GUI and send whatever values we want.

This comes real handy because you can pre-provision ONUs before they show up in the PON port.

As an ISP, this means a time saver, because you can deliver ONUs ready to plug and play, and the OLT will hand out all the configuration without further intervention.

The web GUI is handy (altough not so stable), but it really doesn’t makes sense to have to manually provision each ONU when they are connected in the PON port.
Technicians have to spend to a lot time on customers houses waiting for the NOC to configure each new customer. This is no-bueno in pandemic times. You want to install as many new customers as possible, as fast as possible, and staying in foreign homes as less as possible.

It seems it’s time for me to code something.

If you are reading this, Robert Pera, please make me a CLI.

Categories
Ansible Juniper Networking

Ansible and Juniper Junos – Interfaces

Previously we had made our first incursions connecting an Ansible control node with a Juniper router. In this post, we’ll see how to retrieve information about the router interfaces, both layer 2 and layer 3, and configure new interfaces.

The official Ansible modules reference will be your main guide for any additional information.
https://docs.ansible.com/ansible/latest/modules/list_of_network_modules.html#junos

If you are interested on this subject, but don’t have access to physical gear, most of it should work on virtual appliances like vMX, vQFX, which you can operate on a stand-alone mode or on a network environment like GNS3 or EVE-NG.

Juniper vLabs will also give you an introduction to the Juniper platform.
https://jlabs.juniper.net/vlabs/portal/index.page

Layer 2 Interfaces

A basic layer 2 interface configuration in Junos looks like this:

ge-0/0/1 {
     description "L2 interface";
     speed 1g;
     unit 0 {
         family ethernet-switching {
             interface-mode access;
             vlan {
                 members vlan30;
             }
         }
     }
 }

This configuration can be written as an Ansible playbook like this:

- name: "Replace provided configuration with device configuration"
  junos_l2_interfaces:
    config:
      - name: ge-0/0/1
        access:
          vlan: v30
    state: merged

Currently, I do not have any EX series or QFX series to decomission and run tests against it, so stay tuned for any updates on this.

The official module documentation is on https://docs.ansible.com/ansible/latest/modules/junos_l2_interfaces_module.html.

Layer 3 Interfaces

A basic layer 3 interface configuration in Junos looks like this:

ge-0/0/1 {
     unit 0 {
         family inet {
             address 192.168.1.10/24;
         }
     }
 }

This configuration can be written as an Ansible playbook like the following, using the same format as the last post.

---
- hosts: all
  gather_facts: no

  tasks:

  - name: Config ge-0/0/1
    junos_l3_interfaces:
      config:
        - name: ge-0/0/1
          ipv4:
            - address: 192.168.1.10/24
      state: merged

Let’s run it and check the result.

$ ansible-playbook juniper.yml -i juniper-hosts.yml

PLAY [all] ********************************************************************************************************************

TASK [Config ge-0/0/1] ********************************************************************************************************
[WARNING]: Platform linux on host 192.168.15.220 is using the discovered Python interpreter at /usr/bin/python, but future
installation of another Python interpreter could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.

changed: [192.168.15.220]

PLAY RECAP ********************************************************************************************************************
192.168.15.220             : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Did you noticed the changed output?
What about the configuration on the router now?

admin> show configuration interfaces
ge-0/0/1 {
    unit 0 {
        family inet {
            address 192.168.1.10/24;
        }
    }
}
fxp0 {
    unit 0 {
        family inet {
            address 192.168.15.220/24;
        }
        family inet6;
    }
}

That’s awesome! We just configured and IP address on ge-0/0/1.

How does Ansible knows what to replace, what to override, and what to delete?

If you take a closer look to the playbook, you will see a line with state: merged. This is a module parameter that specifies the state of the router configuration after the module finishes its job.

The possible values are:

  • merged
  • replaced
  • overriden
  • deleted

In fact, the module matches whatever configuration you build on its parameters, applies a configuration action, and commits the result.

The official module documentation is on https://docs.ansible.com/ansible/latest/modules/junos_l3_interfaces_module.html.

Categories
Ansible Juniper Networking

Ansible and Juniper Junos – First Steps

On the previous post, I proposed a quick and dirty method to provision an Ansible control node, using Vagrant and VirtualBox. But, if you really want to spin the volume up to 11, the best is to work in a dedicated Linux machine.

On this lab, I will be using Linux Mint, which is a Ubuntu derivate, but most examples will work on any Linux distribution.

First, install Ansible if already don’t have it. There are two easy methods.

  • Using pip to install the ansible Python module. You’ll need a working installation of Python2 (altough the latests releases of Ansible works with Python 3 too). Also, pip has to be installed and on path.

    $ pip install ansible
  • Using apt by adding the Ansible ppa repository and offload all the work to the system package manager.

    $ sudo apt-add-repository ppa:ansible/ansible
    $ sudo apt-get update
    $ sudo apt install ansible

I preffer the last one and leave apt handle all the job, because the ppa repository is usually up-to-date with the latest release.

Once installed, check your setup with ansible --version or ansible localhost -m setup.

Ansible manages Junos using NETFCONF over SSH. In order to be able to connect to Junos via Ansible, both SSH and NETCONF services has to be enabled on the remote host.

SSH can be used for sending raw commands using the junos_command module, but NETCONF is definetly more versatile and supports the whose set of Ansible modules, which you can see here.

To use NETCONF, you will need an additional Python module in your Ansible control node.

$ pip install ncclient

And to enable SSH and NETCONF in your Juniper host , do the following. Probably you already have SSH enabled on your router so you can ignore that setting.

admin> edit  
Entering configuration mode
[edit]
admin# set system services ssh 
admin# set system services netconf ssh 
[edit]
admin# commit 
commit complete

Building the Ansible Inventory

Ansible inventory files can use many formats, depending on the plugins you have. The two main formats are INI files, and YAML.

INI-style Inventory

[junos]
192.168.15.194

[junos:vars]
ansible_connection=netconf
ansible_network_os=junos
ansible_user=admin
ansible_password=Password$1

YAML Inventory

all:
  hosts:
    "192.168.15.194":
  vars:
    ansible_connection: netconf
    ansible_network_os: junos
    ansible_user: admin
    ansible_password: Password$1

Both files represent the same set of information on a different style. Personally, I preffer the YAML notation because it is more human-friendly and readeable, and it allows you to learn YAML which is used on many other automation and orchestration tools.

What does this means

all:
  hosts:
    "192.168.15.194":

all: stands for all hosts and its always required. All hosts on an Ansible inventory belongs to at least two groups, all and ungrouped. All groups of course, all hosts, and ungrouped contains all hosts which don’t belong to another specific group besides all.

hosts: specifies the start position for the hosts. Hosts groups can be then written like:

mendoza:
   hosts:
     "192.168.15.194":
     core.thisnetwork.net:

Hosts groups can share a set of variables, which provide information for the Ansible modules. For this example, as the vars section is a direct children of the allgroup, these variables will apply to all hosts in the inventory.

vars:
     ansible_connection: netconf
     ansible_network_os: junos
     ansible_user: admin
     ansible_password: Password$1

The Junos modules for Ansible use the netconf connection, which uses SSH and NETCONF, so ensure to allow TCP/22 and TCP/830 on your rules.

Although a fixed inventory file with manually added hosts can be enough for some users, on future entries we’ll set up a dynamic inventory calling a source of truth like Netbox, or a monitoring system like Zabbix.

Testing our setup

The real fun with Ansible is on ansible-playbook, but first let’s fire up an Ansible module to warm up your Ansible-fu.

$ ansible -m junos_facts -i juniper-hosts.yml all
  • -m junos_facts instructs Ansible to use the junos_facts module
  • juniper-hosts.yml is our previously configured inventory file
  • all tells Ansible the group of hosts inside the inventory to use

If everythig works allright, this will output a long JSON with a bunch of information about your device.

192.168.15.195 | SUCCESS => {
    "ansible_facts": {
        "ansible_net_api": "netconf", 
        "ansible_net_filesystems": [
            "/dev/md0.uzip", 
            "devfs", 
            "/dev/gpt/junos", 
            "devfs", 
            "/dev/md1.uzip", 
            "/dev/md2.uzip", 
            "tmpfs", 
            "procfs", 
            "/dev/ada1s1e", 
            "/dev/ada1s1f", 
            "/dev/md3.uzip", 
            "/dev/md4.uzip", 
            "/dev/md5.uzip", 
            "/dev/md6.uzip", 
            "/dev/md7.uzip", 
            "/dev/md8.uzip", 
            "/dev/md9.uzip", 
            "/dev/md10.uzip", 
            "/dev/md11.uzip", 
            "/packages/mnt/junos-libs-compat32/usr/lib32", 
            "/packages/mnt/os-libs-compat32-10/usr/lib32", 
            "/packages/mnt/os-compat32/libexec", 
            "/var/jails/rest-api", 
            "/dev/md12", 
            "/dev/md13.uzip", 
            "/dev/md14.uzip", 
            "/dev/md15.uzip", 
            "/dev/md16.uzip", 
            "/dev/md17.uzip", 
            "/dev/md18.uzip", 
            "/dev/md19.uzip", 
            "/dev/md20.uzip", 
            "/dev/md21.uzip", 
            "/dev/md22.uzip", 
            "/dev/md23.uzip", 
            "/dev/md24.uzip", 
            "/dev/md25.uzip", 
            "/dev/md26.uzip", 
            "/dev/md27.uzip", 
            "/dev/md28.uzip", 
            "tmpfs", 
            "junosprocfs"
        ], 
        "ansible_net_gather_network_resources": [], 
        "ansible_net_gather_subset": [
            "hardware", 
            "default", 
            "interfaces"
        ], 
        "ansible_net_has_2RE": false, 
        "ansible_net_hostname": "None", 
        "ansible_net_interfaces": {
            ".local.": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Loopback"
            }, 
            "cbp0": {
                "admin-status": "up", 
                "macaddress": "00:05:86:cc:c1:11", 
                "mtu": "9192", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Ethernet"
            }, 
            "demux0": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "9192", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Software-Pseudo"
            }, 
            "dsc": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Software-Pseudo"
            }, 
            "em1": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:cf:9b:01", 
                "mtu": "1514", 
                "oper-status": "up", 
                "speed": "1000mbps", 
                "type": "Ethernet"
            }, 
            "esi": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Software-Pseudo"
            }, 
            "fxp0": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:cf:9b:00", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "Unspecified", 
                "type": "Ethernet"
            }, 
            "ge-0/0/0": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:f6:47:02", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "1000mbps", 
                "type": null
            }, 
            "ge-0/0/1": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:f6:47:03", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "1000mbps", 
                "type": null
            }, 
            "ge-0/0/2": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:f6:47:04", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "1000mbps", 
                "type": null
            }, 
            "ge-0/0/3": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:f6:47:05", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "1000mbps", 
                "type": null
            }, 
            "ge-0/0/4": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:f6:47:06", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "1000mbps", 
                "type": null
            }, 
            "ge-0/0/5": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:f6:47:07", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "1000mbps", 
                "type": null
            }, 
            "ge-0/0/6": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:f6:47:08", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "1000mbps", 
                "type": null
            }, 
            "ge-0/0/7": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:f6:47:09", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "1000mbps", 
                "type": null
            }, 
            "ge-0/0/8": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:f6:47:0a", 
                "mtu": "1514", 
                "oper-status": "down", 
                "speed": "1000mbps", 
                "type": null
            }, 
            "ge-0/0/9": {
                "admin-status": "up", 
                "macaddress": "0c:b8:15:f6:47:0b", 
                "mtu": "1514", 
                "oper-status": "up", 
                "speed": "1000mbps", 
                "type": null
            }, 
            "gre": {
                "admin-status": "up", 
                "macaddress": null, 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "GRE"
            }, 
            "ipip": {
                "admin-status": "up", 
                "macaddress": null, 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "IPIP"
            }, 
            "irb": {
                "admin-status": "up", 
                "macaddress": "00:05:86:cc:c8:f0", 
                "mtu": "1514", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Ethernet"
            }, 
            "jsrv": {
                "admin-status": "up", 
                "macaddress": "00:05:86:cc:c8:c0", 
                "mtu": "1514", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Ethernet"
            }, 
            "lc-0/0/0": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "0", 
                "oper-status": "up", 
                "speed": "800mbps", 
                "type": "Unspecified"
            }, 
            "lo0": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Loopback"
            }, 
            "lsi": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Software-Pseudo"
            }, 
            "mtun": {
                "admin-status": "up", 
                "macaddress": null, 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Multicast-GRE"
            }, 
            "pfe-0/0/0": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "0", 
                "oper-status": "up", 
                "speed": "800mbps", 
                "type": "Unspecified"
            }, 
            "pfh-0/0/0": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "0", 
                "oper-status": "up", 
                "speed": "800mbps", 
                "type": "Unspecified"
            }, 
            "pimd": {
                "admin-status": "up", 
                "macaddress": null, 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "PIMD"
            }, 
            "pime": {
                "admin-status": "up", 
                "macaddress": null, 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "PIME"
            }, 
            "pip0": {
                "admin-status": "up", 
                "macaddress": "00:05:86:cc:c8:b0", 
                "mtu": "9192", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "Ethernet"
            }, 
            "pp0": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "1532", 
                "oper-status": "up", 
                "speed": "Unspecified", 
                "type": "PPPoE"
            }, 
            "rbeb": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Software-Pseudo"
            }, 
            "tap": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Software-Pseudo"
            }, 
            "vtep": {
                "admin-status": "up", 
                "macaddress": "Unspecified", 
                "mtu": "Unlimited", 
                "oper-status": "up", 
                "speed": "Unlimited", 
                "type": "Software-Pseudo"
            }
        }, 
        "ansible_net_memfree_mb": 179384, 
        "ansible_net_memtotal_mb": 2058336, 
        "ansible_net_model": "vmx", 
        "ansible_net_modules": [
            {
                "name": "Midplane"
            }, 
            {
                "description": "RE-VMX", 
                "name": "Routing Engine 0"
            }, 
            {
                "description": "VMX SCB", 
                "name": "CB 0"
            }, 
            {
                "description": "VMX SCB", 
                "name": "CB 1"
            }, 
            {
                "chassis_sub_module": null, 
                "description": "Virtual FPC", 
                "name": "FPC 0"
            }
        ], 
        "ansible_net_python_version": "2.7.15+", 
        "ansible_net_routing_engines": {
            "0": {
                "cpu_background": "0", 
                "cpu_background1": "0", 
                "cpu_background2": "0", 
                "cpu_background3": "0", 
                "cpu_idle": "61", 
                "cpu_idle1": "94", 
                "cpu_idle2": "95", 
                "cpu_idle3": "95", 
                "cpu_interrupt": "2", 
                "cpu_interrupt1": "1", 
                "cpu_interrupt2": "1", 
                "cpu_interrupt3": "1", 
                "cpu_system": "27", 
                "cpu_system1": "4", 
                "cpu_system2": "3", 
                "cpu_system3": "3", 
                "cpu_user": "10", 
                "cpu_user1": "1", 
                "cpu_user2": "1", 
                "cpu_user3": "1", 
                "last_reboot_reason": "Router rebooted after a normal shutdown.", 
                "load_average_fifteen": "0.62", 
                "load_average_five": "0.73", 
                "load_average_one": "0.75", 
                "mastership_priority": "master (default)", 
                "mastership_state": "master", 
                "memory_buffer_utilization": "13", 
                "memory_dram_size": "2010 MB", 
                "memory_installed_size": "(2048 MB installed)", 
                "model": "RE-VMX", 
                "slot": "0", 
                "start_time": "2019-11-26 12:06:10 UTC", 
                "status": "OK", 
                "up_time": "11 hours, 46 minutes, 19 seconds"
            }
        }, 
        "ansible_net_serialnum": "VM5DDBEA932E", 
        "ansible_net_system": "junos", 
        "ansible_net_version": "17.1R1.8", 
        "ansible_network_resources": {}, 
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false
}

If you take a closer look, you’ll find out that I am using a vMX appliance, can you spot where is that information?

What if we could use this and more information to gather more useful data about our router, and execute actions of provision new configuration depending on the gathered data?

If Ansible modules are the tools in your workshop, playbooks are your instruction manuals, and your inventory of hosts are your raw material.

The First Playbook

Playbooks are expressed in YAML and are composed of one or more ā€˜playsā€™ in a list. The goal of a play is to map a group of hosts to some well defined roles, represented by things ansible calls tasks. At a basic level, a task is nothing more than a call to an ansible module.

Go ahead and create a new file, juniper.yml or whatever you like, and put the following inside:

---
- hosts: all
  gather_facts: no

  tasks:

  - name: Get Configuration
    junos_command:
      commands:
        - show configuration

Going from top to bottom this playbook tells Ansible to:

  • hosts: all, use the all hosts from the inventory
  • gather_facts: no, don’t gather any facts for now. More on this later.
  • tasks:, this is the list of all the tasks I want you to do.
  • - name:, this is the name of the task. It starts with a - because it’s a list, even if it has only a single entry
  • junos_command:use this module for this task, like -m junos_command. This module, like most, supports a set of parameters which you can see here, and they are below!
  • commands: this is the list of commands to execute
  • - show configuration: this is one of the commands

Now, run ansible-playbook juniper.yml -i juniper-hosts.yml. This will run your playbook, using all the hosts on the previously defined juniper-hosts.yml inventory.

$ ansible-playbook juniper.yml -i juniper-hosts.yml

PLAY [all] ***********************************************************************************************************************************************************************

TASK [Get Configuration] *********************************************************************************************************************************************************
[WARNING]: Platform linux on host 192.168.15.195 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could
change this. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.

ok: [192.168.15.195]

PLAY RECAP ***********************************************************************************************************************************************************************
192.168.15.195             : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Well, that was fun but.. there is nothing useful here, besides a warning! In fact, Ansible connected sucessfully to the router, and retrieved the configuration. We didn’t instructed it to show the configuration to us.

Modify your playbook so it looks like this:

---
- hosts: all
  gather_facts: no

  tasks:

  - name: Get Configuration
    junos_command:
      commands:
        - show configuration
    register: config

  - name: Show Config
    debug: var=config

We added an additional line to the first task, register: config, which saves the result of the task in a variable called config. This name can be whatever you like.

There is also an additional task named Show Config, with a debug: var=config instruction, which outputs the config variable.

Run the playbook once again like you did before, and check the result. Of course it will depend on your configuration. On my vMX, which is almost blank, and just using DHCP for a couple interfaces, the result was this.

$ ansible-playbook juniper.yml -i juniper-hosts.yml

PLAY [all] *****************************************************************************

TASK [Get Configuration] ***************************************************************
[WARNING]: Platform linux on host 192.168.15.195 is using the discovered Python
interpreter at /usr/bin/python, but future installation of another Python interpreter
could change this. See
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html
for more information.

ok: [192.168.15.195]

TASK [Show Config] *********************************************************************
ok: [192.168.15.195] => {
    "config": {
        "ansible_facts": {
            "discovered_interpreter_python": "/usr/bin/python"
        }, 
        "changed": false, 
        "failed": false, 
        "stdout": [
            "## Last changed: 2019-11-26 19:02:46 UTC\nversion 17.1R1.8;\nsystem {\n    root-authentication {\n        encrypted-password \"$6$5LBS/EfQ$tL9utW2Aj4T56SfJUxdnVaF/.RIbaZ65keFn1SbCgOTi6r.LDiGt3FvvoP2WuxuuosVtx0RobNk67obTMeNgF.\";\n    }\n    scripts {\n        inactive: language python;\n    }\n    login {\n        user admin {\n            uid 2000;\n            class super-user;\n            authentication {\n                encrypted-password \"$6$9YynK9hD$Is6rEd7WNnEYGF7q2MqQJoRU/9vGjkQv7Qig.V2WT1905ShVlow4LXKeATM5HR8F1vTwROz2gUpF7z7eCJruo1\";\n            }\n        }\n    }\n    services {\n        ssh;\n        netconf {\n            ssh;\n        }\n    }\n    syslog {\n        user * {\n            any emergency;\n        }\n        file messages {\n            any notice;\n            authorization info;\n        }\n        file interactive-commands {\n            interactive-commands any;\n        }\n    }\n}\ninterfaces {\n    ge-0/0/1 {\n        unit 0 {\n            family inet {\n                dhcp;\n            }\n        }\n    }\n    ge-0/0/9 {\n        unit 0 {\n            family inet {\n                dhcp;\n            }\n        }\n    }\n}"
        ], 
        "stdout_lines": [
            [
                "## Last changed: 2019-11-26 19:02:46 UTC", 
                "version 17.1R1.8;", 
                "system {", 
                "    root-authentication {", 
                "        encrypted-password \"$6$5LBS/EfQ$tL9utW2Aj4T56SfJUxdnVaF/.RIbaZ65keFn1SbCgOTi6r.LDiGt3FvvoP2WuxuuosVtx0RobNk67obTMeNgF.\";", 
                "    }", 
                "    scripts {", 
                "        inactive: language python;", 
                "    }", 
                "    login {", 
                "        user admin {", 
                "            uid 2000;", 
                "            class super-user;", 
                "            authentication {", 
                "                encrypted-password \"$6$9YynK9hD$Is6rEd7WNnEYGF7q2MqQJoRU/9vGjkQv7Qig.V2WT1905ShVlow4LXKeATM5HR8F1vTwROz2gUpF7z7eCJruo1\";", 
                "            }", 
                "        }", 
                "    }", 
                "    services {", 
                "        ssh;", 
                "        netconf {", 
                "            ssh;", 
                "        }", 
                "    }", 
                "    syslog {", 
                "        user * {", 
                "            any emergency;", 
                "        }", 
                "        file messages {", 
                "            any notice;", 
                "            authorization info;", 
                "        }", 
                "        file interactive-commands {", 
                "            interactive-commands any;", 
                "        }", 
                "    }", 
                "}", 
                "interfaces {", 
                "    ge-0/0/1 {", 
                "        unit 0 {", 
                "            family inet {", 
                "                dhcp;", 
                "            }", 
                "        }", 
                "    }", 
                "    ge-0/0/9 {", 
                "        unit 0 {", 
                "            family inet {", 
                "                dhcp;", 
                "            }", 
                "        }", 
                "    }", 
                "}"
            ]
        ], 
        "warnings": [
            "Platform linux on host 192.168.15.195 is using the discovered Python interpreter at /usr/bin/python, but future installation of another Python interpreter could change this. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information."
        ]
    }
}

PLAY RECAP *****************************************************************************
192.168.15.195             : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

I know we all love show config | display set. Try adding display: set as an additional parameter of junos_command. It should look like this.

    junos_command:
      commands:
        - show configuration
      display: set

Run the playbook again, and your output should be as expected if you ran it on the Junos CLI.

Stay tuned for more posts to learn how to configure Juniper Junos using Ansible.

Categories
Projects

Quick Ansible control node with Vagrant

Ansible is an IT automation tools which can configure systems, deploy files and software and orchestrate almost every possible task you can imagine.

Unlike other configuration management tools like Chef or Puppet, Ansible is agent-less, and does all its magic over SSH. Forget about keeping up client daemons updated and additional certificates. Just use your existing SSH keys, or usernames and passwords.

Due to it agent-less nature, we just need a host to initiate SSH sessions towards the managed hosts. This node is called a control node in the Ansible jargon.

On this post, we’ll catch up with the basics of Vagrant and Ansible. The repository for this post is on https://github.com/baldoarturo/ansible-control-node.

Download VirtualBox

VirtualBox is a general-purpose full virtualizer for x86 hardware, targeted at server, desktop and embedded use. Get it from here.

Get Vagrant

We’ll use Vagrant to quickly spin up a control node, based on a ubuntu/xenial64 box. If you are not familiar with Vagrant, you just need to know it is a tool capable to provision VMs on different hypervisors, using a Vagrantfile a VM template.

First, get Vagrant for your system here, https://www.vagrantup.com/downloads.html, and install it.

Use Git

In order to make everything easier, we’ll init a new git repository to keep track of all the configuration changes. Also, we can push this repository to a remote and clone from wherever we need it, making a portable Ansible control node. 

C:\Users\Arturo\Desktop\ansible-control-node>git init
Initialized empty Git repository in C:/Users/Arturo/Desktop/ansible-control-node/.git/

C:\Users\Arturo\Desktop\ansible-control-node>git config user.name "Arturo Baldo"

C:\Users\Arturo\Desktop\ansible-control-node>git config user.email "[email protected]"

C:\Users\Arturo\Desktop\ansible-control-node>

Although everything can be done from a shell, I prefer to use the integrated terminal on VSCode, and make use of the universe of plugins. There is support for Ansible and Ansible Playbooks, Vagrant, a lot more!

Also, even if this post uses a Windows system, everything is reproducible on Linux platforms because all the tools are multiplatform.

Init the Vagrant environment

The Vagrant CLI is self explanatory and easy to understand.

C:\Users\Arturo\Desktop\ansible-control-node>vagrant
Usage: vagrant [options] <command> [<args>]

    -v, --version                    Print the version and exit.
    -h, --help                       Print this help.

Common commands:
     box             manages boxes: installation, removal, etc.
     cloud           manages everything related to Vagrant Cloud
     destroy         stops and deletes all traces of the vagrant machine
     global-status   outputs status Vagrant environments for this user
     halt            stops the vagrant machine
     help            shows the help for a subcommand
     init            initializes a new Vagrant environment by creating a Vagrantfile
     login
     package         packages a running vagrant environment into a box
     plugin          manages plugins: install, uninstall, update, etc.
     port            displays information about guest port mappings
     powershell      connects to machine via powershell remoting
     provision       provisions the vagrant machine
     push            deploys code in this environment to a configured destination
     rdp             connects to machine via RDP
     reload          restarts vagrant machine, loads new Vagrantfile configuration
     resume          resume a suspended vagrant machine
     snapshot        manages snapshots: saving, restoring, etc.
     ssh             connects to machine via SSH
     ssh-config      outputs OpenSSH valid configuration to connect to the machine
     status          outputs status of the vagrant machine
     suspend         suspends the machine
     up              starts and provisions the vagrant environment
     upload          upload to machine via communicator
     validate        validates the Vagrantfile
     version         prints current and latest Vagrant version
     winrm           executes commands on a machine via WinRM
     winrm-config    outputs WinRM configuration to connect to the machine

First, run vagrant intit to initialize a new environment.

C:\Users\Arturo\Desktop\ansible-control-node>vagrant init
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

Wait! You have a new file on your git repository. Make it the first commit.

C:\Users\Arturo\Desktop\ansible-control-node>git add .

C:\Users\Arturo\Desktop\ansible-control-node>git commit -m "First commit"
[master (root-commit) 1416f75] First commit
 1 file changed, 70 insertions(+)
 create mode 100644 Vagrantfile

C:\Users\Arturo\Desktop\ansible-control-node>

On the newly create Vagrantfile, we can set configuration and provisioning options for the VM.

First, replace config.vm.box = "base" with config.vm.box = "ubuntu/xenial64" to use this box.

If you want to see the VM, config vb.gui = true. By default this setting is commented out, so the VM will start in headless mode.

Feel free to customize the VM memory by commenting out the vb.memory = "1024" setting and adjusting it to your needs.

Once you are done, save your changes, commit, execute vagrant up, and watch Vagrant create a brand new VM for you.

C:\Users\Arturo\Desktop\ansible-control-node>vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/xenial64'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'ubuntu/xenial64' version '20191114.0.0' is up to date...
==> default: Setting the name of the VM: ansible-control-node_default_1574353915423_60685
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection reset. Retrying...
    default: Warning: Connection aborted. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: The guest additions on this VM do not match the installed version of
    default: VirtualBox! In most cases this is fine, but in rare cases it can
    default: prevent things such as shared folders from working properly. If you see
    default: shared folder errors, please make sure the guest additions within the
    default: virtual machine match the version of VirtualBox you have installed on
    default: your host and reload your VM.
    default:
    default: Guest Additions Version: 5.1.38
    default: VirtualBox Version: 6.0
==> default: Mounting shared folders...
    default: /vagrant => C:/Users/Arturo/Desktop/ansible-control-node

Well, how do we login into the new VM? Try vagrant ssh

C:\Users\Arturo\Desktop\ansible-control-node>vagrant ssh
Welcome to Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-169-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage


0 packages can be updated.
0 updates are security updates.

New release '18.04.3 LTS' available.
Run 'do-release-upgrade' to upgrade to it.


vagrant@ubuntu-xenial:~$ whoami
vagrant
vagrant@ubuntu-xenial:~$

Type exit and you will return to your system prompt.

You’ll see that you have new items on your directory. A .log file, with the output from Vagrant, and a .vagrant/ directory.

To exclude them from the repository, create a .gitignore, with the following contents.

.vagrant/
*.log

Save and commit.

Provisioning the VM

Return to the Vagrantfile, and add a section like this:

  config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install software-properties-common -y
    apt-add-repository --yes --update ppa:ansible/ansible
    apt-get install ansible -y
  SHELL

This block instructs Vagrant to execute commands on the shell. First, the package list is updated, then the software-properties-common package is installed. To install Ansible, the ppa:ansible/ansible repository is added, and then Ansible is installed.

Notice the usage of -y to avoid interactive prompts.

Save the file, commit, and run vagrant provision. This will re-provision the VM with the new instructions from the Vagrantfile.

Once Vagrant finishes, connect once again with vagrant ssh, and run ansible --version.

vagrant@ubuntu-xenial:~$ ansible --version
ansible 2.9.1
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/vagrant/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Oct  8 2019, 14:14:10) [GCC 5.4.0 20160609]

That’s great! We have just installed Ansible on our new VM.

Seek and Destroy

You might wonder, where is this VM? Head to the VirtualBox Manager, and you’ll find it inside.

Here it is!

Without asking anyone.. kill the machine and delete it.

Hard power off
Delete all files

Head back to the console, and you will see that the SSH session has died. Of course, this makes perfect sense as the VM no longer exists.

vagrant@ubuntu-xenial:~$ Connection to 127.0.0.1 closed by remote host.
 Connection to 127.0.0.1 closed.

If you try to run vagrant provision, it will fail, as there is no VM to provision.

C:\Users\Arturo\Desktop\ansible-control-node>vagrant provision
==> default: VM not created. Moving on...

But what happens if you run vagrant up once again? Surprise! The VM will be recreated with all the previous configuration settings.

Using Ansible

Well, the VM is ready, Ansible is installed.. now what? Let’s write an inventory file and see what can we do.

I’ll create a simple file to connect to a testing docker instance on 192.168.85.253. No need to worry about ssh keys now, as we are testing, username and password will do the trick.

vagrant@ubuntu-xenial:~$ cat docker
[all:vars]
ansible_connection=ssh
ansible_user=MY_USERNAME
ansible_password=MY_PASSWORD

[docker]
192.168.85.253

The [all:vars] section configures the connection settings for all hosts in the file. The [docker] section contains the lists of hosts, in this case just 192.168.85.253.

Let’s run: ansible -m ping -i docker all, where -m is the module to run, -i is the inventory file, and all is the section of the inventory file which contains the hosts. Notice this is a very special ping.

vagrant@ubuntu-xenial:~$ ansible -m ping -i docker all
[DEPRECATION WARNING]: Distribution Ubuntu 16.04 on host 192.168.15.253 should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release
 will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information. This feature will be removed   
in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
192.168.15.253 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

Whoa! That definetly doesn’t look like a ICMP ping. First, this is an old box about to be decomissioned, so it has been very unmantained. Ansible warns us about an old version of Python, and a old version of Ansible which was installed previously on the remote hosts.

Now, try ansible -m setup -i docker all. Be aware this will throw a ton of data to you. The setup module gathers data about the host.

What it does and how does it, is documented here.

Categories
Projects

Machine Learning – Weighted Train Data

Last post talked about an introduction to Machine Learning and how outcomes can be predicted using sklearn’s LogisticReggression.

Sometimes, the input data could require additional processing to prefer certain classes of information, that it considered more valuable or more representative to the outcome.

The LogisticRegression model allows to set the preference, or weight, at the time of being created, or later when being fitted.

The data used on the previous entry had four main classes: DRAFT, ACT, SLAST and FLAST. Once it is encoded and fitted, it can be selected by its index. I prefer to initialize some mnemonics selectors to ease the coding and make the entire code more human friendly.

x_columns_names = ['DRAFT', 'ACT', 'SLAST', 'FLAST']
y_columns_names = ['PREDICTION']

# Indexes for columns, used for weighting
DRAFT = 0
ACT = 1
SLAST = 2
FLAST = 3

# Weights
DRAFT_WEIGHT = 1
ACT_WEIGHT = 1
SLAST_WEIGHT = 1
FLAST_WEIGHT = 1

The model can be initialized lated using the following method, where the class_weight parameter is used referencing the previous helpers.

model = LogisticRegression(
    solver='lbfgs',
    multi_class='multinomial',
    max_iter=5000,
    class_weight={
        DRAFT: DRAFT_WEIGHT,
        ACT: ACT_WEIGHT,
        SLAST: SLAST_WEIGHT,
        FLAST: FLAST_WEIGHT,
    })

Categories
Projects

Machine Learning – Classification and Regression Analysis

Machine Learning is the science and art of programming computers so they can learn from data.

For example, your spam filter is a Machine Learning program that can learn to flag spam given examples of spam emails (flagged by users, detected by other methods) and examples of regular (non-spam, also called ā€œhamā€) emails.

The examples that the system uses to learn are called the training set. The new ingested data is called the test set. The performance measure of the prediction model is called accuracy and it’s the objetive of this project.

The tools

To tackle this, Python (version 3) will be used, among the package scikit-learn. You can find more info about this package on the official page.

https://scikit-learn.org/stable/tutorial/basic/tutorial.html

Supervised learning

In general, a learning problem considers a set of n samples of data and then tries to predict properties of unknown data. If each sample is more than a single number and, for instance, a multi-dimensional entry (aka multivariate data), it is said to have several attributes or features.

Supervised learning consists in learning the link between two datasets: the observed data X and an external variable y that we are trying to predict, usually called ā€œtargetā€ or ā€œlabelsā€. Most often, y is a 1D array of length n_samples.

All supervised estimators in scikit-learn implement a fit(X, y) method to fit the model and a predict(X) method that, given unlabeled observations X, returns the predicted labels y.

If the prediction task is to classify the observations in a set of finite labels, in other words to ā€œnameā€ the objects observed, the task is said to be a classification task. On the other hand, if the goal is to predict a continuous target variable, it is said to be a regression task.

When doing classification in scikit-learn, y is a vector of integers or strings.

The Models

LinearRegression, in its simplest form, fits a linear model to the data set by adjusting a set of parameters in order to make the sum of the squared residuals of the model as small as possible.

LogisticRegression, which has a very counter-intuitive model, is a better choice when linear regression is not the right approach as it will give too much weight to data far from the decision frontier. A linear approach is to fit a sigmoid function or logistic function.

../../_images/sphx_glr_plot_logistic_001.png

The Data

Data is presented on a CSV file. It has around 2500 rows, with 5 columns. Correct formatting and integrity of values cannot be assured, so additional processing will be needed. The sample file is like this.

The Code

We need three main libraries to start:

  • numpy, which basically is a N-dimensional array object. It also has tools for linear algebra, Fourier transforms and random numbers.
    It can be used as an efficient multi-dimensional container of generic data, where arbitrary data-types can be defined.
  • pandas, which provides high-performance and easy-to-use data structures and data analysis tools simple and efficient tools for data mining and data analysis
  • sklearn, the main machine learning library. It has capabilities for classification, regression, clustering, dimensionality reduction, model selection and data preprocessing.

A non essential, but useful library is matplotlib, to plot sets of data.

In order to provide data for sklearn models to work, it has to be encoded first. As the sample data has strings, or labels, a LabelEncoder is needed. Next, the prediction model is declared, where a LogisticRegression model is used.

The input data file path is also declared, in order to be loaded with pandas.read_csv().

import pandas as pd
import numpy as np
import matplotlib.pyplot as pyplot

from sklearn.preprocessing import LabelEncoder
from sklearn.linear_model import LogisticRegression

encoder = LabelEncoder()
model = LogisticRegression(
    solver='lbfgs', multi_class='multinomial', max_iter=5000)

# Input dataset
file = "sample_data.csv"

The CSV file can be loaded into a pandas dataframe in a single line. The library also provides a convenient method to remove any rows with missing values.

# Use pandas to load csv. Pandas can eat mixed data with numbers and strings
data = pd.read_csv(file, header=0, error_bad_lines=False)
# Remove missing values
data = data.dropna()

print("Valid data items : %s" % len(data))

Once loaded, the data needs to be encoded in order to be fitted into the prediction model. This is handled by the previously declared LabelEncoder. Once encoded, the x and y datasets are selected. The pandas library provides a way to drop entire labels from a dataframe, which allows to easily select data.

encoded_data = data.apply(encoder.fit_transform)
x = encoded_data.drop(columns=['PREDICTION'])
y = encoded_data.drop(columns=['DRAFT', 'ACT', 'SLAST', 'FLAST'])

The main objective is to test against different lengths of train and test data, to find out how much data provides the best accuracy. The lengths of data will be incremented in steps of 100 to get a broad variety of results.

length = 100
scores = []
lenghts = []
while length < len(x):
    x_train = x[:length]
    y_train = y[:length]
    x_test = x.sample(n=length)
    y_test = y.sample(n=length)
    print("Fitting model for %s training values" % length)
    trained = model.fit(x_train, y_train.values.ravel())
    score = model.score(x_test, y_test)
    print("Score for %s training values is %0.6f" % (length, score))
    length = length + 100
    scores.append(score)
    lenghts.append(length)

Finally, a plot is made with the accuracy scores.

pyplot.plot(lenghts,scores)
pyplot.ylabel('accuracy')
pyplot.xlabel('values')
pyplot.show()
Categories
Projects

Customizing NetBox Templates

NetBox is an IP address management (IPAM) and data center infrastructure management (DCIM) tool. Initially conceived by the network engineering team at DigitalOcean, NetBox was developed specifically to address the needs of network and infrastructure engineers.

Image result for netbox device types

When I started using NetBox on my daily job, I planned to use it as a replacement for all the spreadsheets I had for switch configurations, IP address management, secrets, and VLAN assignments. NetBox can handle all of this and more, but the interface didn’t suit my needs.

NetBox is built using the Python Django framework, which I have used for another projects. I used Visual Studio Code to clone the repository and debug, as it has native support for the Django template language.

I keep a copy of the repository on my local machine for ease of modifications. Prior, I have set DEBUG=TRUE on netbox/configuration.py, and allowed localhost and my local network to access the development server. Also, I set the correct settings to connect to the existing postgresql database.

Connecting the existing DB to my local development server

This environment works for test purposes, but the best you can do is to set up separated development and production environments, and commit your changes to production once everything is tested.

Using VSCode to debug Django

The URL definition for the single device view is around line #147 of the netbox/dcim/urls.py file, and it looks like this.

 url(r'^devices/(?P<pk>\d+)/$', views.DeviceView.as_view(), name='device'),

Heading to the DeviceView view, I put a breakpoint on the interfaces
QuerySet of the view definition, and launched the debugger. The default location is at http://localhost:8000.

Setting up the debugger
Breakpoints

I headed to http://localhost:8000/dcim/devices/570/, where I had defined a switch with several VLANs, to hit the breakpoint and find out if the
QuerySet had information about the VLANs, or if they were queried in a per-interface basis, on the interface view.

QuerySet returns this

Lucky me, the QuerySet recovered all the information I needed, and it is passed to the template via a render()Ā call.

All the information I want is rendered on this table. This is the power of the Django framework. I added line #513 as an additional header for the VLANs column.

This table has a for loop which iterates for each interface of the device, so I edited the included template file at dcim/inc/interface.html.

Both tagged and untagged VLANs groups have a bolded title, and the VID and VLAN name is shown after it. I used the dictsort filter, which is part of the Django framework, to sort all the VLANs by their VID.

dcim/inc/interface.html

The final result looks like the following image, and it allows to keep track of all the VLANs on all ports, at first sight. This is easier and more user friendly than getting that information interface per interface, or making a new custom view.

New Template Rendering