Error-Disable

Posted on 30 May 2017 by Robert

When a network port encounters an error. This port is automatically shutdown to ensure network stability. When error-disable is enabled.

The predefined error are as follows;

arp-inspection 
bpduguard
channel-misconfig
dhcp-rate-limit              
gbic-invalid                 
l2ptguard                  
link-flap
mac-limit
link-monitor-failure
loopback
oam-remote-failure
pagp-flap
port-mode-failure
psecure-violation
security-violation
sfp-config-mismatch
storm-control
udld
vmps

Primary function of error-disable is fault detection. Secondary you can set up recovery, per predefined error.  This can be done by ;

errdisable recovery cause

Be sure to set the recovery interval to your specific needs;

errdisable recovery interval 60

Whenever  a port is disabled due to an error-disable condition you should see something like ;

somerouter#show interface GigabitEthernet1/6
GigabitEthernet1/6 is down, line protocol is down (err-disabled)
--//output omitted for brevity//--

Interesting to see that the port is in down/down status with an err-disabled condition.

If you do not resolve the root cause the log will be flooded with

Jan 10 08:51:35.074 CET: %PM-4-ERR_RECOVER: Attempting to recover from link-flap err-disable state on Gi1/6
Jan 10 08:51:37.179 CETT: %PM-4-ERR_DISABLE: link-flap error detected on Gi1/6, putting Gi1/6 in err-disable state

To resolve the problme one has to know the root cause, for this you’ll have to issue the command “show errdisable recovery”. First you see a list of errors where recovery is enabled, or disabled. (See column ‘Timer Status’)

somerouter#sh errdisable recovery
ErrDisable Reason            Timer Status
-----------------            --------------
arp-inspection               Disabled
bpduguard                    Enabled
channel-misconfig            Enabled
dhcp-rate-limit              Disabled
gbic-invalid                 Enabled
l2ptguard                    Disabled
link-flap                    Enabled          
mac-limit                    Disabled
link-monitor-failure         Disabled
loopback                     Disabled
oam-remote-failure           Disabled
pagp-flap                    Enabled
port-mode-failure            Disabled
psecure-violation            Enabled
security-violation           Enabled
sfp-config-mismatch          Disabled
storm-control                Disabled
udld                         Enabled
vmps                         Enabled

Timer interval: 60 seconds                     

Interfaces that will be enabled at the next timeout:

Interface       Errdisable reason       Time left(sec)
---------       -----------------       --------------
Gi1/6                 link-flap           47

Root cause for the problem with interface Gi1/6 lies in link-flap. This is somewhat cryptic but will give you an idea where to find the solution.

Possible root causes are;
dodgy cabling
faulty optics (sfp/gbic)
wrong speed/duplex settings
hardware loop

In this particular case I suspected the speed/duplex settings.

somerouter(config-if)#speed nonegotiate
somerouter(config-if)#shut
somerouter(config-if)#no shut
somerouter(config-if)#
Jan 10 11:20:31.589 CET: %LINK-5-CHANGED: Interface GigabitEthernet1/6, changed state to administratively down
somerouter(config-if)#
Jan 10 11:20:33.703 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/6, changed state to up
somerouter(config-if)#
Jan 10 11:20:33.712 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/6, changed state to up

Link was forced down and up to get rid of the error-disable condition.
Terminal monitor was turned on to instantly see what is going on.
Logging tells me that the port is admin down and that changed to up.
Et voilà, line protocol is also changed to up.

Root cause found and resolved.

Posted in CCNP | Tagged , , | Comments Off on Error-Disable

SSH error on cisco router

Posted on 14 March 2017 by Robert

Enabling SSH on a Cisco router is not enough to guarantee SSH is working.
Enabling ssh version 2.0 on a routers is as straight forward as;.

>ip ssh version 1-2

I surely recommend the use of version 2, because ssh v1 has a bug which allows root access.

Anyhow… ssh needs some sort of encryption to work. In other words create a crypto key.

host(config)#crypto key generate rsa modulus 2048
The name for the keys will be: host.somenetwork.com
% The key modulus size is 2048 bits
% Generating 2048 bit RSA keys, keys will be non-exportable...
[OK] (elapsed time was 2 seconds)
host(config)#
Nov  5 14:17:17.240 UTC: %SSH-5-ENABLED: SSH 2.0 has been enabled

The good thing is the crypto key is not saved in config. The bad thing is, whenever you commission a router and forget to create the crypto key it can cost you some time in troubleshooting. Worst case you have to travel on-site to create the crypro key.

By the way, make sure you enable ssh on the VTY ports and if applicable deny telnet input.

line vty 0 15 
transport input ssh
no transport input telnet

And create an access-list and apply it to the VTY ports.

FYI: SSH enables encrypted CLI access towards a host. And the crypto key validates the host communication i.e. detection of a man-in-the-middle attack.

Posted in CCNP | Comments Off on SSH error on cisco router

DHCP Helper address

Posted on 6 March 2017 by Robert

This week a project manager contacted me with an rather peculiar question. I work with this project manager now for almost two years. And I know he is technically very skilled.  So if he doesn’t know the answer then the answer is not always straight forward.

I currently work for a large telco in the Netherlands. One of the services offered is DHCP-relay. Technically this is a very basic and straight forward service. The customer wanted to know which address is used as source address in the packets destined to the DHCP server. This was needed to set some static routes.

Okay lets start labbing, within GNS3 I made the following lab setup;

DHCPHelper_Source

IP addresses used are;

  • DHCP-Server
    • e0/0   172.16.1.1
  • Router
    • e0/0   172.16.1.2
    • e1/0.1 192.168.1.1 (switch management vlan)
    • e1/0.100 192.168.100.1 (customer vlan)
  • Switch
    • vlan1 192.168.1.10
  • PC1
    • e0 will get ip from DHCP server.

Router config  interface facing the dhcp client.

interface Ethernet1/0.100
 encapsulation dot1Q 100
 ip address 192.168.100.1 255.255.255.0
 ip helper-address 172.16.1.1  
end

With the set up ready I am able to get an ip address. During the DHCP negotiation wireshark is gathering information on the link between the router and the dhcp-server.

DHCP request and reply

Opcode: 1 (REQUEST)
Client IP Address: 192.168.100.13
Your IP Address: 0.0.0.0
Server IP Address: 0.0.0.0
Gateway IP Address: 0.0.0.0
Client MAC Address: 00:50:79:66:68:00
Option 53: Message Type = Request
Option 54: DHCP Server = 172.16.1.1
Option 50: Requested IP Address = 192.168.100.13
Option 61: Client Identifier = Hardware Type=Ethernet MAC Address = 00:50:79:66:68:00
Option 12: Host Name = PC11

Some messages have been omitted for readability. In the process prior to assigning an IP address the DHCP server chooses an IP and checks uniqueness in the network using ping.

Opcode: 2 (REPLY)
Client IP Address: 192.168.100.13
Your IP Address: 192.168.100.13
Server IP Address: 0.0.0.0
Gateway IP Address: 192.168.100.1 
Client MAC Address: 00:50:79:66:68:00
Option 53: Message Type = Ack
Option 54: DHCP Server = 172.16.1.1
Option 51: Lease Time = 86400
Option 58: Renewal Time = 43200
Option 59: Rebinding Time = 75600
Option 1: Subnet Mask = 255.255.255.0
Option 3: Router = 192.168.100.1

 IP 192.168.100.13/24 GW 192.168.100.1

To be honest I set a static route on the DHCP-server

ip route 192.168.100.0 255.255.255.0 172.16.1.2

What does Wireshark tell me ?  This whole exercise is undertaken to ‘see’ what information passes from the router to the dhcp-server and vice-versa.

I expect a dhcp discover followed by a offer. Between discover and offer the dhcp-server should ping the ip address which it is going to serve to the dhcp-client. No ping reply means; ip address if not currently in use. No duplicate ip addresses are injected.

DHCPHelper_Wireshark

A little close-up reveals the answer;

DHCPHelper_Wireshark2

The DHCP unicast packet is sourced by 192.168.100.1 which is the router interface facing the dhcp-client. This interface contains the ip helper-address 172.16.1.1 statement.

NOTE: the network diagram is overly simplified. First because I do not wish to reveal any business proprietary information. Second it withdraws focus from the question at hand.

NOTE2: Thank you David Theuns for being your ‘vraagbaak’.

Posted in CCNP, GNS3 | Tagged , , , | 1 Comment

GNS3 caveat – port status

Posted on 26 February 2017 by Robert

Creating a network lab in GNS3 is reliefs on budget. But there are some drawbacks. One of which is port status. I have noticed that using GNS3 in combination with iouyap results in some unexpected behavior.

Once a topology is made in GNS3. And one interface is set administrative operational, with no shut. The port immediately becomes operational i.e. up/up. Within GNS3 there is no such thing as admin up/operational down.

At this point I know that two protocols can help detect L2 communication with adjacent node. Those are UDLD and BDF.

Beware not to waste time in resolving/troubleshooting GNS3 problem. Instead of resolving network problems.

Posted in Commands, GNS3 | Tagged , , , | Comments Off on GNS3 caveat – port status

Extended IP Access-list

Posted on 19 February 2017 by Robert

As we all have learned in our pursuit to CCNA there are many different types of access-list. In reality most commonly used are ip access lists.

Major difference between ip standaard access-list an ip extended access-list is the ability to insert and remove entries within the ACL. When an entry has to be removed with in an standard ACL the entire ASL need to be removed and added again. The latter off course without the ACE which needs removal.

A extended ACL has slightly more steps but has some major benefits. Let say you have the following ACL;

access-list 130 permit ip host 1.1.1.1 host 99.99.99.99
access-list 130 permit ip host 3.3.3.3 any
access-list 130 deny ip any any

And the entry “permit ip host 3.3.3.3 any” must be replace with “permit ip host 3.3.3.3 host 99.99.99.99”

The following steps need to be taken are;

  • Obtain information about the ‘rule number’
  • Insert new ACE
  • Remove wrong ACE
  • Resequence rule numbers

Obtain rule number information

R4#sh access-lists 130
Extended IP access list 130
 10 permit ip host 1.1.1.1 host 99.99.99.99
 20 permit ip host 3.3.3.3 any
 30 deny ip any any

Insert new ACE

Rule 11 until 19 are free to use. Same goes for 21 until 29. Let use number 15.

R4#conf
*Feb 19 13:13:35.871: %SYS-5-CONFIG_I: Configured from console by console
R4#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R4(config)#ip access-list extended 130 
R4(config-ext-nacl)#15 permit ip host 3.3.3.3 host 99.99.99.99
R4(config-ext-nacl)#end
R4#
*Feb 19 13:14:24.206: %SYS-5-CONFIG_I: Configured from console by console
R4#sh access-lists 130 
Extended IP access list 130
 10 permit ip host 1.1.1.1 host 99.99.99.99
 15 permit ip host 3.3.3.3 host 99.99.99.99
 20 permit ip host 3.3.3.3 any
 30 deny ip any any

Remove wrong ACE

ACE that needs removal has rule number 20. Removal is as easy as “no 20” within the access-list.

R4#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R4(config)#ip access-list extended 130 
R4(config-ext-nacl)#no 20
R4(config-ext-nacl)#exit
R4(config)#end 
*Feb 19 13:18:32.452: %SYS-5-CONFIG_I: Configured from console by console
R4#sh ip access-lists 130 
Extended IP access list 130
 10 permit ip host 1.1.1.1 host 99.99.99.99
 15 permit ip host 3.3.3.3 host 99.99.99.99
 30 deny ip any any
R4#

The rule numbers are somewhat borked. This needs a little repair.

Resequence rule numbers

Repair command is;

ip access-list resequence

R4#
*Feb 19 13:22:42.838: %SYS-5-CONFIG_I: Configured from console by console
R4#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R4(config)#ip access-list resequence 130 10 10
R4(config)#end
R4#sh 
*Feb 19 13:23:12.473: %SYS-5-CONFIG_I: Configured from console by console

R4#sh access-list 130 
Extended IP access list 130
 10 permit ip host 1.1.1.1 host 99.99.99.99
 20 permit ip host 3.3.3.3 host 99.99.99.99
 30 deny ip any any
R4#

Naturally the most important benefit is that the access-list is applied all the time. In regard of the standard ip access-list; a slight modification results in removing the ACL and re-instating the ACL again. You will never know which traffic passes the interface while working on the access-list.

The same applies for named access-list.

NOTE: 

Cisco NEXUS devices have a slight different syntax to resequence

NEXUS-R3(config)# resequence ip access-list some_vlan 10 10
Posted in CCNP, Commands | Tagged , , , , | Comments Off on Extended IP Access-list

EIGRP neighbors

Posted on 5 February 2017 by Robert

Cisco EIGRP protocol allows different methods to form neighborship relations. Default method is form neighborship automatically via multicast (multicast address of 224.0.0.10 is used)

After an adjacency is formed ‘hellos’ are  sent using IP protocol 88 (eigrp). This means that;

  1. an interface need to be IP enabled
  2. both interfaces to a link need to have an ip address within the same subnet.

Alternative method of creating an neighborship relation is setting the neighborship ‘hard-coded’. This is done withing the eigrp routing context. Setting a neighborship ‘over’ an interface implicitly disables the ability to form adjacencies using multicast. Inbound EIGRP multicast packets are discarded and outbound are not send out that particular interface.

This alternative method can be used when you want to prevent a rogue router in your network inject all sorts of garbage subnets.

Off course a better way to set a md5 key on eigrp update exchanges.

Posted in Commands | Comments Off on EIGRP neighbors

Cisco Express Forwarding

Posted on 7 January 2017 by Robert

An topic that entertained my imagination is Cisco’s IP CEF. So I had to take a dive into whichever forwarding methods there are for Cisco routers.

Currently there are three methods of packet forwarding;

  • Processor based forwarding
  • Fast forwarding (reactive)
  • Cisco Express Forwarding (proactive)

Processor based forwarding is as the name describes processor intensive. When for each packet the CPU has to harvest information from various tables (sources)  this would delay packet forwarding significantly. Or you would need to upgrade a router chassis with a more powerful routing engine. Routers have various tasks at hand; forwarding packets, running routing protocols, maintaining an arp table and handling other processes e.g. ntp, snmp, syslog. Just to name a few.

At some point Cisco and other vendors decide to walk another path. Here is the Cisco solution.

Fast Forwarding uses a specially created table fastswitching-cache. When a packet arrives at a router for which the destination is not in the cache. The router invokes the IP Input process (cpu intentive). IP input gathers information about how to forward the packet to its destination. This information is stored in the fastswitching-cache. And the packet is forwarded accordingly. At this point there is no advantage for this single packet. But for the next packet going to the same destination there is a advantage.

The CPU is used only once in determining the destination information. However there is a downside; if the outgoing interface would change, for example by a network failure, then this information is NOT passed through to the fastswitching-cache. The entry in the fastswitching-cache is only cleared after a time-out is reached.

Cisco Express Forwarding is an entire different beast. The table used is called  forwarding information base or FIB for short. When a packet arrives it is handled bij the IP input process. Same process as processor switching and fast switching. But when all relevant information is gathered is it stored into the FIB. At this point no major difference to Fast Forwarding. CEF also , proactively, stores information from network protocols into the FIB as well. This enables shorter forwarding delay times. And while  CEF uses specific information (FIB) in a specific process (CEF) the processing can be done in specialized hardware; Application Specific Integrated Circuit or ASIC for short. For starters this relieves that main CPU from packet forwarding tasks. And secondly this also eliminates any forwarding delay.

In the next post I will dive into how to read the output of show ip cef.

Sources:

https://en.wikipedia.org/wiki/Forwarding_information_base

https://en.wikipedia.org/wiki/Cisco_Express_Forwarding

http://www.cisco.com/c/en/us/td/docs/ios/12_2/switch/configuration/guide/fswtch_c/xcfcef.html

https://learningnetwork.cisco.com/thread/12668

Posted in CCNP, Commands | Tagged , | Comments Off on Cisco Express Forwarding

Virtual Lab setup

Posted on 27 December 2016 by Robert

Today I want to talk about the lab setup I have made. But first a slight warning; I will not discuss any use of illegal software nor will I tell you how to get IOS/IOU files.

For quite some years I used Windows as an operating system. In 2009 I made a radical change; I installed Ubuntu as a main operating system. Gradually Ubuntu was installed on the PC’s in my house-hold. As Ubuntu has a less demanding hardware requirement list the usage span of the hardware was also stretched. Spending less money on computer hardware is big and very welcome bonus. Ubuntu can be found on https://www.ubuntu.com/

Virtual LAB software

For network simulation/lab software I use GNS3. GNS3 also helped my decision to run Ubuntu. As GNS3 is much more stable on a linux environment. GNS3 can be found on https://www.gns3.com/ Create a free account and search for the installation manual.

The standard Ubuntu telnet client doesn’t support tabbed views. That is why Konsole was added to the installation. The GNS3 preferences need some adjustment, add the following; konsole –new-tab -p tabtitle=”%d” -e “telnet %h %p”. To the console application command for telnet setting.

Hardware

My laptop with an Intel I5, carries 16Gb of RAM. And my desktop also an Intel I5 is fully loaded with 32Gb of RAM. Lots of RAM make simulations run smooth as silk. The laptop runs with an SSD drive. Remember that each virtual router has its own memory footprint e.g. more virtual router equals to more memory usage. Cisco csr1000v gobbles up a staggering 3Gb memory per virtual router and if I remember correctly Juniper olive still eats a modest 2Gb of memory. IOS and IOU virtual routers have much smaller memory footprint.

Overview

  • Software:
    • Ubuntu
    • GNS3
    • Konsole
    • Network OS image files of your liking. (Cisco IOS/Cisco IOU/Juniper Olive …)
  • Hardware:
    • Intel I5
    • Memory slots completely filled up and maxed out.

Summary

You dont have to be brave to make a setup with GNS3. Although  little persistence is needed and some help from your friend McGoogle.

 

UPDATE: 2017-feb-26 Today I wrote a little article about a short-coming I found in GNS3. GNS3 caveat – port status I am sure some more will follow. GNS3 is a fantastic tool for emulating a netwerk. But it is still an emulation. Nothing beats the real thing; a stack of hardware.

 

 

 

 

Posted in GNS3 | Tagged , , , | Comments Off on Virtual Lab setup

CCNP Books have arrived

Posted on 17 December 2016 by Robert

The 300-115 (switch)  book was already in my possession.  This week the last two books arrived; 300-101 (route)  and 300-135 (tshoot).
For some time I was cramming labs from whichever tutorial I could lay my hands on. Now the study will be more structured using the official cert guides.

 

Posted in CCNP | Tagged , , , | Comments Off on CCNP Books have arrived

T-Mobile – Evernote course

Posted on 29 June 2014 by Robert

evernote_smallLast week I received an email with en invite to attend a free Evernote course. Free is good and Evernote is great tool to store and organize brainfarts. Immediately I jumped to the occasion to enroll to the course.

I am very  curious to the new things I am about to learn in using Evernote. The course will be given in an auditorium of T-Mobile head quarters in The Hague.

Enabling initiative like this, T-Mobile is giving something back to the community.Well done….. very well done.

Posted in Lifehacking | Tagged , | Comments Off on T-Mobile – Evernote course