A little about Edgerouters IPtables.
I wanted to follow the traffic through my firewall rules on my Edgerouter. Everything is based on IPtables and from the CLI I can get the statistic of how much traffic hits each rule. The following command shows all the firewall rules that are generated from the Edgerouters configuration:
root@edge:/# iptables -L -v -x | grep '*'
75936 8212002 DROP all -- any any anywhere anywhere /* WAN_LOCAL-10000 default-action drop */
75158612 100458947029 RETURN all -- any any anywhere anywhere /* WAN_IN-10 */ state RELATED,ESTABLISHED/* WAN_IN-10 */ state RELATED,ESTABLISHED
bla bla bla....
To break down this:
- First column shows the number of packets hitting the rule
- Second column shows the number of bytes hitting the rule
- Third column shows the action of the rule.
- The text in the comment /* xxxxx-yy */: xxxxx = Ruleset name, yy = rule number
Collecting IPtables
I made a small script “/config/scripts/get-snmp-iptables” (chmod 755):
#!/bin/bash /sbin/iptables -L -v -x > /tmp/iptables-list
I scheduled it in the edgerouters config with these lines:
set system task-scheduler task get-snmp-iptables executable path /config/scripts/get-snmp-iptables set system task-scheduler task get-snmp-iptables interval 5m
The reason I did this is because the iptables command need to be run as root. I was not able to run it from inside the snmpd.
Every 5 minutes the IPtables are dumped to a file in the tmp-dir. This file will be read by snmpd when it is queried.
Filtering IPtables
I then created this script “/config/scripts/filter-snmp-iptables” (chmod 755):
#!/bin/bash function findRule () { echo "$ipt" | /usr/bin/awk -v param="$1" '\ BEGIN { split(param,paramsplit,":") rule=paramsplit[1] action=paramsplit[2] counter=paramsplit[3] } match($0, action ".+" rule) { if ( counter=="BYTES" ) { print $2 } if ( counter=="PACKETS" ) { print $1 } } ' } ipt=$(cat /tmp/iptables-list) for param in "$@" do findRule "$param" done
This script can be called with as many parameters as wanted. Each parameter represents a firewall rule that should be monitored. Here is an example of calling it:
root@edge:/# filter-snmp-iptables WAN_IN-10:RETURN:BYTES WAN_LOCAL-10000:DROP:PACKETS 100458947029 75936
In this example it will output two lines:
- The first line is the number of bytes going through rule number 10 in the rule-set with then name “WAN_IN”. We are looking for rules with the action of RETURN
- The second line is the number of packet going through rule number 10000 in the rule-set “WAN_LOCAL”. We are looking for rules with the action of DROP. Rule 10000 in edgerouter is the default action rule.
The output of this script is used by the SNMP daemon on the edgerouter.
Setting up SNMP
To setup SNMP we need to add the following to the configuration:
set service snmp community SEEEEECRET authorization ro
To add my iptables-snmp script to snmpd, I added this line to “/etc/snmp/snmpd.conf”:
extend iptables /config/scripts/filter-snmp-iptables WAN_IN-10:RETURN:BYTES WAN_IN-10:RETURN:PACKETS WAN_LOCAL-10000:DROP:PACKETS
After that I restarted snmpd:
service snmpd restart
Now I tested that I could get my counters:
root@edge:/# snmpwalk -c SEEEEECRET -v2c localhost .1.3.6.1.4.1.8072.1.3.2.4.1.2.8.105.112.116.97.98.108.101.115
iso.3.6.1.4.1.8072.1.3.2.4.1.2.8.105.112.116.97.98.108.101.115.1 = STRING: "100458947029"
iso.3.6.1.4.1.8072.1.3.2.4.1.2.8.105.112.116.97.98.108.101.115.2 = STRING: "75158612"
iso.3.6.1.4.1.8072.1.3.2.4.1.2.8.105.112.116.97.98.108.101.115.3 = STRING: "75936"
Making it permanent – almost!
Whenever the Edgerouter is rebooted the configuration in snmpd.conf gets removed because vyatta rebuilds this file from the current configuration. In order to make it survive, I made a script “/config/scripts/post-config.d/customize-snmpd” (chmod 755):
#!/bin/bash echo extend iptables /config/scripts/filter-snmp-iptables WAN_IN-10:RETURN:BYTES WAN_IN-10:RETURN:PACKETS WAN_LOCAL-10000:DROP:PACKETS >> /etc/snmp/snmpd.conf /sbin/iptables -L -v -x > /tmp/iptables-list # make sure that we have some initial stats when snmpd is up and running service snmpd restart
Whenever a change is made to Edgerouters configuration regarding SNMP, then the manually added extend-line will disapper. But a reboot or running this script manually will bring back the lost extend section of snmpd.conf
Whats next
Setting it up in opennms! See part2
Hi,
This will work only if your edgerouter don’t have hadware offload since the offloaded packets are not seen by the OS.
Alesk
Hi Alesk,
I have enabled HW offload but I still see the packages.
Are your sure?
Best regards,
Alex
100% sure on ERL, the packets the kernel see (and viewable with tcpdump or the iptables counter) are the ones the hardware failed to process. This is why the counter still increase, but it’s far from the real traffic.
Example live test @home:
Before:
Chain outside-inside (1 references)
pkts bytes target prot opt in out source destination
552K 131M RETURN all — * * 0.0.0.0/0 0.0.0.0/0 /* outside-inside-1 */ state RELATED,ESTABLISHED
Then, I download a 1GB file, then:
Chain outside-inside (1 references)
pkts bytes target prot opt in out source destination
553K 131M RETURN all — * * 0.0.0.0/0 0.0.0.0/0 /* outside-inside-1 */ state RELATED,ESTABLISHED
The kernel saw 1K packets but that represents less than a MB, which is far from the GB downloaded.
OK, maybe I have to look into that. Then I don’t think my device is offloading good enough.
Well – offloading or not, I can still get the speed my ISP promises 🙂
Thanks for the tip.