This document is intended as a training guide for the Application Pack team members. This guide is an abridged version of the technical document written by Stephane Robin. The entire technical document can be found at:
http://pmg:8080/CME/uploads/326/QLAB2090C_US-SNMP-Formulas-Technical-Guide.1.doc.
2 Vocabulary
Element managed node or host
Sub-element sub-component of an element (physical or logical object), such as port, interface, virtual circuits, or DLCI
Resource element or sub-element: any managed item
Metric calculation performed against raw SNMP or Bulk data, formerly known as “Formula” in Netcool/Proviso. It is a combination of OIDs, functions, and standard mathematical operations.
Lite Metric is a specific class of metric that only uses a subset of metric features. This allows a collector to optimize the calculations. All collection metrics should be translated into “lite” metrics for performance reasons.
Complex Metric is a specific class of metric that uses all of the metric features. Discovery metrics can be “complex”.
Discovery Metric used with the Netcool/Proviso Inventory Tool during the process of discovering and analyzing the network
Collection Metric used in the Netcool/Proviso Request Editor to collect information (polling) about various devices in the network
Variable an internal container for a set of temporary results. Variables can be produced and populated as result of an expression and used in other expressions. Variables are not saved between executions of the same formula.
OID is the basic information that a SNMP agent can provide. An OID is typed, and indexed within a MIB table. Each line of a table corresponds to a different instance of the information represented in the table.
MIB (Management Information Base) a definition of the management information that can be read from and possibly written to the management interface of an SNMP compliant device.
Meta Data configuration data (slowly varying data)
Data collected data (dynamic data)
3 Variables
There are 3 classes of variables: Host (Hx), Instance (Ix), and Temporary (Vx). Where “x” is an integer starting at one - H1, I1, V1.
NOTE: It’s good practice to use a two digit number format for Temporary variables - V01. This leaves room for 99 Temporary variables. Using only V1 through V9 leaves you with only 9 variables in that simply continuing on with V10 etc is not supported.
3.1 Magic variable
Usage: %
H1 The value of this variable represents the current host name. This value is set either by the formula editor, by setting a target Element or Sub-Element for test, or by defining a target in the request editor table
RCOMMUNITY The value of this variable represents the read community string name
USEDRCOMMUNITY
WCOMMUNITY The value of this variable represents the write community string name
USEDWCOMMUNITY
HOSTNAME
HOSTLONGNAME
HOSTUSEDNAME
HOSTIP The value of this variable represents IP address of the host.
3.2 Instance variable
(Ix variables) - where x is an integer starting at 1. These variables represent input values for a formula. There 2 ways of setting values for an Instance variable.
Formula Editor settings values in the instance field.
Request Editor assigning a Sub-Element to a request. Values for those Ix variables are deducted from the content of the instance field associated with each Sub-Element.
Example 1: (Formula Editor)
Dim I1 as Integer Default ’10’;
Dim I2 as Integer Default ‘11’;
Will produce the following substitutions: I1 in {10} ; I2 in {11}
Example 2: (Formula Editor)
Dim I1 AS Integer Default * NAME Interface;
# 800 is 8 for octets to bits times 100 for division by seconds instead of deciseconds;
800 * delta(ifInOctets.%I1) / delta(sysUpTime.0)
Will produce the following substitutions for a device with three (3) interfaces: I1 in {1 2 3}
NOTE: If no “Default” is defined, and a replacement value is required for an Ix variable, then ‘*’ will be used.
3.2.1 Special Case %I0
There is an I0 Instance variable which in some cases makes sense to use for Collections. Using the %I0 is potentially dangerous, but can be very useful. BE CAREFUL using the I0 variable because it can create a tremendous performance hit for Proviso if used inappropriately.
When to us %I0:
# This is a collection formula;
Dim I0 AS Integer Default *;
ave( *, rbnCPULoad.%I0);
Explanation:
The discovery formula created a NULL sub-element ONLY. There are no sub-elements for the CPUs. The customer wants a report showing the Device with CPU statistics as a column in the report table, without discovering the CPUs. Therefore the only sub-element is the
When not to use %I0:
# This is a collection formula;
Dim I0 AS Integer Default *;
ave( *, vlanIfTx.%I0);
Explanation:
The discovery formula created a NULL sub-element ONLY. There are no sub-elements for the VLAN Interfaces. The customer wants a report showing the Device with VLAN statistics as a column in the report table, without discovering the VLAN interfaces. Therefore the only sub-element is the
3.2.2 Instance variable with multiple dimensions
Based on the complexity of the agent table, there may be multiples dimensions (input variables) required to get a single result.
3.2.2.1 DLCI table
The DLCI table is an example of this situation. Each DLCI is referenced by 2 numbers, the interface number, and the dlci number. There may be multiple DLCIs sharing the same interface number
This is the device configuration:
- dlciName.1.101 = ‘dlciA’
- dlciName.1.105 = ‘dlciB’
- dlciName.2.107 = ‘dlciC’
There is a question on type definition which needs to be examined. The choice is between using one single input variable, which will have no special type and will contain both interface number and DLCI number, or using 2 separate inputs variables, both typed as integer. The first will contain variations on the interface number. The second will contain variations on the DLCI number.
It is always more efficient and more predictable to have well defined typed variables. This ensures the correct key matching in the response matching mechanism, and this allows dimension operations (see corresponding chapter).
Thus, the formula will contain explicit type definition: I1 as integer, and I2 as integer
Using V1 = OIDVAL ( dlciName.%I1.%I2) will produce the following result in memory.
V1 =
I1 I2 DLCI Name
1 101 dlciA
1 105 dlciB
2 107 dlciC
The SNMP GetNext on the dlciName OID will return 3 valid results, corresponding to the device configuration. The formula response matching mechanism will detect that responses are valid for values of I1 in the {1, 2) set and for values of I2 in the {101, 105, 107} set. This mechanism will then build 3 key values and store 3 associated result values { dlciA, dlciB, dlciC } in the V1 variable.
The stored variable will have 3 keys:
1 (I1 = 1; I2=101)
2 (I1 = 1; I2=105)
3 (I1 = 2; I2=107)
Stored values in V1 will be:
V1 [I1 = (int) 1] ; [I2 = (int) 101] → (string) dlciA
[I1 = (int) 1] ; [I2 = (int) 105] → (string) dlciB
[I1 = (int) 2] ; [I2 = (int) 107] → (string) dlciC
Based on the number of required dimensions to build a unique key, it’s required to provide 2 replacement values (I1 and I2) to access a single result row.
3.2.2.2 TCP Connection table
There are MIB tables with more than 2 dimensions. The tcpConnTable contains all TCP connections on the current machine. A TCP connection is identified by the source IP address and port number, and the destination IP address and port number. For each connection the agent returns information about the connection “state”.
Using the same mechanism as described for the DLCI table, it’s necessary to type each input variables: I1 as IP address (source), I2 as integer (source port number), I3 as IP address (destination), I4 as integer (destination port number)
Using V1 = OIDVAL ( tcpConnState.%I1.%I2.%I3.%I4 ) will produce the following result in memory.
V1 [I1 = (IP) 127.0.0.1]; [I2 = (int) 1056] ; [I3 = (IP) 127.0.0.1] ; [I4 = (int) 16354] → (string) closed
[I1 = (IP) 127.0.0.1]; [I2 = (int) 1089] ; [I3 = (IP) 127.0.0.1] ; [I4 = (int) 18796] → (string) closed
[I1 = (IP) 127.0.0.1]; [I2 = (int) 2456] ; [I3 = (IP) 127.0.0.1] ; [I4 = (int) 21547] → (string) listen
[I1 = (IP) 127.0.0.1]; [I2 = (int) 2874] ; [I3 = (IP) 127.0.0.1] ; [I4 = (int) 29874] → (string) listen
[I1 = (IP) 192.168.64.29]; [I2 = (int) 3002]; [I3 = (IP) 192.168.64.31]; [I4 = (int) 31245] → (string) listen
[I1 = (IP) 192.168.64.29]; [I2 = (int) 3008]; [I3 = (IP) 192.168.64.31]; [I4 = (int) 38975] → (string) listen
3.3 Temporary variables
(Vx variables) - where x is an integer starting at 1 or 01 or 001… etc. It’s not mandatory to use temporary variables in a formula. In fact, it’s recommended not to use temporary variables in a collection formula, since collection formulas with temporary variables are considered as ‘complex’ and perform poorly in comparison with ‘lite’ formulas.
If V1 is the starting variable, then a total of 9 variables are available for use – V1 through V9. If V01 is the starting variable, then a total of 99 variables are available for use – V01 through V99. If V001 is the starting variable, then 999 variables are available for use – V001 through V999. This rule can continue if more than 999 variables are required.
Example 1: (complex)
V1=OIDVAL(sysName.0,once);
V2=OIDVAL(sysLocation.0,once);
%V1 index "
Example 2: (lite)
Dim I1 AS Integer Default * NAME Interface;
# 800 is 8 for octets to bits times 100 for division by seconds instead of deciseconds;
800 * delta(ifInOctets.%I1) / delta(sysUpTime.0)
It’s possible to store the results of an expression into a variable - V1, and then use the result from V1 in a second expression.
Lets imagine the expression V1 = OIDVAL ( oidA.%I1 ), which returns the following results…
V1 [I1 = (int) 1] → (int) 1001
[I1 = (int) 2] → (int) 1002
[I1 = (int) 3] → (int) 1003
A second expression could then use V1, as in V2 = OIDVAL ( oidB.%V1 ). All possible values of V1 will be used to build the query for oidB.
The second expression would return the following results…
V2 [V1 = (int) 1001] → (int) 1000001
[V1 = (int) 1002] → (int) 1000002
[V1 = (int) 1003] → (int) 1000003
4 Operators
4.1 Mathematical conventions
The formula language uses standard mathematical conventions and priority rules between operators. Parenthesis can be inserted to force priority, or to enforce priorities for readability purposes. There is no performance cost in adding a lot of parenthesis.
4.2 Binary operators
4.2.1 Mathematical operators
Symbol Definition Comments
+ Addition When used between numbers, this is the mathematical addition. When used between strings, this is the concatenation operator.
- Subtraction, or negation The ‘–‘ symbol can either be used as a subtraction operator, or in front of an expression to negate the value of the expression.
* Multiplication
/ Division
% Modulo Returns the remainder - Example: 155 % 10 5
4.2.2 Boolean operators
The result of a Boolean operation is an integer value: 0 for False, 1 for True. An integer value is considered as True if it is greater than or less than 0, and False if the value is equal to 0.
Symbol Definition Comments
> Greater than
>= Greater or equal
< Less than
<= Less or equal
== Equal
!= Not equal
Like Like This is a string comparison tool. Right argument is a string that could contain wildcards characters.
* is used to replace 0 or more characters,
? is used to replace one character.
&& And
|| Or
4.3 Dimensional operators
Operations between variables with the same or different dimensions are permitted in the SNMP language.
4.3.1 One to one dimension operations
Same dimensions, Same sizes
ifInOctets.%I1 + ifOutOctets.%I1
ifInOctets [I1 = (int) 1] → (int) 0
[i1 = (int) 2] → (int) 102145
[i1 = (int) 3] → (int) 14
ifOutOctets [I1 = (int) 1] → (int) 0
[I1 = (int) 2] → (int) 254784
[I1 = (int) 3] → (int) 1254
The computed result will be…
Operator + ( ) [I1 = (int) 1] → (int) 0 + (int) 0 → (int) 0
[I1 = (int) 2] → (int) 102145 + (int) 254784 → (int) 356929
[I1 = (int) 3] → (int) 14 + (int) 1254 → (int) 1268
The result may be stored to a temporary variable by using:
V1 = OIDVAL ( ifInOctets.%I1 + ifOutOctets.%I1 )
V1 [I1 = (int) 1] → (int) 0
[I1 = (int) 2] → (int) 356929
[I1 = (int) 3] → (int) 1268
Same dimensions, Different sizes
Let’s consider 2 result sets from an SNMP query from 2 different tables.
packetCount [I1 = (int) 1] → (int) 0
[I1 = (int) 2] → (int) 0
[I1 = (int) 3] → (int) 256
[i1 = (int) 4] → (int) 10
[i1 = (int) 5] → (int) 14
packetSize [I1 = (int) 1] → (int) 512
[I1 = (int) 3] → (int) 1024
[I1 = (int) 4] → (int) 4096
The dimension sets returned are 2 vectors of [I1]. The possible dimension intersection points are: I1 in { 1,2,3,4,5 }. This is because both packetCount and packetSize have interfaces ranging from 1 to 5.
Operator * ( ) [I1 = (int) 1] → (int) 0 * (int) 512 → (int) 0
[I1 = (int) 2] → (int) 0 * #NoValue
[I1 = (int) 3] → (int) 256 * (int) 1024 → (int) 262144
[I1 = (int) 4] → (int) 10 * (int) 4096 → (int) 40960
[I1 = (int) 5] → (int) 14 * #NoValue
Notice that for interface 2 and interface 5 the multiply operation is not possible because the packetSize is not defined for key [I1=2] and key [I1=5]. There is no way of storing “#NoValue” as a valid formula response and those keys are removed from the result set.
The final result of the * operator is:
Operator * ( ) [I1 = (int) 1] → (int) 0
[I1 = (int) 3] → (int) 262144
[I1 = (int) 4] → (int) 40960
4.3.2 More than one dimension operations
Sometimes vector dimensions used during operations ( + - * / % ) won’t match one to one. The vector dimensions and/or sizes are different. Dimension operations will produce result sets based on intersections or matches of left and right members of dimension sets. If there is no intersection between the left and right dimension sets, then all dimensions are returned; this is because the operator was unable to “filter” and result sets.
Different dimensions, different sizes
Let’s consider 2 result sets from an SNMP query from 2 different tables. One result set is a multi-dimensional vector, while the other is has a single dimension.
Single dimension:
interfaceCapacity [I1 = (int) 1] → (int) 6000
[I1 = (int) 2] → (int) 30000
[I1 = (int) 3] → (int) 5000
Multi-dimensional:
dlciThroughput [I1 = (int) 1]; [I2 = (int) 101] → (int) 1000
[I1 = (int) 1]; [I2 = (int) 102] → (int) 2000
[I1 = (int) 2]; [I2 = (int) 103] → (int) 5000
[I1 = (int) 2]; [I2 = (int) 104] → (int) 8000
[I1 = (int) 2]; [I2 = (int) 105] → (int) 10000
Let’s consider we may want to know the percentage of interface usage for each DLCI. We know that several DLCIs may share the same interface, and that the first index of the DLCI table contains the interface number.
dlciThroughput.%I1.%I2 / interfaceCapacity.%I1 * 100
Operator / ( ) [I1 = (int) 1]; [I2 = (int) 101] → (int) 1000 / (int) 6000 → (float) 0.16
[I1 = (int) 1]; [I2 = (int) 102] → (int) 2000 / (int) 6000 → (float) 0.33
[I1 = (int) 2]; [I2 = (int) 103] → (int) 5000 / (int) 30000 → (float) 0.16
[I1 = (int) 2]; [I2 = (int) 104] → (int) 8000 / (int) 30000 → (float) 0.26
[I1 = (int) 2]; [I2 = (int) 105] → (int) 10000 / (int) 30000 → (float) 0.33
[I1 = (int) 3]; [I2 = #NoValue] →
The dlciThroughput table reports knowing about interfaces 1 and 2. The interfaceCapacity table reports knowing about interfaces 1, 2, and 3. The intersections or matches therefore only occur for interfaces 1 and 2.
5 Functions
5.1 Common functions
Symbol Definition Comments
Not( ) Negation operator. Values 0 are transformed into 1, values different that 0 are transformed into 0.
Abs( ) absolute value Example: Abs( -10 ) 10
Ln( ) log( x ) for x > 0
Log( ) log( ) log10( x ), for x > 0
Exp( ) E function Examples: Exp( 5 ) 148.41316 ;
Exp( 1 ) 2.7182818
Int( ) Convert any number to the closest lower integer.
Example: Int( 5 / 3) 1
Round( ) Returns a number rounded to the nearest multiple of significance.
Syntax: Round(number [,significance] )
number: is the value you want to round.
significance: (optional) is the multiple to which you want to round. This is defaulted to 1, which means the Number will be rounded by default to the closest integer.
For example, network traffic in bytes can be rounded to the closest value in Kb and then converted.
Round( valueInBytes, 1024 ) / 1024
or a percentage value can be rounded to the closest even number:
Round( percentageValue, 2 )
99.2 -> 100
95.1 -> 96
101.02 -> 100
Remarks: When Number is exactly between 2 possible 'rounded' values, the highest one is selected.
Round( 2.5 ) = 3
Round( -2.5 ) = -2
If number is an exact multiple of significance, no rounding occurs.
Examples:
Round(2.5, 1) Rounds 2.5 up to nearest multiple of 1 (3)
Round(-2.5, 2) Rounds -2.5 up to nearest multiple of 2 (-2)
Round(1.52, 0.1) Rounds 1.5 down to the nearest multiple of 0.1 (1.5)
Round(0.234, 0.01) Rounds 0.234 down to the nearest multiple of 0.01 (0.23)
Round(1548, 10) Rounds 1548 up to the nearest multiple of 10 (1550)
Last( ) last( x ) - returns the value of x computed during previous execution of the formula
Delta( ) delta( x ) - returns the difference between the current value of x and a previous value. If x is of type Counter and the diff is negative, then the collector assumes a counter roll over and adds MAX_INT to the value. Every other case where the diff is negative is rejected and no value is returned
Diff( ) diff( x ) - same as delta, except that there is no control of the type and the result can be either positive or negative
Sin( )
Cos( )
Tan( )
Asin( )
Acos( )
Atan( )
5.2 Conversion functions
asIp( ) This function tries to represent the argument as a clean IP address. The argument can be 4 numbers (prefixed by the number 4) or 4 ascii characters. They all will be translated to the same format: 1.2.3.4 or A.B.C.D
asMac( ) This function tries to represent the argument as a clean MAC address. The argument can be 6 numbers (with or without – a prefix of the number 6). It will be translated to the format: 0x000000000000.
AsString( ) This function tries to convert the OID path to it’s Object name. (See below example – new as of Netcool/Proviso 4.3.1)
Example of AsString():
Dim I1 AS DisplayString Default * NAME VpnVrfName;
Dim I2 AS DisplayString Default * NAME VPNDestinationNetwork;
Dim I3 AS DisplayString Default * NAME VPNRouteMask;
Dim I4 AS Integer Default * NAME IfIndex;
Dim I5 AS DisplayString Default * NAME VPNNextHop;
V1 = OIDVAL( mplsVpnVrfRouteIfIndex.%I1.%I2.%I3.%I4.%I5 );
V2 = OIDVAL( AsString( IndexAsValue ( I1, %V1 ))) ;
V3 = OIDVAL( AsIp( IndexAsValue ( I2, %V1 ))) ;
V4 = OIDVAL( AsIp( IndexAsValue ( I5, %V1 ))) ;
%V1 index "ifIndex<%V1>||- N/A|| - VpnVrfName<%V2>VPNDestinationNetwork<%V3>VPNNextHop<%V4>"
-- Results --
162.1.2.100:ifIndex<0>||-
N/A||VpnVrfName
162.1.2.100:ifIndex<4>||-
N/A||VpnVrfName
162.1.2.100:ifIndex<0>||-
N/A||VpnVrfName
162.1.2.100:ifIndex<10>||-
N/A||VpnVrfName
162.1.2.100:ifIndex<0>||-
5.3 Filtering functions
The filtering function reduce the number of lines of a result set.
Symbol Definition Comments
Filter( expression ) Keep only values that are ‘True’ Keeps only lines of the result set that have a value > 0. Usually the expression is a Boolean expression with result is 1 or 0
Distrib( expression, format string ) Convert ranges of values into other values This function performs range transformation of each result line. The main goal is to solve:
If value is between 0% and 60% -> quality = 1, between 61% and 80 -> quality = 2, over 81% quality = 3.
The syntax of the distrib in the format string has been extended to support a little more.
‘Expression’ is any combination of OIDs and operators
‘Format string’, is a list of pair: Boolean condition – associated value. Each line of the expression result set is treated independently, and all Boolean conditions are evaluated in their specified order. As soon as one condition is met for a specified line of the result set, the value is substituted with the associated value, and no more conditions are evaluated for that line. If after evaluating all conditions, no one matches, then the corresponding result line is discarded from the expression result set. This allow to use distrib( ) as a filter.
All pairs in the format string are separated by commas: ‘,’
A pair is a Boolean condition and an associated numeric value. They are separated by column: ‘:’
A pair can consist of a Boolean condition and an associated string value. They are still separated by colon: ‘:’. This makes the column character an invalid character for the string value, as well as the comma “,”. Even if this is not enforced by collector, it is unsupported configuration to mix in the same distrib() format string both numeric and string replacement. This can produce unexpected results.
Valid conditions are:
default: is always true and is usually used as last chance condition.
Valid associated numeric values are :
N : which is a number
* : which is the number that was used for the evaluation
Example 1:
Everything that is less than 5 is replaced by 0.
Everything that is more than 95 is replaced by 100.
Everything that is more than 110 is discarded.
For all others cases, the value is kept.
This is solved with: “<5:0,<95:*,<110:100”
Example 2:
Translate the ifSpeed into a unit for display purposes. The first distrib( ) expression is used to convert the value of the speed into the desired display unit. The second distrib( ) expression is used to output the correct unit string for the speed.
int( 100 * ifSpeed.%I1 / distrib( ifSpeed.%I1 ,“<1000:1,<1000000:1000,<1000000000:1000000,default:1000000000"
)) / 100
+ " "
+ distrib( ifSpeed.%I1 , "<1000:bps,<1000000:Kbps,<1000000000:Mbps,default:Gbps" ) ;
This gives back a string with rounded speed, followed by the Unit.
Example 3:
Convert a numeric value for a day of week, into a string value.
distrib( oidForDayOfWeek.%I1 , "==1:Mon,==2:Tue,==3:Wed,==4:Thu,==5:Fri,==6:Sat,==7:Sun,default:invalid)
topN( N, expression ) Keep only N higher results This function sorts all results in the result set based on their result value and only keeps the ‘N’ biggest values lines.
bottomN( N, expression ) Keep only N lower results Same as TopN( ), but keeps only the N lowest ones
FirstN(N, expression ) Keep only First N results This function only keeps the first N results of a result set. The order is the returned order from the SNMP device.
LastN(N, expression ) Keep only Last N results Same as FirstN( ), but keeps the N last results.
Examples:
V1 [I1 = 1] → (int) 40
[I1 = 2] → (int) 20
[I1 = 3] → (int) 10
[I1 = 4] → (int) 30
[I1 = 5] → (int) 50
• Filter ( %V1 > 30 )
Returns lines
I1=1 -> (int) 40
I1=5 -> (int) 50
• Distrib( %V1 , “>40:3,>20:2,default:1“)
Returns lines
I1=1 -> (int) 2
I1=2 -> (int) 1
I1=3 -> (int) 1
I1=4 -> (int) 2
I1=5 -> (int) 3
• TopN( 3 , %V1 )
Returns lines
I1=5 -> (int) 50
I1=1 -> (int) 40
I1=4 -> (int) 30
• BottomN( 2 , %V1 )
Returns lines
I1=3 -> (int) 10
I1=2 -> (int) 20
• FirstN( 2 , %V1 )
Returns lines
I1=1 -> (int) 40
I1=2 -> (int) 20
• LastN( 3 , %V1 )
Returns lines
I1=3 -> (int) 10
I1=4 -> (int) 30
I1=5 -> (int) 50
5.4 Dimensional functions
5.4.1 Aggregation (across dimensions)
These functions change the dimension set of the result. They can reduce along one or all dimensions at the same time. They perform aggregation operations on the result side.
Aggregations functions:
Sum( Dimensions , Expression ) performs the sum of all values to aggregate.
Max( Dimensions , Expression ) only keeps the biggest value of all values to aggregate.
Min( Dimensions , Expression ) only keeps the lowest value of all values to aggregate.
Ave( Dimensions , Expression ) keeps the average value of all values to aggregate.
Count( Dimensions , Expression ) keeps the number of values in the aggregations set.
Concat( Dimensions , Expression ) consider each value as a string, and concatenate all strings of the aggregations set.
The first argument is the dimension that will be used for the aggregation. It can only be one dimension or all dimensions. Aggregating using all dimensions will produce a unique result, regardless of the size of the initial Expression result set, and the result will be without dimensions, like a numerical constant.
Aggregating using one dimension will remove that dimension from the Expression result set by grouping all result lines that share the exact same dimension set and values (except for the specified dimension).
Example:
V01 [I1 = (int) 1] [I2 = (int) 1] → (int) 10
[I1 = (int) 1] [I2 = (int) 2] → (int) 20
[I1 = (int) 1] [I2 = (int) 3] → (int) 30
[I1 = (int) 2] [I2 = (int) 1] → (int) 40
[I1 = (int) 2] [I2 = (int) 2] → (int) 50
[I1 = (int) 2] [I2 = (int) 3] → (int) 60
We may want to remove all dimensions using the SUM aggregation. This will produce a single result value, without dimensions, which will be the SUM of all values.
Sum( * , %V01 )
Result set:
Result [] → (int) 210
If we want to remove dimension I1 using aggregation SUM, then the only remaining dimension will be I2. In this example, there are 3 possible values for I2 = (1, 2 or 3).
For I2=1, we will have to aggregate: [I1=1] → 10 + [I1=2] → 40 Result will be 50
For I2=2, we will have to aggregate: [I1=1] → 20 + [I1=2] → 50 Result will be 70
For I2=3, we will have to aggregate: [I1=1] → 30 + [I1=2] → 60 Result will be 90
Sum( I1 , %V01 )
Result set:
result [I2 = (int) 1] → (int) 50
[I2 = (int) 2] → (int) 70
[I2 = (int) 3] → (int) 90
We may also want, from the same sample, to remove dimension I2 using the same SUM aggregation. The only remaining dimension would be I1, with 2 possible values I1 = (1, 2).
For I1=1, we will have to aggregate: [I2=1] → 10 + [I2=2] → 20 + [I2=3] → 30 Result will be 60
For I1=2, we will have to aggregate: [I2=1] → 40 + [I2=2] → 50 + [I2=3] → 60 Result will be 150
Sum( I2 , %V01 )
Result set:
result [I1 = (int) 1] → (int) 60
[I1 = (int) 2] → (int) 150
5.4.2 Manipulation
These functions allow manipulation of the dependencies between variables.
Expand( Dimension , Expression )
AddForMissing( Expression , reference Vector [, default replacement ] )
IndexAsValue( Dimension, Expression )
5.4.2.1 Expand( Dimension , Expression )
The expand function manipulates a variable’s dimensional dependency by allowing a variable which requires one dimension to work with more than one dimension.
Example:
The discovery formula must create VLAN Sub-Elements (slot + port), and also associate the interface information in property settings. The link between the Interface table and the VLAN table is through the portIfIndex in the VLAN table; it defines an ifIndex number which is in the Interface table.
VLAN table (gray is for primary key)
slot port portIfIndex portDuplex vlanPortOperStatus vlanPortVlan …
S1 P1 1
S2 P2 2
Interface table (gray is for primary key)
ifIndex ifSpeed ifAdminStatus ifOperStatus ifName …
1
2
The formula will have the following structure:
Dim I1 as integer Default * name Slot;
Dim I2 as integer Default * name Port;
…
V02 = OIDVAL (portIfIndex.%I1.%I2 );
… (more objects from the VLAN table ) …
V07 = OIDVAL ( expand( V02, ifOperStatus.%V02 ));
… ( more objects from the Interface table ) …
%V02 index “…%V07…%V02…%I1….%I2….”
The Expand is necessary to change the dependencies of the V07 object, from V07 [ V02 ], to V07 [ I1, I2 ]
This way, the %V07 and %V02 values will have common dimension and will be allowed together in the INDEX format string.
The complete formula is:
Dim I1 AS Integer Default * NAME Slot;
Dim I2 AS Integer Default * NAME Port;
V01=OIDVAL( portName.%I1.%I2 );
V02=OIDVAL( portIfIndex.%I1.%I2 );
V12=OIDVAL( portDuplex.%I1.%I2 format clean);
V03=OIDVAL( vlanPortIslOperStatus.%I1.%I2 format clean);
V04=OIDVAL( vlanPortVlan.%I1.%I2 format clean);
V05=OIDVAL( expand( V02, int(ifSpeed.%V02 /1000000) ) );
V06=OIDVAL( expand( V02, ifAdminStatus.%V02 format clean) );
V07=OIDVAL( expand( V02, ifOperStatus.%V02 format clean) );
V08=OIDVAL( expand( V02, ifType.%V02 format clean) );
V09=OIDVAL( expand( V02, ifSpeed.%V02) );
V10=OIDVAL( expand( V02, ifName.%V02) );
V11=OIDVAL( expand( V02, vlanTrunkPortEncapsulationType.%V02 format clean) );
%V01 index "Module/Port:<%I1/%I2>||Module/Port:<%I1/%I2>||InterfaceName<%V10>PortName<%V01>TrunkingStatus<%V03>Vlan-ID<%V04>InterfaceSpeed<%V09>InterfaceType<%V08>InterfaceStatus<%V06>EncapsType<%V11>PortDuplex<%V12>||Module<%I1>Port<%I2>";
5.4.2.2 AddForMissing( Expression , reference Vector [, default replacement ] )
This function allows extending the size of a result set, and adding new result entries by using entries and dimensions from a reference vector. The only prerequisite of the function is that both the Expression and the reference Vector have exactly the same dimension set.
The result of the function is an extension of Expression. For each dimension entry that is present in the reference Vector, a new dimension entry is added to Expression using either the result in the reference vector or the default replacement.
Example 1:
Expression result set.
Expression [I1 = (int) 1] → (int) 10
[I1 = (int) 3] → (int) 30
[I1 = (int) 4] → (int) 40
Reference vector result set.
Reference vector [I1 = (int) 1] → (int) 11
[I1 = (int) 2] → (int) 21
[I1 = (int) 3] → (int) 31
[I1 = (int) 4] → (int) 41
[I1 = (int) 5] → (int) 51
[I1 = (int) 6] → (int) 61
The addForMissing operator will detect that replacement values (2, 5, and 6) were not defined in the original “Expression” result set and will create entries for them.
If no default replacement is provided, then corresponding results from the “Reference” vector will be used.
This would generate the following result:
Result [I1 = (int) 1] → (int) 10 // from Expression
[I1 = (int) 2] → (int) 21 // from reference vector
[I1 = (int) 3] → (int) 30 // from Expression
[I1 = (int) 4] → (int) 40 // from Expression
[I1 = (int) 5] → (int) 51 // from reference vector
[I1 = (int) 6] → (int) 61 // from reference vector
If a default replacement is provided, then this default value will be inserted for all new entries.
This would generate the following result:
Result [I1 = (int) 1] → (int) 10 // from Expression
[I1 = (int) 2] → (int) default value
[I1 = (int) 3] → (int) 30 // from Expression
[I1 = (int) 4] → (int) 40 // from Expression
[I1 = (int) 5] → (int) default value
[I1 = (int) 6] → (int) default value
Example 2:
This example is taken from the IETF_IF discovery formula. V104 (ifDescr) and V106 (ifName plus ifAlias) supply the input for V004, which is part of the Sub-Element Label. The AddForMissing function, which sets V004, will return V106 if it exists or V104 if the interface only supports ifDescr.
Dim I1 AS Integer Default * NAME I1;
V001 = OIDINST(ifAdminStatus.%I1 == 1);
V002 = OIDVAL(ifType.%V001 format clean);
V003 = OIDVAL(ifSpeed.%V001);
# Comment 1;
V104 = OIDVAL(ifDescr.%V001);
V105 = OIDVAL(ifName.%V001);
V106 = OIDVAL(ifName.%V001 + " " + ifAlias.%V001);
V004 = OIDVAL(AddForMissing(%V106, V104));
Example 3:
This example is taken from the IETF_IF discovery formula. Interfaces that only support the original ifTable will not respond to ifConnectorPresent. V002 is used as a reference vector to fill in V005 for such “missing” interfaces. The replacement value is “notSupported”. For interfaces which do respond, the value retrieved (“true” or “false”) is stored in V005 and returned as an ifConnectorPresent property.
V005 = OIDVAL(AddForMissing(ifConnectorPresent.%V001 format clean, V002, "notSupported"));
V006 = OIDVAL(sysLocation.0, once);
V007 = OIDVAL(sysName.0, once);
V107 = OIDVAL(ifOperStatus.%V001 format clean);
V207 = OIDVAL(ifAdminStatus.%V001 format clean);
Example 4:
This is a sample to make the point of how a discovery formula could be simplified with the use of the AddForMissing( ) operator. This sample shows the possible return values of an SNMP request based on the ifTable - ifInOctets and ifOutOctets.
Valid <-> ifInOctets >0
Null <-> ifInOctets = 0
Valid <-> ifOutOctets >0
Null <-> ifOutOctets = 0
Assuming all 4 combinations are possible, a discovery formula (not using AddForMissing technology) would be like:
V1=OIDVAL( ifIndex.%I1 );
V2=OIDVAL( distrib( ifInOctets.%I1, ">0:Valid"));
V3=OIDVAL( distrib( ifOutOctets.%I1, ">0:Valid"));
%V1 index "In<%V2>Out<%V3>";
%V1 index "In<%V2>Out
%V1 index "In
%V1 index "In
Results:
192.168.127.1 = In
192.168.127.1 = In
192.168.127.1 = In
192.168.127.1 = In
192.168.127.1 = In
This can be rewritten using AddForMissing( ) …
V1=OIDVAL( ifIndex.%I1 );
V2=OIDVAL( AddForMissing( distrib( ifInOctets.%I1, ">0:Valid") , V1 , "Null" ));
V3=OIDVAL( AddForMissing( distrib( ifOutOctets.%I1, ">0:Valid") , V1 , "Null" ));
%V1 index "In<%V2>Out<%V3>"
Same results:
192.168.127.1 = In
192.168.127.1 = In
192.168.127.1 = In
192.168.127.1 = In
192.168.127.1 = In
5.4.2.3 IndexAsValue( Dimension, Expression )
In Version 4.3.1, Netcool/Proviso introduced this new function that formats, in a human readable way, the strings and IP addresses that are part of the SNMP table indexes. For example, 4.67.65.56.68 can now be translated to ACME.
Example:
Dim I1 AS DisplayString Default * NAME VpnVrfName;
Dim I2 AS DisplayString Default * NAME VPNDestinationNetwork;
Dim I3 AS DisplayString Default * NAME VPNRouteMask;
Dim I4 AS Integer Default * NAME IfIndex;
Dim I5 AS DisplayString Default * NAME VPNNextHop;
V1 = OIDVAL( mplsVpnVrfRouteIfIndex.%I1.%I2.%I3.%I4.%I5 );
V2 = OIDVAL( AsString( IndexAsValue ( I1, %V1 ))) ;
V3 = OIDVAL( AsIp( IndexAsValue ( I2, %V1 ))) ;
V4 = OIDVAL( AsIp( IndexAsValue ( I5, %V1 ))) ;
%V1 index "ifIndex<%V1>||- N/A|| - VpnVrfName<%V2>VPNDestinationNetwork<%V3>VPNNextHop<%V4>"
-- Results --
162.1.2.100:ifIndex<0>||-
N/A||VpnVrfName
162.1.2.100:ifIndex<4>||-
N/A||VpnVrfName
162.1.2.100:ifIndex<0>||-
N/A||VpnVrfName
162.1.2.100:ifIndex<10>||-
N/A||VpnVrfName
162.1.2.100:ifIndex<0>||-
6 Common Formula structure
6.1 Comments in Formulas
To add comments to a formula:
put each comment on a line by itself
start each line with the pound sign (#)
end each line with a semi-colon (;)
# This is a comment;
NOTE: Do not include commas (,) or semi-colons (;) within the body of the comment. The SNMP collector parses the text inside the comments and may become confused by punctuation and special characters. Restrict yourself to very simple comments or none at all.
6.2 OIDVAL expression line
The purpose of the OIDVAL expression line is to execute an expression (based on an OID or statistic variable) and to store the result of this expression in a temporary variable. Dimensions of the temporary variable are a result of the dimensions of the Expression.
This declaration uses the following syntax:
Vn = OIDVAL(
where
Expression: OID or STAT variable or mathematical operations
Validity: once | every | xxP | ssss
The
For the
With all OID and STAT results, the
The result set is built.
The Vx variable is set.
Example 1:
Dim I1 as integer default 2
V1=OIDVAL(ifInOctets.%I1 / ifSpeed.%I1 * 100.0, every);
OID query results:
ifInOctets [I1 = (int) 2] → (int) 1000
ifSpeed [I1 = (int) 2] → (int) 512
1000 / 512 * 100.0 result:
result [I1 = (int) 2] → (int) 195.31
V1 = 195.31
6.3 OIDINST expression line
The purpose of the OIDINST expression line is to execute an expression which stores the value(s) of keys that have a result other than 0. This type of expression is mainly used in discovery formulas to filter a sub-set of indexes from a large set, based on properties.
This declaration uses the following syntax:
Vn = OIDINST(
where
Expression: OID or STAT variable or mathematical operations with a Boolean result
Validity: once | every | xxP | ssss
Example:
V1=OIDINST( ifStatus.%I1 like ‘up’ )
SNMP query results:
ifStatus [I1 = (int) 1] → (string) “up”
[I1 = (int) 2] → (string) “down”
[I1 = (int) 3] → (string) “up”
Boolean expression comparison results:
result [I1 = (int) 1] → (int) 1
[I1 = (int) 2] → (int) 0
[I1 = (int) 3] → (int) 1
The OIDINST stores in the V1 vector all keys with a result other than 0.
2 keys (I1=1, and I1=3) are selected.
V1 will be constructed with no dimension and multiple values.
V1 [] → { (int) 1 ; (int) 3 }
The V1 variable may be used in another expression line, exactly in the same way as an Ix variable.
This OIDINST operator can be used with a vector having more than one dimension. In this case, all substitution values are concatenated, separated by the character dot. Types of each dimension are lost, and the resulting type of V1 variables is string; this is default type when no explicit type is declared. We have already discussed that not using explicit types has more side effects than benefits. This OIDINST operator should not be used when there is more that one dimension because the individual dimensions are lost. There is more about this in the section ‘building efficient discovery formulas’.
Example: (why not to use OIDINST when more than one dimension exists)
V1=OIDINST( dlciStatus.%I1.%I2 like “up” )
dlciStatus [I1 = (int) 1]; [I2 = (int) 101] → (string) “up”
[I1 = (int) 1]; [I2 = (int) 102] → (string) “down”
[I1 = (int) 2]; [I2 = (int) 103] → (string) “up”
2 keys ( I1=1,I2=101 and I1=2,I2=103 are selected )
Since there is more than one dimension involved, keys are transformed to: 1.101 and 2.103
V1 is set to:
V1 [] → { (string) “1.101” ; (string) “2.103” }
On next substitution, using V1 values, we will lose the
6.4 Dim section
Syntax: Dim {input number} AS {type} Default {default value} Name {mnemonic name};
This section declares types and default values for input variables of the formula. It is mandatory to declare the type of an input value, if that type is different from Integer.
Valid types are:
Integer
IPAddress
Gauge
Counter
MacAddress
OctetString
DisplayString
Timeticks
These names are case insensitive when used in a formula.
Default instances can be a list of values, or * to indicate that all instances should be used. Spaces are not allowed within a list of values for an input variable.
These are valid:
Integer 3
list of integers 1,3,6,9,8,109
interval 4-59
list of integers and intervals 1,3,6,9,8-96,109,200-250
character string “this is a STRING”
pointer to a string 128.3.56.7
wildcard *
Example:
Dim I1 AS Integer Default * Name TunIndex;
Dim I2 AS Integer Default * Name TunInstance;
Dim I3 AS Integer Default * Name LsrInID;
Dim I4 AS Integer Default * Name LsrOutID;
Will match Sub-Element instances: TunIndex<1000> TunInstance<123> LsrInID<456> LsrOutID<789>
6.5 Def section
6.5.1 Def SaveAlias
Syntax: Def SaveAlias {metricId};
This is used to specify an alternative storage metric ID, called a Generic Metric ID. The formula itself is stored in the database with a metric ID called a Specific Metric ID. There two rules the Generic ID must follow.
1.) The Generic Metric ID must exist in the formula table
2.) The Generic Metric ID must be of the same DataType as the current ‘Specific’ formula ( float / string )
Generic formulas
Generic formulas are used to insert a hardware abstraction layer into the formula language. This allows a formula writer to have different expressions based on different hardware capabilities. The Specific formulas handle the hardware specific capabilities. To merge all results into one unique formula the Generic Metric is used.
Example
The Specific/Generic mechanism is based on formula IDs. If nothing is specified in the formula definition, a formula saves its results to a unique ID. It’s possible to force a formula redirect all of the results to a Generic ID.
Cisco Utilization % defined using the Cisco private MIB
Juniper Utilization % defined using the Juniper private MIB
Id 100 Utilization % #Generic
Id 201 Cisco Utilization % SaveAlias = 100
Id 301 Juniper Utilization % SaveAlias = 100
Both formula results are rolled up into a generic formula called: Utilization %
6.5.2 Def DefaultResult
Syntax: Def DefaultResult {a number};
This is used to specify a number that will be returned by a formula. This is useful if either no SNMP data was available or a mathematical error occurred.
6.5.2.1 Def DefaultNoRespResult
Syntax: Def DefaultNoRespResult {a number};
This is used to specify a number that will be returned by a formula. This is useful if no SNMP data (at all) can be retrieved from the agent.
NOTE: If partial data is retrieved or a mathematical error occurs, the value specified by DefaultNoRespResult will not be used. The DefaultResult value will be used.
6.5.3 Def UseQuotedStrings
Syntax: Def UseQuotedStrings {yes|no};
This flag sets the default behavior of the collector for displaying strings. If UseQuotedStrings is set to yes (True), which is the default value, then all strings will be displayed within a pair of double quotes (”). If UseQuotedStrings is set to no (False), then strings will not be delimited by double quotes. It is always possible, inside the INDEX format string to add explicit double quotes by escaping them with backslash.
Example:
ERX VR element discovery formula to handle the VR community string:
Dim I1 as Integer default *;
Def UseQuotedStrings no;
V1=OIDINST(usdRouterRowStatus.%I1 > 0);
V2=OIDVAL(usdRouterName.%V1);
V3=OIDVAL(sysName.0 );
V4=OIDVAL(sysDescr.0 );
%V2 index "NULL||%HOSTNAME||ipAddress<%HOSTIP>IP.name<%HOSTNAME_%V2>rCommunity<%USEDRCOMMUNITY@%V2>sysDescr<%V4>physAddress<%HOSTNAME:%V2>sysName<%V3:%V2>||%V3/%V2";
6.5.4 Def MaxLines
Syntax: Def MaxLines {a number};
This is used to limit the number of SNMP GetNext requests when browsing a table. This is especially useful when only the first responding OID is important.
Example:
RFC2233_IF_match formula:
Def MaxLines 1;
V1=OIDVAL(count(*,ifInBroadcastPkts.*));
V2=OIDVAL(count(*,ifInMulticastPkts.*));
V3=OIDVAL(count(*,ifCounterDiscontinuityTime.*));
%V1+%V2+%V3;
6.6 Result Expression line
6.6.1 Definition
The Result line(s) in a formula is/are usually located at the end of the formula, after all variables assignments. A result line defines the format of the information that will be sent back to the calling application. Execution of a result line in a formula will result in the creation in memory of one (or more) value line(s). Each line contains the name of the device that has been queried, a result value (number or string), and an instance number for the result (either 0 if the result is not indexed, or a number corresponding to values of dimensions of that result).
6.6.2 Collection
In the case of a collection formula, applied to a Sub-Element, the expected format for a result is a single value. Therefore, it is invalid for a collection formula to specify more than one result line, and it is also invalid for that result line to generate more than one line of result.
6.6.3 Discovery
In the case of a discovery formula, applied to an Element, the expected format is a long string composed of 4 fields separated by double pipe ‘||’. Each field of that string represents a part of a Sub-Element being discovered. It is possible for a discovery formula to specify several results lines.
6.6.4 Default return form
By default the format of the instance string is deducted by the collector from the list of dimensions contained in the result vector. If the result is Res [ I1 , I2 ], then a string “%I1.%I2” will be used for constructing the instance field (usually order is: Ixx variables ordered by increasing numbers, then Vxx variables in increasing number order)
Collection
The returned instance is used for standard collections, for matching the result with the instance field of the Sub-Element.
Discovery
The returned instance is used for discovery formulas, to send back the complete description of each Sub-Element. This is done by using a user defined instance string.
Exemple1:
V1=OIDVAL( ifInOctets.%I1 + ifOutOctets.%I1 );
%V1 This is the result line, which will output the content of the V1 vector.
Testing this formula gives:
Information:Expression: compiling 'V1=OIDVAL( ifInOctets.%I1 + ifOutOctets.%I1 )' ...
Information:Expression: compiling '%V1' ...
Information:Running Formula ...
Information:Executing: OIDVAL( ifInOctets.%I1 + ifOutOctets.%I1 ) ...
Debug:OIDVAL( ): list of all SNMP values returned ...
Debug:192.168.64.29:ifInOctets.1:0
Debug:192.168.64.29:ifOutOctets.1:0
Debug:192.168.64.29:ifInOctets.2:2329703221
Debug:192.168.64.29:ifOutOctets.2:1621920323
Information:Partial result for var V1 ...
Information:V1( [H1=192.168.64.29] [I1=1] ) = [float] 0
Information:V1( [H1=192.168.64.29] [I1=2] ) = [float] 3951623544
Information:Executing: Expression %V1 ...
Debug:Expression: list of all SNMP values returned ...
Debug:Merging results into current result set (init size = 0) with index mask ='%I1'
Debug:[1] 0
Debug:[H1=192.168.64.29] [I1=1] ==> result at index '1'
Debug:[2] 3951623544
Debug:[H1=192.168.64.29] [I1=2] ==> result at index '2'
192.168.64.29 = 1:0
192.168.64.29 = 2:3951623544
The formula has been tested against an element, without specifying a specific instance number. Therefore, all instances available, in this case 1 and 2, have been collected.
The number of value lines returned by the execution of that formula, is driven by the size of the vector V1. This vector has 2 entries in his dictionary.
Interface 1
Interface 2
Therefore 2 value lines are produced. The index format for the result lines is automatically produced by parsing the dependencies of the value/vector that is used to drive the results. The H1 part is ignored since multi hosts formula are not supported anymore.
On the previous example, it has been determined that the index format should be expressed using variations of %I1 only. If the V1 vector was defined using more than one dimension (I1 and I2), then the index format would have been “%I1.%I2”.
It is possible for the user to control the index format string by using the keyword “index”. This is covered more in the section ‘User defined returned form’.
Results are displayed as:
{Device name} = {instance}: {value}
192.168.64.29 = 1:0
192.168.64.29 = 2:3951623544
6.6.5 User defined return form
This default string can be changed by using “INDEX format string”. Where “format string” is a free text string containing %Ixx, and/or %Vxx every time a substitution is needed.
Example:
ifNumber.%I1 index “Number=%I1” will produce results:
ReqId1: Number=1: 1
ReqId1: Number=2: 2
The default output (without INDEX) would have been:
ReqId1: 1: 1
ReqId1: 2: 2
The format string can be any combination of free text and variable substitutions. Each variable substitution can appear multiple times in the same format string.
It’s only required that both the returned vector (obtained from the expression) and the format string use common dimensions and variables. If the collector is not able to match dimensions for each result line with a corresponding dimension value during substitution, then the defined index string will fail.
Example 1: (valid)
Assuming the expression:
ifSpeed.%I1 index “Interface<%I1>”
Res [I1 = (int) 1] → (int) 6000
[I1 = (int) 2] → (int) 30000
[I1 = (int) 3] → (int) 5000
This will produce result set:
ReqId:Interface<1>:6000
ReqId:Interface<2>:30000
ReqId:Interface<3>:5000
Example 2: (invalid)
ifSpeed.%I1 index “Interface<%I1>Name<%I2>”
This will be rejected because I1 is the only dimension of the result vector, and there is no way to get a corresponding value for I2 by only knowing I1.
Example 3: (valid)
V1=OIDVAL( ifName.%I1 ) ;
ifSpeed.%I1 index “Interface<%I1>Speed<%V1>”
This will work and produce result lines containing both the interface number and the speed. This is because knowing I1, V1 [ I1 ] gives back a unique result, that can be used for substitution.
Example 4:
V1=OIDVAL( ifDescr.%I1 ) ;
V2=OIDVAL( ifSpeed.%I1 );
%V2 index “If %I1, Descr %V1”
Testing this formula gives:
Information:Expression: compiling 'V1=OIDVAL( ifDescr.%I1 )' ...
Information:Expression: compiling 'V2=OIDVAL( ifSpeed.%I1 )' ...
Information:Expression: compiling '%V2 index "If %I1, Descr %V1"' ...
Information:Running Formula ...
Information:Executing: OIDVAL( ifDescr.%I1 ) ...
Debug:OIDVAL( ): list of all SNMP values returned ...
Debug:192.168.64.29:ifDescr.1:"lo0"
Debug:192.168.64.29:ifDescr.2:"hme0"
Information:Partial result for var V1 ...
Information:V1( [H1=192.168.64.29] [I1=1] ) = [float] "lo0"
Information:V1( [H1=192.168.64.29] [I1=2] ) = [float] "hme0"
Information:Executing: OIDVAL( ifSpeed.%I1 ) ...
Debug:OIDVAL( ): list of all SNMP values returned ...
Debug:192.168.64.29:ifSpeed.1:10000000
Debug:192.168.64.29:ifSpeed.2:100000000
Information:Partial result for var V2 ...
Information:V2( [H1=192.168.64.29] [I1=1] ) = [float] 10000000
Information:V2( [H1=192.168.64.29] [I1=2] ) = [float] 100000000
Information:Executing: Expression %V2 index "If %I1, Descr %V1" ...
Debug:Expression: list of all SNMP values returned ...
Debug:Merging results into current result set (init size = 0) with index mask ='%I1'
Debug:[H1=192.168.64.29] [I1=1] ==> result at index '1' = If 1, Descr "lo0"
Debug:[H1=192.168.64.29] [I1=2] ==> result at index '2' = If 2, Descr "hme0"
192.168.64.29 = If 1, Descr "lo0":10000000
192.168.64.29 = If 2, Descr "hme0":100000000
Results:
{Device name} = {instance}: {value}
192.168.64.29 = If 1, Descr "lo0":10000000
192.168.64.29 = If 2, Descr "hme0":100000000
This feature of redefining the format of the instance field is used by discovery formulas. This method allows a discovery formula to build the Sub-Element definition string (4 specific fields).
In the case of a discovery formula, the value produced by the formula is irrelevant, only the instance of that value is used.
7 Writing discovery formulas
The 4.x discovery implementation consist of several steps.
• At discovery time “capabilities” of each Sub-Element are detected and stored into specific properties (either one property name for each capability, or a single property containing a list of capabilities, or a mix).
• Specific grouping rules are applied, inside the Sub-Element collection tree, in order to group Sub-Elements that have common capabilities together.
• Requests (a.k.a Collection) are deployed against part or all of the created groups. Formulas used in each request use OIDs that have previously been validated during the discovery phase. For example, a formula that computes the traffic, using 64 bits counters, will be deployed only against a sub group that contains interfaces that have 64 bit counters. Regular traffic formula that use 32 bit counters should be deployed against sub group with 32 bit interfaces only. Because, at discovery time, a capability is either 64 or 32, a Sub-Element will only end up in one collection group.
• If properties/capabilities of an interface change over time, another discovery run will catch the change and update the property. This change will force a re-grouping of that Sub-Element to place it into the “new” group. The collection formula will then move to the proper collection group when the collector does a “resync” of its model.
7.1 Structure of a discovery formula
7.1.1 Dim section
Syntax: Dim {input number} AS {type} Default {default value} Name {mnemonic name};
This section declares types and default values for input variables of the formula. It is mandatory to declare the type of an input value, if that type is different from Integer.
Valid types are:
Integer
IPAddress
Gauge
Counter
MacAddress
OctetString
DisplayString
Timeticks
These names are case insensitive when used in a formula.
Default instances can be a list of values, or * to indicate that all instances should be used. Spaces are not allowed within a list of values for an input variable.
These are valid:
Integer 3
list of integers 1,3,6,9,8,109
interval 4-59
list of integers and intervals 1,3,6,9,8-96,109,200-250
character string “this is a STRING”
pointer to a string 128.3.56.7
wildcard *
Example:
Dim I1 AS Integer Default * Name TunIndex;
Dim I2 AS Integer Default * Name TunInstance;
Dim I3 AS Integer Default * Name LsrInID;
Dim I4 AS Integer Default * Name LsrOutID;
Will match Sub-Element instances: TunIndex<1000> TunInstance<123> LsrInID<456> LsrOutID<789>
7.1.2 Def section
7.1.2.1 Def MaxLines
Syntax: Def MaxLines {a number};
This is used to limit the number of SNMP GetNext requests when browsing a table. This is especially useful when only the first responding OID is important.
Example:
RFC2233_IF_match formula:
Def MaxLines 1;
V1=OIDVAL(count(*,ifInBroadcastPkts.*));
V2=OIDVAL(count(*,ifInMulticastPkts.*));
V3=OIDVAL(count(*,ifCounterDiscontinuityTime.*));
%V1+%V2+%V3;
7.1.2.2 Def UseQuotedStrings
Syntax: Def UseQuotedStrings {yes|no};
This flag sets the default behavior of the collector for displaying strings. If UseQuotedStrings is set to yes (True), which is the default value, then all strings will be displayed within a pair of double quotes (”). If UseQuotedStrings is set to no (False), then strings will not be delimited by double quotes. It is always possible, inside the INDEX format string to add explicit double quotes by escaping them with backslash.
Example:
ERX VR element discovery formula to handle the VR community string:
Dim I1 as Integer default *;
Def UseQuotedStrings no;
V1=OIDINST(usdRouterRowStatus.%I1 > 0);
V2=OIDVAL(usdRouterName.%V1);
V3=OIDVAL(sysName.0 );
V4=OIDVAL(sysDescr.0 );
%V2 index "NULL||%HOSTNAME||ipAddress<%HOSTIP>IP.name<%HOSTNAME_%V2>rCommunity<%USEDRCOMMUNITY@%V2>sysDescr<%V4>physAddress<%HOSTNAME:%V2>sysName<%V3:%V2>||%V3/%V2";
7.1.3 OID gathering section
This section explains the retrieval of SNMP values on a device and the storing of the results into a temporary variable (vectors).
7.1.3.1 Storing raw OID in a temporary vector
V1 = OIDVAL ( ifDescr.%I1 )
Will save in memory:
V1( [H1=192.168.64.29] [I1=1] ) = [float] "lo0"
V1( [H1=192.168.64.29] [I1=2] ) = [float] "hme0"
7.1.3.2 Storing result of an expression in a temporary vector
V2 = OIDVAL ( ifSpeed.%I1 / 1E6 )
Will save in memory:
V2( [H1=192.168.64.29] [I1=1] ) = [float] 10
V2( [H1=192.168.64.29] [I1=2] ) = [float] 100
7.1.3.3 Getting list of indexes that satisfy a condition
V3 = OIDINST ( ifOperStatus.%I1 == “up” format clean );
Will save in memory a vector without dimensions.
V3( ) = [float] 1
V3( ) = [float] 2
7.1.3.4 Use another vector’s value(s) to drive a collection
V4 = OIDVAL ( ifDescr.%V3 )
Will save in memory:
V4( [H1=192.168.64.29] [V3=1] ) = [float] "lo0"
V4( [H1=192.168.64.29] [V3=2] ) = [float] "hme0"
For additional information see the section on performance improvements.
7.1.4 Result line section
This section explains the use of the result line, with a custom INDEX format string, to define characteristics of a Sub-Element to add to the database.
NOTE: If multiple lines are used, then priority rules apply (top to bottom).
Lines are on the form:
%V1 index “…%Vxx… || …%Vxx…||…%Vxx…||…%Vxx… “
The %V1 is the variable that will drive the size of the result set. The collector will loop through all of the values of the variable %V1, and will substitute every time it is possible, all Vxx variables used in the index line with their corresponding values. If Vxx variable has no appropriate value, then no result is produced and the current result of the variable V1 is omitted.
7.1.4.1 Field Definitions
Fields are the following:
1st field contains the Sub-Element instance string. This is the string that will be transmitted to each collection formula when that Sub-Element will be deployed for collection.
2nd field contains the custom label of the Sub-Element.
3rd field contains the property list for the Sub-Element. This is a list of pairs {property name}<{property value}>
4th field (optional) contains (when available) the invariant of the Sub-Element. The invariant is used during the synchronization process, to detect a change of element properties, including the instance string (1st field).
All fields are separated by ‘||’ tokens.
Example:
1213_IF formula for discovering 1213 interface Sub-Elements:
Dim I1 AS Integer Default * NAME If;
V1 = OIDVAL(ifType.%I1 format clean, once);
V2 = OIDVAL(sysLocation.0, once);
V3 = OIDVAL(ifSpeed.%I1, once);
V4 = OIDVAL(int(%V3/1000000), once);
V5 = OIDVAL(ifAdminStatus.%I1 format clean, once);
%V1 index "If<%I1>||IF: %I1 (%V4Mbps)||type<%V1>location<%V2>physicalCapacity<%V3>status<%V5>siteID
This produces the following Sub-Elements, assuming 2 interfaces:
Subelement #1
Host: 192.168.64.29
Instance If<1>
Label IF: 1 (10Mbps)
Properties typesoftwareLoopback
location"System administrators office"
physicalCapacity10000000
statusup
siteIDIP:192.168.64.29ifIndex:1
Invariant: {empty}
Subelement #2
Host: 192.168.64.29
Instance If<2>
Label IF: 2 (100Mbps)
Properties typeethernetCsmacd
location"System administrators office"
physicalCapacity100000000
statusup
siteIDIP:192.168.64.29ifIndex:2
Invariant: {empty}
7.1.4.2 Use cases and implementation
1.) Discover a small section of a large MIB table
Discovering a small section of a large MIB table is required for performance reasons. There is no sense in querying the entire table when the number of desired results in the table is considerably smaller than the size of the table. The way this is handled is by doing a single scan of the table, extracting all indexes that satisfy a condition, and later on use the list of indexes to directly access others OIDs in the table.
Example:
For Juniper ERX pack only interfaces of type DS1 are considered. IfIndex values corresponding to valid DS1 interfaces are stored in V1, and reused everywhere in the formula. This avoids scanning the huge ifTable for every OID.
Dim I1 AS Integer Default * NAME Interface;
V1 = OIDINST(dsx1IfIndex.%I1);
V2 = OIDVAL(ifDescr.%V1);
V3 = OIDVAL(ifSpeed.%V1);
V4 = OIDVAL(ifType.%V1 format clean);
V5 = OIDVAL(usdIfType.%V1 format clean);
V6 = OIDVAL(ifAlias.%V1);
%V3 index "Interface<%V1>||Name %V2 Index %V1||physicalCapacity<%V3>type<%V4>usdIfType<%V5>ifAlias<%V6>||";
2.) Browsing a table which requires multiple indexes
This is from the CISCO_MPLS_LSP formula. The table requires 4 index numbers.
Example:
Dim I1 AS Integer Default * Name TunIndex;
Dim I2 AS Integer Default * Name TunInstance;
Dim I3 AS Integer Default * Name LsrInID;
Dim I4 AS Integer Default * Name LsrOutID;
V1=OIDVAL(sysName.0,once);
V2=OIDVAL(sysLocation.0,once);
V3=OIDVAL(mplsTunnelName.%I1.%I2.%I3.%I4);
V4=OIDVAL(mplsTunnelDescr.%I1.%I2.%I3.%I4);
V5=OIDVAL(mplsTunnelIfIndex.%I1.%I2.%I3.%I4);
%V3 index "TunIndex<%I1>TunInstance<%I2>LsrInID<%I3>LsrOutID<%I4>||Cisco LSP: %V3||hostName<%V1>location<%V2>name<%V3>description<%V4>siteID
3.) Generate default values for OIDs that don’t respond (property detection)
This scenario explains the use of a formula to discover interfaces with varying speeds (64, 32 or 0) for both inbound and outbound traffic. There are 9 possibilities and thus 9 result lines to describe each combination.
Example:
Dim I1 AS Integer Default * NAME If;
V1 = OIDVAL( ifIndex.%I1 );
V2 = OIDVAL( filter( ifHCInOctets.%I1 >0) );
V3 = OIDVAL( filter( ifInOctets.%I1 >0) );
V5 = OIDVAL( filter( ifHCOutOctets.%I1 > 0 ) );
V6 = OIDVAL( filter( ifOutOctets.%I1 > 0) );
%V2+%V5 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V2+%V6 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V3+%V5 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V3+%V6 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V2 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V3 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V5 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V6 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V1 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
Rewrite of the formula using AddForMissing( ) feature:
Dim I1 AS Integer Default * NAME If;
V1 = OIDVAL( ifIndex.%I1 );
V2 = OIDVAL( distrib( ifHCInOctets.%I1, ">0:64" )); -> support Inbound 64;
V3 = OIDVAL( distrib( ifInOctets.%I1, ">0:32" )); -> support Inbound 32;
V4 = OIDVAL( AddForMissing( AddForMissing( %V2, V3 ), V1, 0 )); -> Inbound speed;
V5 = OIDVAL( distrib( ifHCOutOctets.%I1, ">0:64" )); -> support Outbound 64;
V6 = OIDVAL( distrib( ifOutOctets.%I1, ">0:32" )); -> support Outbound 32;
V7 = OIDVAL( AddForMissing( AddForMissing( %V5, V6 ), V1, 0 )); -> Outbound speed;
%V1 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
7.2 Merging
History
The current formula language stores OID values into variables. There are two possible results returned from an OID query.
Response Received a response has been provided for a given interface index and a property has to be filled with the value.
No Response Received no response has been provided for a given interface index and a property has to be filled with a default value.
If each OID query required 2 result lines, this would consume a lot of space. A formula body size is limited to 4096 characters; this is based on a database storage setting. If 2n required result lines takes more than 4096 bytes, the formula can not be saved. Even under the 4KB limit, a formula with many lines is difficult to read and debug.
Solution
A large formula could be broken into several smaller formulas. The main or top-level formula would discover the Sub-Element and some basic properties, and a series of ‘expansion’ formulas would define and populate additional properties. There is no limit to the number of ‘expansion’ formulas allowed. The goal would be to create a unique Sub-Element as a result of the execution of several discovery formulas.
This will simplify the formula writing by allowing the splitting by technology. An IF using rfc1213, an IF using rfc2233 ... all contributing to create more/less a complete IF Sub-Element. Several discovery formulas will run sequentially and the result will be a 'merged' Sub-Element with all its required properties.
Application
To accomplish this property merge configuration, a common 'key' is required. The ‘key’ will be a pair: Element Name and Sub-Element instance field. This is the unique ‘key’ in the database. If the execution of a discovery formula produces a Sub-Element for which the ‘key’ already exists, then all of the fields of that Sub-Element will be merged with the existing ones. Additionally, logic needs to be put in to place for values that already exist as a result of a previous execution of the discovery formula, and for which a new value is provided by the latest discovery.
The merge will be applied on all 3 fields: Sub-Element Label, Properties, and Invariant. Properties will be treated one by one, and key-ed by their name.
A ‘Base’ formula will be used to refer to the first discovery formula that defines/creates a Sub-Element and some base properties.
An ‘Expansion’ formula will be used to refer to all other formulas that enrich an already existing Sub-Element.
Merge rules
1.) When a formula defines a value for a Sub-Element Label, Property, Invariant that is not defined, the value is immediately assigned to the Sub-Element Label, Property, Invariant of that Sub-Element.
2.) When a formula defines a value for a Sub-Element Label, Property, Invariant that is already defined, then the merge process is used.
At the beginning of the fields Sub-Element Label, Property, Invariant, a tag or magic character will indicate merging preferences. The absence of a tag will result in default merging behavior; this assumes the + behavior. The tag will be removed for file output.
3 magic characters are required ( *, +, - ):
* the value will overwrite the previous content, no matter what.
+ the value will only overwrite previous content if previous content was either missing or empty.
- the value will only overwrite previous content if previous content was missing.
This is summarized by the following table:
* Current + Current (2) - Current
Previous is missing Current Current (2) Current
Previous is empty Current Current (2) Previous
Previous is defined (1) Current Previous (2) Previous
(1) ‘Defined’ meaning: exists and content is not empty.
(2) This represents the default behavior if 'magic character' is missing.
NOTE: The absence of a tag will result in default merging behavior; this assumes the + behavior. A Sub-Element Label, Property, and Invariant can not start with (*, + or -).
3.) Inventory_subelement.txt file
- All ‘Expansion’ formulas should use the same family name, as their ‘Base’ formula. This is not a requirement for the feature, but this helps understanding of the inventory_subelement.txt file.
- If a match formula is used, for a ‘Base’ formula, all ‘Expansion’ formulas associated, should also use the same matching rules.
- No ‘Expansion’ formula should define family to cancel. Only the ‘Base’ formula should do so. This is not a requirement, but this improves readability of the inventory_subelement.txt file.
4.) SubElement Label field
- The general rule is that no ‘Expansion’ formula should rewrite the original Sub-Element Label name provided by the ‘Base’ formula. However, the ‘Expansion’ formula needs to define the Sub-Element Label field (the second one) because it’s mandatory in the formula language syntax. Therefore, the Sub-Element Label field should contain a word prefixed by -, to indicate that a ‘rewrite’ is not selected. It is suggested to use “-UNSPECIFIED_LABEL” so that in case there is a problem, and the creation of a Sub-Element uses the ‘Expansion’ formula label, this will catch the user’s attention.
Example 1:
V1=OIDVAL(indexAsValue( I1, firstN(1, filter( not(ifPhysAddress.%I1 like "") && not(ifPhysAddress.%I1 like "6.0.0.0.0.0.0") ) ) ) );
V2=OIDVAL(concat(V1,ifPhysAddress.%V1 format clean));
V9=OIDINST(dot3StatsIndex.%I1);
V8=OIDVAL(concat(V9,firstN(1, ifPhysAddress.%V9 format clean)));
V3=OIDVAL(sysName.0 );
V4=OIDVAL(sysDescr.0 );
V5=OIDVAL(ifNumber.0);
%V2 index "NULL||%HOSTNAME||ipAddress<%HOSTIP>rCommunity<%USEDRCOMMUNITY>sysName<\"%V3\">sysDescr<\"%
V4\">ifNumber<%V5>physAddress<%V2>||\"%V3\"/%V5/%V2";
%V8 index "NULL||%HOSTNAME||ipAddress<%HOSTIP>rCommunity<%USEDRCOMMUNITY>sysName<\"%V3\">sysDescr<\"%
V4\">ifNumber<%V5>physAddress<%V8>||\"%V3\"/%V5/%V8";
The 2 result lines will be merged and will create only one answer. If the first line found a valid ifPhysAddress in the ifTable, the result will use %V2. If that fails, we return the answer found through the dot3StatsTable which is stored in V8.
Example 2:
This example shows the use of ‘Expansion’ formulas as seen in the Inventory_subelement.txt.
The following examples implement:
- Octet capability detection: inbound and outbound, 64, 32, or undefined.
- Packets capability detection: 64 or 32 or undefined
- Custom invariant, either from Mib II ifDescr, or RFC2233 ifName
- Customer specific key, and properties
Inventory_subelement.txt
#{FAMILY} |{Elem. FAMILY}|{sysObjId Mask} |{Match Formula} |{I}|{Discovery Formula} |{I}|{cancel FAMILY};[{...};]|
1213_Device |Generic~Agent |1.3.6.1 |NULL |<*>|AP~1213_Device |<*>| |
1213_IF |Generic~Agent |1.3.6.1 |NULL |<*>|StF~1213_IF |<*>| |
2233_IF |Generic~Agent |1.3.6.1 |AP~2233_IF_match |<*>|StF~2233_IF |<*>|1213_IF;|
2233_IF(Exp) |Generic~Agent |1.3.6.1 |AP~2233_IF_match |<*>|StF~2233_HCOctets_Support |<*>| |
2233_IF(Exp) |Generic~Agent |1.3.6.1 |AP~2233_IF_match |<*>|StF~2233_HCPackets_Support |<*>| |
1213_IF(Exp) |Generic~Agent |1.3.6.1 |NULL |<*>|StF~1213_IF_Invariant |<*>| |
2233_IF(Exp) |Generic~Agent |1.3.6.1 |AP~2233_IF_match |<*>|StF~2233_IF_Invariant |<*>| |
1213_IF(Exp) |Generic~Agent |1.3.6.1 |NULL |<*>|StF~CUSTOMER_KEY |<*>| |
1213_IF Discovery formula
Dim I1 AS Integer Default * NAME If;
V1 = OIDVAL(ifType.%I1 format clean, once);
V2 = OIDVAL(sysLocation.0, once);
V3 = OIDVAL(ifSpeed.%I1, once);
V4 = OIDVAL(int(%V3/1000000), once);
V5 = OIDVAL(ifAdminStatus.%I1 format clean, once);
V6 = OIDVAL(ifDescr.%I1, once);
%V1 index "If<%I1>||IF %I1 (%V4Mbps)||type<%V1>location<%V2>physicalCapacity<%V3>status<%V5>||";
2233_IF Discovery formula
Dim I1 AS Integer Default * NAME If;
V1 = OIDVAL(ifType.%I1 format clean, once);
V2 = OIDVAL(sysLocation.0, once);
V3 = OIDVAL(ifSpeed.%I1, once);
V4 = OIDVAL(int(%V3/1000000), once);
V5 = OIDVAL(ifName.%I1 format clean, once);
V6 = OIDVAL(ifAdminStatus.%I1 format clean, once);
V7 = OIDVAL(ifAlias.%I1 format clean, once);
V8 = OIDVAL(ifDescr.%I1, once);
%V1 index "If<%I1>||IF: %I1 (%V5) (%V4Mbps)||type<%V1>location<%V2>physicalCapacity<%V3>name<%V5>status<%V6>||";
2233_HCOctets_Support Discovery formula
Dim I1 AS Integer Default * NAME If;
V1 = OIDVAL( ifIndex.%I1 );
V2 = OIDVAL( filter( ifHCInOctets.%I1 >0) );
V3 = OIDVAL( filter( ifInOctets.%I1 >0) );
V5 = OIDVAL( filter( ifHCOutOctets.%I1 > 0 ) );
V6 = OIDVAL( filter( ifOutOctets.%I1 > 0) );
%V2+%V5 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V2+%V6 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V3+%V5 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V3+%V6 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V2 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V3 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V5 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V6 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
%V1 index "If<%I1>||-UNSPECIFIED_LABEL||Octets
2233_HCPackets_Support Discovery formula
Dim I1 AS Integer Default * NAME If;
V1 = OIDVAL( ifSpeed.%I1 );
V2 = OIDVAL( filter (( ifHCInUcastPkts.%I1 + ifHCInMulticastPkts.%I1 + ifHCInBroadcastPkts.%I1 ) > 0) );
V3 = OIDVAL( filter (( ifInUcastPkts.%I1+ ifOutUcastPkts.%I1 + ifInNUcastPkts.%I1 + ifOutNUcastPkts.%I1 ) > 0) );
%V2 index "If<%I1>||-UNSPECIFIED_LABEL||Packets<64>||";
%V3 index "If<%I1>||-UNSPECIFIED_LABEL||Packets<32>||";
%V1 index "If<%I1>||-UNSPECIFIED_LABEL||Packets
1213_IF_Invariant Discovery formula
# provide ifDescr and ifType as default invariant;
Dim I1 AS Integer Default * NAME If;
V1 = OIDVAL(ifType.%I1 format clean, once);
V2 = OIDVAL(ifDescr.%I1, once);
%V1 index "If<%I1>||-UNSPECIFIED_LABEL||||%V1-%V2";
2233_IF_Invariant Discovery formula
# provide invariant based on ifName when non empty;
Dim I1 AS Integer Default * NAME If;
V1 = OIDVAL( filter ( not (ifName.%I1 like "") ), once);
V2 = OIDVAL( ifName.%I1 , once);
%V1 index "If<%I1>||-UNSPECIFIED_LABEL||||*%V2";
CUSTOMER_KEY Discovery formula
Dim I1 AS IPAddress Default * NAME I1;
V01=OIDVAL( ipAdEntAddr.%I1 );
V02=OIDVAL( ipAdEntIfIndex.%I1 );
V09=OIDVAL(expand(V02, ifDescr.%V02) );
%V09 index "If<%V02>||-UNSPECIFIED_LABEL||CustomerKEY<%V09>HPKEY<%V09>IP@<%V01>||";
Example 3:
This formula is included as an example of how to create an Inventory formula which includes an invariant field. The formula is an ‘Expansion’ formula; it relies on the IETF_IF formula to discover the basic attributes of the sub-element and adds only the attributes specific to a Riverstone interface sub-element.
1 Dim I1 AS Integer Default * NAME I1;
2 V9 = OIDINST(ifAdminStatus.%I1 == 1);
3 V1 = OIDINST(ifConnectorPresent.%V9 like "true" format clean);
4 V2 = OIDVAL(ifName.%V1);
5 %V2 index "If<%V1>||-UNSPECIFIED||Invariant
Line by line explanation…
L2 This line repeats the filter on ifAdminStatus that is used in IETF_IF. This is required to ensure that Riverstone_IF does not create sub-elements for interfaces which IETF_IF filtered out. Any such sub-elements would be created without all the attributes set by IETF_IF and would not work properly. If you change the filter in IETF_IF, this filter must be changed to match.
L3 The invariant on Riverstone interfaces applies only to physical interfaces, so this line applies another filter to eliminate all logical interfaces.
L5 The Instance attribute (If<%V1>) must be set to exactly the same value as the Instance set by IETF_IF. The Label attribute (-UNSPECIFIED) is set by IETF_IF and should not be overwritten by this formula. The leading “-“ indicates that the value given here should only be used if the attribute has not been set by another formula. The value “UNSPECIFIED” is intended to make it obvious that the attribute has been set from an unexpected source. If a sub-element ever appears in the database with the Label attribute set to this value, something has gone wrong. If you want to have a more specific Label set for all Riverstone physical interfaces, you can put a different value in this field. In that case, be sure to put “*” as the first character to indicate that this value should overwrite any previous value (the “*” will be removed by the sub-element merge code).
7.2.1 Understanding the inventory_subelements.txt file
The inventory_subelements.txt file contains the list of rules the Discovery uses. It is located in the conf directory under the PVM directory.
When you create new discovery formulas, you need to edit this file to create new rules. You must edit the file before the new Discovery formula can be used by the Inventory Tool. (You do not need to edit this file before a new collection formula can be used.)
Each row (one per FAMILY) of the file is executed for every IP address that responds during Discovery. To prevent conflicting definitions for any one Sub-Element (from two or more different rows), any row can be set to override one or more other rows.
Inventory_subelement.txt excerpt and explanation:
#{FAMILY}|{Elem. FAMILY}|{sysObjId Mask}|{Match Formula}|{I}|{Discovery Formula}|{I}|{cancel FAMILY};[{...};]|
1213_Device|Generic~Agent|1.3.6.1|NULL|<*>|AP~1213_Device|<*>| |
1213_Device|Generic~Agent|1.3.6.1.4.1.9|NULL|<*>|AP~Cisco~Cisco_Device|<*>| |
FAMILY The new Sub-Element is assigned to this FAMILY.
Elem. FAMILY The Element family is defined in the inventory_element.txt. Currently there are only three element families:
Generic~Agent
Juniper~ERX
Juniper~ERX~VR.
The Element family defined in the inventory_subelement.txt must match one of the Element families defined in the inventory_element.txt.
sysObjId Mask This field is the sysObjectID. The Inventory Tool checks to see if the sysObjectID of each element matches the sysObjectID listed here. If they match, it checks to see if there is an entry for Match Formula. If the element matches both entries, the Inventory Tool runs the Match Formula specified.
Only a partial match for the sysObjectID is required. There are generic formulas included with Netcool/Proviso DataMart that specify 1.3.6.1 as the sysObjectID. This is not an exact match for anything, but it is a partial match for everything. Since only a partial match for the sysObjectID is required, these generic formulas will be used against every element encountered by the Inventory Tool. Any vendor-specific formula will contain a more specific sysObjectID that will match only that vendor’s sysObjectID.
Match Formula This field specifies the path to the formula that is used to distinguish between different elements with the same sysObjectID. This formula is no longer required in most cases; the SNMP formula language has been enhanced to handle this situation from within a single formula. ‘NULL’ is the value to enter into this field if a Match Formula is not required.
For example, devices such as the Cisco Catalyst 3000 and Catalyst 5000 would have an entry in this field. They both have the same sysObjectID; however they can have different sub-element types. Therefore, they require different discovery formulas.
For example, the Match Formula for the Cisco Catalyst 3000 is PVL~Cisco~Catalyst~C3000~Match. Its discovery formula is PVL~Cisco~Catalyst~C3000~Port_HalfDuplex_10Mb.
The Match Formula for the Cisco Catalyst 5000 is PVL~Cisco~Catalyst~C5000~Match. Its discovery formula is PVL~Cisco~Catalyst~C5000~Port_HalfDuplex_10Mb.
The path is a location within the “Alias Instance and Label Inventory” directory, with tilde (~) as the path separator. The Match Formula name must not contain any spaces, underscores are commonly used as a replacement.
Instance Instance specification used by the Match Formula. ‘<*>’ is the value to enter into this field if a Match Formula is not required.
Discovery Formula The field specifies the path to the formula used to define a Sub-Element's Name, Label, and Properties. It must retrieve all inputs on a particular Instance for a Sub-Element to be generated. In other words, if any part of the Discovery Formula fails, the Sub-Element will not be generated.
The path is a location within the “Alias Instance and Label Inventory” directory, with tilde (~) as the path separator. The Discovery Formula name must not contain any spaces, underscores are commonly used as a replacement.
NOTE: Although the Discovery Formula must succeed in order for a row to "pass" and generate a Sub-Element definition, FAMILY cancellations (see below) will execute regardless of Discovery Formula success. A Match Formula should be used to prevent unwanted cancellations from occurring.
cancel FAMILY This field is used to ensure only a single row passes for certain Sub-Element types. This field contains a semicolon-separated list of FAMILY names. If instances exist with which have a FAMILY name in the cancel list, then those the instances will be cancelled or removed, only leaving the instances created by this row.
NOTE: This list must end with a semicolon, even if it contains only a single entry.
If you leave this blank, the same sub-element can appear under two different groups, the vendor specific group and a generic group.
For example,
1315_FR_DLC_SNMP-CIR|Generic~Agent |1.3.6.1|AP~1315_FR_DLC_Match|<*>|AP~1315_FR_DLC_SNMP-CIR |<*>| |
1315_FR_DLC_noSNMP-CIR|Generic~Agent |1.3.6.1|AP~1315_FR_DLC_Match|<*>|AP~1315_FR_DLC_noSNMP-CIR|<*>| |
Cisco_FR_DLC_SNMP-CIR|Generic~Agent |1.3.6.1|AP~Cisco_FR_DLC_Match|<*>|AP~Cisco_FR_DLC_SNMP-CIR|<*>|1315_FR_DLC_SNMP-CIR;|
The Instances Discovered by AP~1315_FR_DLC_SNMP-CIR will be discarded. The Instances Discovered by AP~Cisco_FR_DLC_SNMP-CIR will be added.
7.3 Complex formulas
Complex formulas do not avoid temporary variables. Complex formulas are used during the discovery of resources. Since the discovery process does not occur frequently, unlike the collection process, this allows for a more lenient performance policy. This is the difference between “complex’ and ‘lite’ formulas.
7.3.1 Match formulas
A match formula should be designed to return a result as quickly as possible. If a table contains a large number of entries, but a single answer is enough to decide on the match, the a ‘DEF MaxLines n’ setting should be considered. This setting will limit the number of GetNext to n. If a single result is enough, ‘DEF MaxLines 1;’ will result in a single GetNext request.
Example: RFC2233_Match
Slow formula:
Def MaxLines 100;
V1=OIDVAL(ifInBroadcastPkts.*);
V2=OIDVAL(ifInMulticastPkts.*);
V3=OIDVAL(ifCounterDiscontinuityTime.*);
%V1+%V2+%V3;
This formula uses up to 300 PDUs and is not guarantee to succeed. An interface may not implement all three counters.
Optimized formula
Def MaxLines 1;
V1=OIDVAL(count(*,ifInBroadcastPkts.*));
V2=OIDVAL(count(*,ifInMulticastPkts.*));
V3=OIDVAL(count(*,ifCounterDiscontinuityTime.*));
%V1+%V2+%V3;
This formula only uses 3 SNMP PDUs.
7.3.2 SNMP Get/GetNext optimization
SNMP GetNext is used when a table browse is required. This is always true when the input variable for an OID contains a start character. The SNMP GetNext is a rather inefficient method of retrieving information. A SNMP PDU is sent and returned across the network for each line of a table. If several OIDs belonging to the same table are used within the same line of the formula language, they will be submitted in the same PDU. If the table is large or the latency of the network is high, then the time required to perform a full browse of the table will be high.
SNMP Get is used when the index of an OID to is known in advance. This is the case when a table has already been scanned and all indexes are stored in a temporary variable (using OIDINST). That variable can then be used to drive the rest of the discovery formula. In this case, a collector knows the list of indexes to query and builds efficient PDUs. Each PDU is complete with OIDs up to the PDU limit size setting. It is possible to store 20 OIDs in a single PDU. There getting OID responses from a table of 100 entries will only require 5 round trips, as compared with 100 using GetNext.
8 Writing collection formulas
A collection formula is designed to be applied against a Sub-Element and to produce a result (string or number) that will be saved in to the database. Since collection formulas are executed on a regular basis (polling period) and applied on a larger scale than inventory formulas, they have to be as simple as possible. The main concern for a collection formula should always be efficiency, prior to readability or cosmetic.
8.1 Structure of a collection formula
8.1.1 Dim Section
Syntax: Dim {input number} AS {type} Default {default value} Name {mnemonic name};
This section declares types and default values for input variables of the formula. It is mandatory to declare the type of an input value, if that type is different from Integer.
Valid types are:
Integer
IPAddress
Gauge
Counter
MacAddress
OctetString
DisplayString
Timeticks
These names are case insensitive when used in a formula.
Default instances can be a list of values, or * to indicate that all instances should be used. Spaces are not allowed within a list of values for an input variable.
These are valid:
Integer 3
list of integers 1,3,6,9,8,109
interval 4-59
list of integers and intervals 1,3,6,9,8-96,109,200-250
character string “this is a STRING”
pointer to a string 128.3.56.7
wildcard *
Example:
Dim I1 AS Integer Default * Name TunIndex;
Dim I2 AS Integer Default * Name TunInstance;
Dim I3 AS Integer Default * Name LsrInID;
Dim I4 AS Integer Default * Name LsrOutID;
Will match Sub-Element instances: TunIndex<1000> TunInstance<123> LsrInID<456> LsrOutID<789>
8.1.2 Def section
8.1.2.1 Def SaveAlias
Syntax: Def SaveAlias {metricId};
This is used to specify an alternative storage metric ID, called a Generic Metric ID. The formula itself is stored in the database with a metric ID called a Specific Metric ID. There two rules the Generic ID must follow.
1.) The Generic Metric ID must exist in the formula table
2.) The Generic Metric ID must be of the same DataType as the current ‘Specific’ formula ( float / string )
Generic formulas
Generic formulas are used to insert a hardware abstraction layer into the formula language. This allows a formula writer to have different expressions based on different hardware capabilities. The Specific formulas handle the hardware specific capabilities. To merge all results into one unique formula the Generic Metric is used.
Example
The Specific/Generic mechanism is based on formula IDs. If nothing is specified in the formula definition, a formula saves its results to a unique ID. It’s possible to force a formula redirect all of the results to a Generic ID.
Cisco Utilization % defined using the Cisco private MIB
Juniper Utilization % defined using the Juniper private MIB
Id 100 Utilization % #Generic
Id 201 Cisco Utilization % SaveAlias = 100
Id 301 Juniper Utilization % SaveAlias = 100
Both formula results are rolled up into a generic formula called: Utilization %
8.1.2.2 Def DefaultResult
Syntax: Def DefaultResult {a number};
This is used to specify a number that will be returned by a formula. This is useful if either no SNMP data was available or a mathematical error occurred.
8.1.2.3 Def DefaultNoRespResult
Syntax: Def DefaultNoRespResult {a number};
This is used to specify a number that will be returned by a formula. This is useful if no SNMP data (at all) can be retrieved from the agent.
NOTE: If partial data is retrieved or a mathematical error occurs, the value specified by DefaultNoRespResult will not be used. The DefaultResult value will be used.
8.1.2.4 Def UseQuotedStrings
Syntax: Def UseQuotedStrings {yes|no};
This flag sets the default behavior of the collector for displaying strings. If UseQuotedStrings is set to yes (True), which is the default value, then all strings will be displayed within a pair of double quotes (”). If UseQuotedStrings is set to no (False), then strings will not be delimited by double quotes. It is always possible, inside the INDEX format string to add explicit double quotes by escaping them with backslash.
Example:
ERX VR element discovery formula to handle the VR community string:
Dim I1 as Integer default *;
Def UseQuotedStrings no;
V1=OIDINST(usdRouterRowStatus.%I1 > 0);
V2=OIDVAL(usdRouterName.%V1);
V3=OIDVAL(sysName.0 );
V4=OIDVAL(sysDescr.0 );
%V2 index "NULL||%HOSTNAME||ipAddress<%HOSTIP>IP.name<%HOSTNAME_%V2>rCommunity<%USEDRCOMMUNITY@%V2>sysDescr<%V4>physAddress<%HOSTNAME:%V2>sysName<%V3:%V2>||%V3/%V2";
8.1.3 OID gathering section
A collection formula may contain several lines with temporary variables like a discovery formula, but these lines make the formula perform slowly and should be avoided. An efficient collection formula is a formula that only has one result line. Collection formulas should be ‘Lite’ formulas, not ‘Complex’ formulas.
Lite formula:
Dim I1 As Integer Default * Name Interface;
Delta (ifInOctets.%I1) / delta (sysUpTime.0) / 100)
Complex formula:
V1=OIDVAL( ifInOctets.%I1 ) ;
%V1 index …
NOTE: Lite and Complex formulas are explained in a following section.
8.1.4 Result line section
A result line defines the format of the information that will be sent back to the calling application. Execution of a result line in a formula will result in the creation in memory of one (or more) value line(s). Each line contains the name of the device that has been queried, a result value (number or string), and an instance number for the result (either 0 if the result is not indexed, or a number corresponding to values of dimensions of that result).
A collection formula is expected to return only one result line for a Sub-Element. If two result lines are produced, they will be discarded as duplicate by the CME.
Since only one result is produced, the instance number is not really used for a collection formula. The instance is already defined in the Sub-Element instance field and does not need to be repeated. Therefore, the instance field of a collection formula is ignored, making the use of the INDEX keyword useless.
Example: (lite formula)
DEF SaveAlias 2212;
Dim I1 AS Integer Default * NAME Interface;
8 * delta(ifInOctets.%I1) * distrib(delta(sysUpTime.0), "default:1") * distrib(delta(ifLastChange.%I1), "==0:1")
8.2 Light formulas
Rules
1.) There should be only one calculation line.
2.) There should be no intermediate variable (no more Vx= … ).
3.) A target has to define all instance values before the execution.
4.) Only Sub-Elements can be targets. Hosts can not be targets.
5.) All aggregation formulas, which take a set of Sub-Elements to produce only one result, and by the way change the instance, are not suitable for Lite formulas. In general, only mono element formulas can be used.
6.) All OID type formulas are considered as Lite formulas.
7.) Each formula is analyzed the first time it is used. If the formula is found to be complex, a performance warning is issued in pvmd.log.
8.) They are marked as ServiceFormLite in the Scheduler
Scheduler
Using statGet ( found in the dataload/bin directory ) or the Collector Information GUI, it’s possible to view all of the requests inside the Scheduler. If formulas are marked as ServiceForm, then you can probably improve the performance by changing the formula syntax.
Example 1:
Formula:
V1=oidval( xxxxx.%I1);
%V1
Should be replaced by:
xxxxx.%I1;
Example 2:
Formula:
V1=oidval ( OID1.%I1);
V2=oidval ( OID2.%I1);
%V1 + %V2
Should be replaced by:
OID1.%I1 + OID2.%I2
Example 3:
Even if an OID is listed several times in the same expression, this OID is only polled once.
Formula:
V1=oidval( xxxxx.%I1);
Delta (OID1.%I1 / %V1) + delta ( OID2.%I1 / %V1)
Should be replaced by:
Delta (OID1.%I1 / xxxxx.%I1) + delta ( OID2.%I1 / xxxxx.%I1)
Example 4:
Even if the expression is very complicated, a lot of distribs and OIDs, it’s always more efficient to use a single line and avoid the use of temporary variables.
Def DefaultNoRespResult 0;
100*(abs ((((1+(distrib( ifOperStatus.%I1,"==1:1,==2:-1,==3:-1")))/2)*(delta(sysUpTime.0)))-((delta(sysUpTime.0)-(sysUpTime.0 -(ifLastChange.%I1)))*(abs (delta (distrib( ifOperStatus.%I1,"==1:1,==2:-1,==3:-1")))/2))))/delta(sysUpTime.0)
Example 5:
Sometimes it’s more powerful to make the entire formula without a filter.
Formula:
V1=oidinst ( ifOperStatus.%I1 == ‘up’);
IfOutOctets.%V1 / sysUpTime.0 ;
Should be replaced by:
(IfOutOctets.%I1 / sysUpTime.0) * distrib (ifOperStatus.%I1 == ‘up’, “==1:1”)
NOTE: The number of OIDs does not matter, but the number of PDUs does.
8.3 CME formulas
Smalltalk is the coding language required for writing CME formulas. Explaining the Smalltalk language is not in the scope of this document. The examples below show a simple and complex CME formula, and give a quick view of the Smalltalk language.
Simple example:
Generic ID Name Code
2208 Inbound Throughput (bps) HC | result newTimeStamp deltaTime |
timeStamp = nil
ifTrue: [timeStamp := inOctets date + timeOffset]
ifFalse: [newTimeStamp := inOctets date + timeOffset.
deltaTime := newTimeStamp - timeStamp.
timeStamp := newTimeStamp.
deltaTime = 0
ifFalse: [result := inOctets * 8 / deltaTime]].
self setOutputMetricRecordDate: timeStamp.
^result
Complex example:
The statements that begin ‘self system logInfo:’ are for debugging; they cause text messages to be output to the DataChannel log file (
Generic ID Name Code
2408 Ping Response Time (msec) “response comes from SNMP formula Ping Response Time.
lastTime is a variable.
testInterval is set to property testActivationFrequency (which is really pingCtlFrequency)."
| substrings timestamp timeFields yr mon day hr min sec |
self system logInfo: #GRAEMEA text: ('input formula value is ', response asString).
substrings := response value findToken: $_.
(substrings size ~= 2)
ifTrue: [^self error: 'response is not two underscore-delimited strings'].
timeFields := (substrings at: 2) tokensBasedOn: $. .
((timeFields size = 9) and: [(timeFields at: 1) asNumber = 8])
ifFalse: [^self error: 'bad timestamp format'].
yr := (timeFields at: 2) asNumber * 256 + (timeFields at: 3) asNumber.
mon := (timeFields at: 4) asNumber.
day := (timeFields at: 5) asNumber.
hr := (timeFields at: 6) asNumber.
min := (timeFields at: 7) asNumber.
sec := (timeFields at: 8) asNumber.
"Ignore the deciseconds value in octet 8 (timeField 9)"
timestamp := PvTimestamp
fromDate: (Date newDay: day monthNumber: mon year: yr)
andTime: (Time new hours: hr minutes: min seconds: sec).
"Check & return nil if timestamp is old. There must be testInterval between tests."
lastTime = nil
ifTrue: [lastTime := timestamp].
self system logInfo: #GRAEMEB text: ('lastTime starts at ', lastTime asString, ' timestamp is ', timestamp asString).
(timestamp asSeconds) < ((lastTime asSeconds) + (testInterval value asNumber))
ifTrue: [^nil].
self system logInfo: #GRAEMED text: ('returning value ', ((substrings at: 1) copyWithoutAll: '"')).
lastTime := timestamp.
^((substrings at: 1) copyWithoutAll: '"') asNumber
8.4 Collection libraries
8.4.1 Device and Interface Availability
Library - DEF UseLib RFC1213Interface;
Functions
deviceReacheability(Percent)
deviceReboot
deviceAvailabilityTime(Seconds)
deviceUnknownAvailabilityTime(Seconds)
deviceAvailability(Percent)
deviceUnknownAvailability(Percent)
ifAvailabilityTime(Seconds)
ifUnavailabilityTime(Seconds)
ifUnknownAvailabilityTime(Seconds)
ifAvailability(Percent)
ifUnavailability(Percent)
ifUnknownAvailability(Percent)
Implementation
There are two ways to implement the functions defined in the Collection library.
1.) The first version is a brand new formula that will produce results under its own ID and will require new report to display the results.
2.) The second version has a save alias. This saves results on the same ID as the previous set of availability formulas. In this case new reports are not required, but all collections using the previous set of availability formulas must be removed and replaced by the new set of formulas to avoid duplicate data requests.
Example 1:
Interface Availability percentage (AP original formula)
DEF SaveAlias 2955;
Def DefaultNoRespResult 0;
Dim I1 AS Integer Default * NAME Interface;
100 * abs( delta(sysUpTime.0) * (1 + distrib(ifOperStatus.%I1,"==1:1,==2:-1,==3:-1") )/2 - delta(sysUpTime.0)/2 * abs( delta(distrib(ifOperStatus.%I1,"==1:1,==2:-1,==3:-1"))/2 ) ) / delta(sysUpTime.0)
Example 2:
Interface Availability percentage (new formula, using new ID)
DEF UseLib RFC1213Interface;
ifAvailability(Percent)
Interface Availability percentage (new formula, using previous ID)
DEF UseLib RFC1213Interface;
DEF SaveAlias 2955;
ifAvailability(Percent)
Example 3:
Device Availability percentage (AP original, which is mainly a device reacheability percentage)
Stat( "Targets" , %H1 , "SNMP Availability (%) [last hour]" )
Example 4:
Device Availability percentage (new formula, using new ID)
DEF UseLib RFC1213Interface;
deviceAvailability(Percent)
Example 5:
Device Availability percentage (new formula, using previous ID)
DEF UseLib RFC1213Interface;
DEF SaveAlias 4382;
deviceAvailability(Percent)
No comments:
Post a Comment