Premier IT Outsourcing and Support Services within the UK

User Tools

Site Tools


Network Working Group N. Brownlee Request for Comments: 2123 The University of Auckland Category: Informational March 1997

        Traffic Flow Measurement:  Experiences with NeTraMet

Status of this Memo

 This memo provides information for the Internet community.  This memo
 does not specify an Internet standard of any kind.  Distribution of
 this memo is unlimited.


 This memo records experiences in implementing and using the Traffic
 Flow Measurement Architecture and Meter MIB. It discusses the
 implementation of NeTraMet (a traffic meter) and NeMaC (a combined
 manager and meter reader), considers the writing of meter rule sets
 and gives some guidance on setting up a traffic flow measurement
 system using NeTraMet.

Table of Contents

1 Introduction 2

 1.1 NeTraMet structure and development . . . . . . . . . . . . . .  3
 1.2 Scope of this document . . . . . . . . . . . . . . . . . . . .  4

2 Implementation 4

 2.1 Choice of meter platform . . . . . . . . . . . . . . . . . . .  4
 2.2 Programming support requirements . . . . . . . . . . . . . . .  5
   2.2.1 DOS environment  . . . . . . . . . . . . . . . . . . . . .  6
   2.2.2 Unix environment . . . . . . . . . . . . . . . . . . . . .  7
 2.3 Implementing the meter . . . . . . . . . . . . . . . . . . . .  7
   2.3.1 Data structures  . . . . . . . . . . . . . . . . . . . . .  7
   2.3.2 Packet matching  . . . . . . . . . . . . . . . . . . . . .  8
   2.3.3 Testing groups of rule addresses . . . . . . . . . . . . .  8
   2.3.4 Compression of address masks . . . . . . . . . . . . . . .  9
   2.3.5 Ignoring unwanted flow data  . . . . . . . . . . . . . . . 10
   2.3.6 Observing meter reader activity  . . . . . . . . . . . . . 11
   2.3.7 Meter memory management  . . . . . . . . . . . . . . . . . 12
 2.4 Data collection  . . . . . . . . . . . . . . . . . . . . . . . 14
 2.5 Restarting a meter . . . . . . . . . . . . . . . . . . . . . . 15
 2.6 Performance  . . . . . . . . . . . . . . . . . . . . . . . . . 16

3 Writing rule sets 16

 3.1 Rule set to observe all flows  . . . . . . . . . . . . . . . . 17
 3.2 Specifying flow direction, using computed attributes . . . . . 18
 3.3 Subroutines  . . . . . . . . . . . . . . . . . . . . . . . . . 21
 3.4 More complicated rule sets . . . . . . . . . . . . . . . . . . 23

Brownlee Informational [Page 1] RFC 2123 Traffic Flow Measurement March 1997

4 Flow data files 26

 4.1 Sample flow data file  . . . . . . . . . . . . . . . . . . . . 27
 4.2 Flow data file features  . . . . . . . . . . . . . . . . . . . 28
 4.3 Terminating and restarting meter reading . . . . . . . . . . . 29

5 Analysis applications 30 6 Using NeTraMet in a measurement system 31

 6.1 Examples of NeTraMet in production use . . . . . . . . . . . . 31

7 Acknowledgments 33 8 References 33 9 Security Considerations 34 10 Author's Address 34

1 Introduction

 Early in 1992 my University needed to develop a system for recovering
 the costs of its Internet traffic.  In March of that year I attended
 the Internet Accounting Working Group's session at the San Diego
 IETF, where I was delighted to find that the Group had produced a
 detailed architecture for measuring network traffic and were waiting
 for someone to try implementing it.
 During 1992 I produced a prototype measurement system, using balanced
 binary trees to store information about traffic flows.  This work was
 reported at the Washington IETF in November 1992.  The prototype
 performed well, but it made no attempt to recover memory from old
 flows, and the overheads in managing the balanced trees proved to be
 unacceptably high.  I moved on to develop a production-quality
 system, this time using hash tables to index the flow information.
 This version was called NeTraMet (the Network Traffic Meter), and was
 released as free software in October 1993.  Since then I have
 continued working on NeTraMet, producing new releases two or three
 times each year.  NeTraMet is now in production use at many sites
 around the world.  It is difficult to estimate the number of sites,
 but there is an active NeTraMet mailing list, which had about 130
 subscribers in March 1996.
 Early in 1996 the Realtime Traffic Flow Measurement Working Group
 (RTFM) was chartered to move the Traffic Flow Measurement
 Architecture on to the IETF standards track.  This document records
 traffic flow measurement experience gained through three years
 experience with NeTraMet.

Brownlee Informational [Page 2] RFC 2123 Traffic Flow Measurement March 1997

1.1 NeTraMet structure and development

 The Traffic Flow Architecture document [1] describes four components:
  1. METERS, which are attached to the network at the points where

it is desired to measure the traffic,

  1. METER READERS, which read data from meters and store it for later


  1. MANAGERS, which configure meters and control meter readers, and
  1. ANALYSIS APPLICATIONS, which process the data from meter readers

so as to produce whatever reports are required.

 NeTraMet is a computer program which implements the Traffic Meter,
 stores the measured flow data in memory, and provides an SNMP agent
 so as to make it available to Meter Readers.  The NeTraMet
 distribution files include NeMaC, which is a combined Manager and
 Meter Reader capable of managing an arbitrary number of meters, each
 of which may be using its own rule set, and having its flow data
 collected at its own specified intervals.  The NeTraMet distribution
 also includes several rudimentary Analysis Applications, allowing
 users to produce simple plots from NeMaC's flow data files (fd_filter
 and fd_extract) and to monitor - in real time - the flows at a remote
 meter (nm_rc and nifty).
 Since the first release the Traffic Meter MIB [2] has been both
 improved and simplified.  Significant changes have included better
 ways to specify traffic flows (i.e. more actions and better control
 structures for the Packet Matching Engine), and computed attributes
 (class and kind).  These changes have been prompted by operational
 requirements at sites using NeTraMet, and have been tested
 extensively in successive versions of NeTraMet.
 NeTraMet is widely used to collect usage data for Internet Service
 Providers.  This is especially so in Australia and New Zealand, but
 there are also active users at sites around the world, for example in
 Canada, France, Germany and Poland.
 NeTraMet is very useful as a tool for understanding exactly where
 traffic is flowing in large networks.  Since the Traffic Meters
 perform considerable data reduction (as specified by their rule sets)
 they significantly reduce the volume of data to be read by Meter
 Readers.  This characteristic makes NeTraMet particularly effective
 for networks with many remote sites.  An example of this (the
 Kawaihiko network) is briefly described below.

Brownlee Informational [Page 3] RFC 2123 Traffic Flow Measurement March 1997

 As well as providing data for post-observation analysis, NeTraMet can
 be used for real-time network monitoring and trouble-shooting.  The
 NeTraMet distribution includes 'nifty,' an X/Motif application which
 monitors traffic flows and attempts to highlight those which are

1.2 Scope of this document

 This document presents the experience gained from three years work
 with the Traffic Flow Measurement Architecture.  Its contents are
 grouped as follows
  1. Implementation issues for NeTraMet and NeMaC,
  1. How rule files work, and how to write them for particular

purposes, and

  1. How to use NeTraMet and NeMaC for short-term and long-term flow


2 Implementation

2.1 Choice of meter platform

 As pointed out in the Architecture document [1], the goal of the
 Realtime Traffic Flow Measurement Working Group is to develop a
 standard for the Traffic Meter, with the goal of seeing it
 implemented in network devices such as hubs, switches and routers.
 Until the Architecture is well enough developed to allow this, it has
 sufficed to implement the meter as a program running on a general-
 purpose computer system.
 The choice of computer system for NeTraMet was driven by the need to
 choose one which would be widely available within the Internet
 community.  One strong possibility was a Unix system, since these are
 commonly used for a variety of network support and management tasks.
 For the initial implementation, however, Unix would have had some
  1. The wide variety of different Unix systems can increase the

difficulties of software support.

  1. The cost of a Unix system as a meter is too high to allow users

to run meters simultaneously at many points within their


Brownlee Informational [Page 4] RFC 2123 Traffic Flow Measurement March 1997

 Another factor in choosing the platform was system performance.  When
 I first started implementing NeTraMet it was impossible to predict
 how much processing workload was needed for a viable meter.
 Similarly, I had no idea how much memory would be required for code
 or data.  I therefore chose to implement NeTraMet on a DOS PC. This
 was because:
  1. It is a minimum system in all respects. If the meter works

well on such a system, it can be implemented on almost any

     hardware (including routers, switches, etc.)
  1. It is an inexpensive system. Sites can easily afford to have

many meters around their networks.

  1. It is a simple system, and one which I had complete control over.

This allowed me to implement effective instrumentation to monitor

     the meter's performance, and to include a wide variety of
     performance optimisations in the code.
 Once the meter was running I needed a manager to download rule files
 to it.  Since a single manager and meter reader can effectively
 support a large number of meters, a Unix environment for NeMaC was a
 natural choice.  There are fewer software support problems for NeMaC
 than for NeTraMet since NeMaC has minimal support needs - it only
 needs to open a UDP socket to the SNMP port on each controlled meter.
 Early NeTraMet distributions contained only the PC meter and Unix
 manager.  In later releases I ported NeTraMet (the meter) to Unix,
 and extended the control features of NeMaC (the combined manager and
 meter reader).  I have also experimented with porting NeMaC to the
 DOS system.  This is not difficult, but doesn't seem to be worth
 The current version of NeTraMet is a production-quality traffic
 measurement system which has been in continuous use at the University
 of Auckland for nearly two years.

2.2 Programming support requirements

 To implement the Traffic Flow Meter I needed a programming
 environment providing good support for the following:
  1. observation of packet headers on the network;
  1. system timer with better than 10 ms resolution;
  1. IP (Internet Protocol), for communications with manager and meter


Brownlee Informational [Page 5] RFC 2123 Traffic Flow Measurement March 1997

  1. SNMP, for the agent implementing the Meter MIB.

2.2.1 DOS environment

 For the PC I chose to use Ethernet as the physical network medium.
 This is simply an initial choice, being the medium used within the
 University of Auckland's data network.  Interfaces for other media
 could easily be added as they are needed.
 In the PC environment a variety of 'generalised' network interfaces
 are available.  I considered those available from companies such as
 Novell, DEC and Microsoft and decided against them, partly because
 they are proprietary, and partly because they did not appear to be
 particularly easy to use.  Instead I chose the CRYNWR Packet Drivers
 [3] .  These are available for a wide variety of interface cards and
 are simple and clearly documented.  They support Ethernet's
 promiscuous mode, allowing one to observe headers for every passing
 packet in a straightforward manner.  One disadvantage of the Packet
 Drivers is that it is harder to use them with newer user shells (such
 as Microsoft Windows), but this was irrelevant since I intended to
 run the meter as the only program on a dedicated machine.
 Timing on the PC presented a challenge since the BIOS timer routines
 only provide a clock tick about 18 times each second, which limits
 the available time resolution.  Initially I made do with a timing
 resolution of one second for packets, since I believed that most
 flows existed for many seconds.  In recent years it has become
 apparent that many flows have lifetimes well under a second.  To
 measure them properly with the Traffic Flow Meter one needs times
 resolved to 10 millisecond intervals, this being the size of
 TimeTicks, the most common time unit within SNMP [4].  Since all the
 details of the original PC are readily available [5], it was not
 difficult to understand the underlying hardware.  I have written PC
 timer routines for NeTraMet which read the hardware timer with 256
 times the resolution of the DOS clock ticks, i.e. about 5 ticks per
 There are many TCP/IP implementations available for DOS, but most of
 them are commercial software.  Instead I chose Waterloo TCP [6],
 since this was available (including full source code) as public
 domain software.  This was necessary since I needed to modify it to
 allow me to save incoming packet headers at the same time as
 forwarding packets destined for the meter to the IP handler routines.
 For SNMP I chose CMU SNMP [7], since again this was available (with
 full source code) as public domain software.  This made it fairly
 simple to port it from Unix to the PC.

Brownlee Informational [Page 6] RFC 2123 Traffic Flow Measurement March 1997

 Finally, for the NeTraMet development I used Borland's Turbo C and
 Turbo Assembler.  Although many newer C programming environments are
 now available, I have been perfectly happy with Turbo C version 2 for
 the NeTraMet project!

2.2.2 Unix environment

 In implementing the Unix meter, the one obvious problem was 'how do I
 get access to packet headers?'  Early versions of the Unix meter were
 implemented using various system-specific interfaces on a SunOS 4.2
 system.  Later versions use libpcap [8], which provides a portable
 method of obtaining access to packet headers on a wide range of Unix
 systems.  I have verified that this works very well for ethernet
 interfaces on Solaris, SunOS, Irix, DEC Unix and Linux, and for FDDI
 interfaces on Solaris.  libpcap provides timestamps for each packet
 header with resolution determined by the system clock, which is
 certainly better than 10 ms!
 All Unix systems provide TCP/IP capabilities, so that was not an
 issue.  For SNMP I used CMU SNMP, exactly as on the PC.

2.3 Implementing the meter

 This section briefly discusses the data structures used by the meter,
 and the packet matching process.  One very strong concern during the
 evolution of NeTraMet has been the need for the highest possible
 level of meter performance.  A variety of interesting optimisations
 have been developed to achieve this; as discussed below.  Another
 particular concern was the need for efficient and effective memory
 managent; this is discussed in detail below.

2.3.1 Data structures

 All the programs in NeTraMet, NeMaC and their supporting utility
 programs are written in C, partly because C and its run-time
 libraries provides good access to the underlying hardware, and partly
 because I have found it to be a highly portable language.

Brownlee Informational [Page 7] RFC 2123 Traffic Flow Measurement March 1997

 The data for each flow is stored in a C structure.  The structure
 includes all the flow's attribute values (including packet and byte
 counts), together with a link field which can be used to link flows
 into lists.  NeTraMet assumes that Adjacent addresses are 802 MAC
 Addresses, which are all six bytes long.  Similarly, Transport
 addresses are assumes to be two bytes long, which is the case for
 port numbers in IP.  Peer addresses are normally four bytes or less
 in length.  They may, however, be as long as 20 bytes (for CLNS). I
 have chosen to use a fixed Peer address size, defined at compile
 time, so as to avoid the complexity of having variable-sized flow
 The flow table itself is an array of pointers to flow data
 structures, which allows indexed access to flows via their flow
 numbers.  There is also a single large hash table, referred to in the
 Architecture document [1] as the flow table's 'search index'.  Each
 hash value in the table points to a circular chain of flows.  To find
 a flow one computes its hash value then searches that value's flow
 The meter stores each rule in a C structure.  All the rule components
 have fixed sizes, but address fields must be wide enough to hold any
 type of address - Adjacent, Peer or Transport.  The rule address
 width is defined at compile time, in the same way as flow Peer
 addresses.  Each rule set is implemented as an array of pointers to
 rule data structures, and the rule table is an array of pointers to
 the rule sets.  The size of each rule set is specified by NeMaC
 (before it begins downloading the rule set), but the maximum number
 of rule sets is defined at compile time.

2.3.2 Packet matching

 Packet matching is carried out in NeTraMet exactly as described in
 the Architecture document [1].  Each incoming packet header is
 analysed so as to determine its attribute values.  These values are
 stored in a structure which is passed to the Packet Matching Engine.
 To facilitate matching with source and destination reversed this
 structure contains two substructures, one containing the source
 Adjacent, Peer and Transport address values, the other containing the
 destination address values.

2.3.3 Testing groups of rule addresses

 As described in the Architecture [1] each rule's address will usually
 be tested, i.e. ANDed with the rule's mask and compared with the
 rule's value.  If the comparison fails, the next rule in sequence is
 executed.  This allows one to write rule sets which use a group of
 rules to test an incoming packet to see whether one of its addresses

Brownlee Informational [Page 8] RFC 2123 Traffic Flow Measurement March 1997

  1. e.g. its SourcePeerAddress - is one of a set of specified IP

addresses. Such groups of related rules can grow quite large,

 containing hundreds of rules.  It was clear that sequential execution
 of such groups of rules would be slow, and that something better was
 The optimisation implemented in NeTraMet is to find groups of rules
 which test the same attribute with the same mask, and convert them
 into a single hashed search of their values.  The overhead of setting
 up hash tables (one for each group) is incurred once, just before the
 meter starts running a new rule set.  When a 'group' test is to be
 performed, the meter ANDs the incoming attribute value, computes a
 hash value from it, and uses this to search the group's hash table.
 Early tests showed that the rule hash chains were usually very short,
 usually having only one or two members.  The effect is to reduce
 large sequences of tests to a hash computation and lookup, with a
 very small number of compares; in short this is an essential
 optimisation for any traffic meter!
 There is, of course, overhead associated with performing the hashed
 compare.  NeTraMet handles this by having a minimum group size
 defined at compile time.  If the group is too small it is not
 combined into a hashed group.
 In early versions of NeTraMet I did not allow Gotos into a hashed
 group of rules, which proved to be an unnecessarily conservative
 position.  NeTraMet stores each group's hash table in a separate
 memory area, and keeps a pointer to the hash table in the first rule
 of the group.  (The rules data structure has an extra component to
 hold this hash table pointer).  Rules within the group don't have
 hash table pointers; when they are executed as the target of a Goto
 rule they behave as ordinary rules, i.e. their tests are performed

2.3.4 Compression of address masks

 When the Packet Matching Engine has decided that an incoming packet
 belongs to a flow which is to be measured, it searches the flow table
 to determine whether or not the flow is already present.  It does
 this by computing a hash from the packet and using it for access to
 the flow table's search index.
 When designing a hash table, one normally assumes that the objects in
 the table have a constant size.  For NeTraMet's flow table this would
 mean that each flow would contain a value for every attribute.  This,
 however, is not the case, since only those attribute values 'pushed'
 by rules during packet matching are stored for a flow.

Brownlee Informational [Page 9] RFC 2123 Traffic Flow Measurement March 1997

 To demonstrate this problem , let us assume that every flow in the
 table contains a value for only one attribute, SourcePeerAddress, and
 that the rule set decides whether flows belong to a specified list of
 IP networks, in which case only their network numbers are pushed.
 The rules perform this test using a variety of masks, since the
 network number allocations range from 16 to 24 bits in width.  In
 searching the flow table, the meter must distinguish between zeroes
 in the address and 'don't care' bits which had been ANDed out.  To
 achieve this it must store SourcePeerMask values in the flow table as
 well as the ANDed SourcePeerAddress values.
 In early versions of NeTraMet this problem was side-stepped by using
 multiple hash tables and relying on the user to write rules which
 used the same set of attributes and masks for all the flows in each
 table.  This was effective, but clumsy and difficult to explain.
 Later versions changed to using a single hash table, and storing the
 mask values for all the address attributes in each flow.
 The current version of the meter stores the address masks in
 compressed form.  After examining a large number of rule sets I
 realised that although a rule set may have many rules, it usually has
 a very small number of address masks.  It is a simple matter to build
 a table of address masks, and store an index to this 'mask table'
 instead of a complete mask.  NeTraMet's maximum number of masks is
 defined at compile time, up to a maximum of 256.  This allows me to
 use a single byte for each mask in the flow data structure,
 significantly reducing the structure's size.  As well as this size
 reduction, two masks can be compared by comparing their indices in
 the mask table, i.e. it reduces to a single-byte comparison.
 Overall, using a mask table seems to provide useful improvements in
 storage efficiency and execution speed.

2.3.5 Ignoring unwanted flow data

 As described in the Architecture document [1], every incoming packet
 is tested against the current rule set by the Packet Matching Engine.
 This section explains my efforts to improve NeTraMet performance on
 the PC by reducing the amount of processing required by each incoming
 On the PC each incoming packet causes an interrupt, which NeTraMet
 must process so as to collect information about the packet.  In early
 versions I used a ring buffer with 512 slots for packet headers, and
 simply copied each packet's first 64 bytes into the next free slot.
 The packet headers were later taken from the buffer, attribute values
 were extracted from them, and the resulting 'incoming attribute
 values' records were passed to the Packet Matching Engine.

Brownlee Informational [Page 10] RFC 2123 Traffic Flow Measurement March 1997

 I modified the interrupt handling code to extract the attribute
 values and store them in a 'buffer slot.'  This reduced the amount of
 storage required in each slot, allowing more space for storing flows.
 It did increase slightly the amount of processing done for each
 packet interrupt, but this has not caused any problems.
 In later versions I realised that if one is only interested in
 measuring IP packets, there is no point in storing (and later
 processing) Novell or EtherTalk packets!  It is a simple matter for
 the meter to inspect a rule set and determine which Peer types are of
 interest.  If there are PushRule rules which test SourcePeerType (or
 DestPeerType), they specify which types are of interest.  If there
 are no such rules, every packet type is of interest.  The PC NeTraMet
 has a set of Boolean variables, one for each protocol it can handle.
 The values of these 'protocol' variables are determined when the
 meter begins running a new rule set.  For each incoming packet, the
 interrupt handler determines the Peer type.  If the protocol is not
 of interest, no further processing is done - the packet is simply
 ignored.  In a similar manner, if Adjacent addresses are never tested
 there is no point in copying them into the packet buffer slot.
 The overall effect of these optimisations is most noticeable for rule
 files which measure IP flows on a network segment which also carries
 a lot of traffic for other network protocols; this situation is
 common on multiprotocol Local Area networks.  On the Unix version of
 NeTraMet the Operating System does all the packet interrupt
 processing, and libpcap [8] delivers packet headers directly to
 NeTraMet.  The 'protocol' and 'adjacent address' optimisations are
 still performed, at the point when NeTraMet receives the packet
 headers from libpcap.

2.3.6 Observing meter reader activity

 The Architecture document [1] explains that a flow data record must
 be held in the meter until its data has been read by a meter reader.
 A meter must therefore have a reliable way of deciding when flow data
 has been read.  The problem is complicated by the fact that there may
 be more than one meter reader, and that meter readers collect their
 data asynchronously.

Brownlee Informational [Page 11] RFC 2123 Traffic Flow Measurement March 1997

 Early versions of NeTraMet solved this problem by having a single MIB
 variable which a meter reader could set to indicate that it was
 beginning a data collection.  In response to such an SNMP SET
 request, NeTraMet would update its 'collectors' table.  This had an
 entry for each meter reader, and variables recording the start time
 for the last two collections.  The most recent collection might still
 be in progress, but its start time provides a safe estimate of the
 time when the one before it actually finished.  Space used for flows
 which have been idle since the penultimate collection started can be
 recovered by the meter's garbage collector, as described below.
 The Meter MIB [2] specifies a more general table of meter reader
 information.  A meter reader wishing to collect data from a meter
 must inform the meter of its intention by creating a row in the
 table, then setting a LastTime variable in that row to indicate the
 start of a collection.  The meter handles such a SET request exactly
 as described above.  If there are multiple meter readers the meter
 can easily find the earliest time any of them started its penultimate
 collection, and may recover flows idle since then.  Should a meter
 reader fail, NeTraMet will eventually time out its entry in the meter
 reader info table, and delete it.  This avoids a situation where the
 meter can't recover flows until they have been collected by several
 meter readers, one of which has failed.

2.3.7 Meter memory management

 In principle, the size of the flow table (i.e. the maximum number of
 flows) could be changed dynamically.  This would involve allocating
 space for the flow table's new pointer array and copying the old
 pointers into it.  NeTraMet does not implement this.  Instead the
 maximum number of flows is set from the command line when it starts
 execution.  If no maximum is specified, a compile-time default number
 is used.
 Memory for flow data structures (i.e. 'flows') is allocated
 dynamically.  NeTraMet requests the C run-time system for blocks of
 several hundred flows, and links them into a free list.  When a new
 flow is needed NeTraMet gets memory space from the free list, then
 searches the flow table's pointer array for an unused flow pointer.
 In practice a 'last-allocated' index is used to point to the flow
 table, so a simple linear search suffices.  The flow index is saved
 in the flow's data record, and its other attribute values are set to

Brownlee Informational [Page 12] RFC 2123 Traffic Flow Measurement March 1997

 To release a flow data record it must first be removed from any hash
 list it is part of - this is straightforward since those lists are
 circular.  The flow's entry in the flow table pointer array is then
 set to zero (NULL pointer), and its space is returned to the free
 Once a flow data record is created it could continue to exist
 indefinitely.  In time, however, the meter would run out of space.
 To deal with this problem NeTraMet uses an incremental garbage
 collector to reclaim memory.
 At regular intervals specified by a 'GarbageCollectInterval' variable
 the garbage collector procedure is invoked.  This searches through
 the flow table looking for flows which might be recovered.  To
 control the resources consumed by garbage collection there are limits
 on the number of in-use and idle flows which the garbage collector
 may inspect  these are set either when NeTraMet is started (as
 options on the command line) or dynamically by NeMaC (using variables
 in an Enterprise MIB for NeTraMet)
 To decide whether a flow can be recovered, the garbage collector
 considers how long it has been idle (no packets in either direction),
 and when its data was last collected.  If it has been collected by
 all known meter readers since its LastTime, its memory may be
 recovered.  This alogrithm is implemented using a variable called
 'GarbageCollectTime,' which normally contains the meter's UpTime when
 the penultimate collection (i.e. the one before last) was started.
 See the section on observing meter reader activity (above) for more
 Should flows not be collected often enough the meter could run out of
 space.  NeTraMet attempts to prevent this by having a low-priority
 background process check the percentage of flows active and compare
 it with the HighWaterMark MIB variable.  If the percentage of active
 flows is greater than the high-water mark, 'GarbageCollectTime' is
 incremented by the current value of the InactivityTimeout MIB
 The Meter MIB [2] specifies that a meter should switch to using a
 'standby' rule set if the percentage of active flows rises above
 HighWaterMark.  In using NeTraMet to measure traffic flows to and
 from the University of Auckland it has not been difficult to create
 standby rules which are very similar to the 'production' rule file,
 differing only in that they push much less information about flows.
 This has, on several occasions, allowed the meter to continue running
 for one or two days after the meter reader failed.  When the meter
 reader restarted, it was able to collect all the accumulated flow

Brownlee Informational [Page 13] RFC 2123 Traffic Flow Measurement March 1997

 The MIB also specifies that the meter should take some action when
 the active flow percentage rises above its FloodMark value.  If this
 were not done, the meter could spend a rapidly increasing proportion
 of its time garbage collecting, to the point where its ability to
 respond to requests from its manager would be compromised.  NeTraMet
 switches to the default rule set when its FloodMark is reached.
 A potentially large number of new flows may be created when the meter
 switches to a standby rule set.  It is important to set a
 HighWaterMark so as to allow enough flow table space for this.  In
 practice, a HighWaterMark of 65% and a FloodMark of 95% seem to work

2.4 Data collection

 As explained above, a meter reader wishing to collect flows begins
 each collection by setting the LastTime variable in its
 ReaderInfoTable row, then works its way through the flow table
 collecting data.  A number of algorithms can be used to examine the
 flow table; these are presented below.
 The simplest approach is a linear scan of the table, reading the
 LastTime variable for each row.  If the read fails the row is
 inactive.  If it succeeds, it is of interest if its LastTime value is
 greater than the time of the last collection.  Although this method
 is simple it is also rather slow, requiring an SNMP GET request for
 every possible flow; this renders it impractical.
 Early versions of NeTraMet used two 'windows' into the flow table to
 find flows which were of interest.  Both windows were SNMP tables,
 indexed by a variable which specified a time.  A succession of
 GETNEXT requests on one of these windows allowed NeMaC (the meter
 reader) to find the flow indices for all flows which had been active
 since the specified time.  The two windows were the ActivityTime
 window (which located active flows), and the CreateTime window (which
 located new flows).  Knowing the index of an active flow, the meter
 reader can GET the values for all the attributes of interest.  NeMaC
 allows the user to specify which these are, rather than simply read
 all the attributes.
 Having the two windows allowed NeMaC to read attributes which remain
 constant - such as the flow's address attributes - when the flow is
 created, but to only read attributes which change with time - such as
 its packet and byte counts - during later collections.  Experience
 has shown, however, that many flows have rather short lifetimes; one
 effect of this is that the improved efficiency of using two windows
 does not result in any worthwhile improvement in collection

Brownlee Informational [Page 14] RFC 2123 Traffic Flow Measurement March 1997

 The current version of the Meter MIB [2] uses a TimeFilter variable
 in the flow table entries.  This can be used with GETNEXT requests to
 find all flows which have been active since a specified time
 directly, without requiring the extra 'window' SNMP variables.  It
 can be combined with SNMPv2's GETBULK request to further reduce the
 number of SNMP packets needed for each collection; I have yet to
 implement this in NeTraMet.
 A disadvantage of using SNMP to collect data from the meter is that
 SNMP packets impose a high overhead.  For example, if we wish to read
 an Integer32 variable (four bytes of data), it will be returned with
 its object identifier, type and length, i.e. at least ten bytes of
 superfluous data.  One way to reduce this overhead is to use an
 Opaque object to return a collection of data.  NeTraMet uses this
 approach to retrieve 'column activity data' from the meter, as
 Each packet of column activity data contains data values for a
 specified attribute, and each value is preceded by its flow number.
 The flow table can be regarded as a two-dimensional array, with a
 column for each flow attribute.  Column activity data objects allow
 the meter reader to read columns of the flow table, so as to collect
 only those attributes specified by the user.  The actual
 implementation is complicated by the fact that since the flow table
 is read column by column, rows can become active after the first
 column has been read.  NeMaC reads the widest columns (those with
 greatest size in bytes, e.g. PeerAddress) first, and ignores any rows
 which appear in later columns.  Newly active rows will, of course, be
 read in the next collection.
 Using Opaque objects in this way dramatically reduces the number of
 SNMP packets required to read a meter.  This has proved worthwhile in
 situations where the number of flows is large (for example on busy
 LANs), and where the meter(s) are physically dispersed over slow WAN
 links.  It has the disadvantage that general-purpose MIB browsers
 cannot understand the column activity variables, but this seems a
 small price to pay for the improved data collection performance.

2.5 Restarting a meter

 If a meter fails, for example because of a power failure, it will
 restart and begin running rule set 1, the default rule set which is
 built into the meter.  Its manager must recognise that this has
 happened, and respond with some suitable action.
 NeMaC allows the user to specify a 'keepalive' interval.  After every
 such interval NeMaC reads the meter's sysUptime and compares it with
 the last sysUptime.  If the new sysUptime is less than the last one,

Brownlee Informational [Page 15] RFC 2123 Traffic Flow Measurement March 1997

 NeMaC decides that the meter has restarted.  It downloads the meter's
 backup rule set and production rule set, then requests the meter to
 start running the production rule set.  In normal use we use a
 keepalive interval of five minutes and a collection interval of 15
 minutes.  If a meter restarts, we lose up to five minutes data before
 the rules sets are downloaded.
 Having the meter run the default rule set on startup is part of the
 Traffic Flow Measurement Architecture [1], in keeping with the notion
 that meters are very simple devices which do not have disk storage.
 Since disks are now very cheap, it may be worth considering whether
 the architecture should allow a meter to save its configuration
 (including rule sets) on disk.

2.6 Performance

 The PC version of the meter, NeTraMet, continually measures how much
 processor time is being used.  Whenever there is no incoming packet
 data to process, 'dummy' packets are generated and placed in the
 input buffer.  These packets are processed normally by the Packet
 Matching Engine; they have a PeerType of 'dummy.'  The numbers of
 dummy and normal packets are counted by the meter; their ratio is
 used as an estimate of the processor time which is 'idle,' i.e. not
 being used to process incoming packets.  The Unix version is intended
 to run as a process in a multiprocessing system, so it cannot busy-
 wait in this way.
 The meter also collects several other performance measures; these can
 be displayed on the meter console in response to keyboard requests.
 The PC meter can be used with a 10 MHz 286 machine, on which it can
 handle a steady load of about 750 packets per second.  On a 25 MHz
 386SX it will handle about 1250 packets per second.  Users have
 reported that a 40 MHz 486 can handle peaks of about 3,000 packets
 per second without packet loss.  The Unix meter has been tested
 metering traffic on a (lightly loaded) FDDI interface; it uses about
 one percent of the processor time on a SPARC 10 system running

3 Writing rule sets

 The Traffic Meter provides a versatile device for measuring a user-
 specified set of traffic flows, and performing useful data reduction
 on them.  This data reduction capability not only minimises the
 volume of data to be collected by meter readers, but also simplifies
 the later processing of traffic flow data.

Brownlee Informational [Page 16] RFC 2123 Traffic Flow Measurement March 1997

 The flows of interest, and the processing to be performed, are
 specified in a 'rule set' which is downloaded to the meter (NeTraMet)
 by the manager (NeMaC). This section explains what is involved in
 writing rule sets.
 NeTraMet is limited to metering packets observed on a network
 segment.  This means that for all the observed flows, Source and Dest
 Type attributes (e.g. SourcePeerType and DestPeerType) have the same
 The NeTraMet implementation uses single variables in its flow data
 structure for AdjacentType, SourceType and TransType.  Nonetheless,
 the rule sets discussed below push values for both Source and Dest
 Type attributes; this make sure that packet matching works properly
 with the directions reversed, even for a meter which allows Source
 and Dest Type values to be different.

3.1 Rule set to observe all flows

 NeMaC reads rule sets from text files which contain the rules, the
 set number which the meter (and meter reader) will identify them by,
 and a 'format,' i.e. a list specifying which attributes the meter
 reader should collect and write to the flow data file.  The #
 character indicates the start of a comment; NeMaC ignores the rest of
 the line.
   SET 2
     SourcePeerType & 255 = Dummy:   Ignore, 0;
     Null & 0 = 0:  GotoAct, Next;
     SourcePeerType     & 255             = 0: PushPkttoAct, Next;
     DestPeerType       & 255             = 0: PushPkttoAct, Next;
     SourcePeerAddress  & = 0: PushPkttoAct, Next;
     DestPeerAddress    & = 0: PushPkttoAct, Next;
     SourceTransType    & 255             = 0: PushPkttoAct, Next;
     DestTransType      & 255             = 0: PushPkttoAct, Next;
     SourceTransAddress & 255.255         = 0: PushPkttoAct, Next;
     DestTransAddress   & 255.255         = 0: CountPkt, 0;
   FORMAT FlowRuleSet FlowIndex FirstTime "  "
      SourcePeerType SourcePeerAddress DestPeerAddress "  "
      SourceTransType SourceTransAddress DestTransAddress "  "
      ToPDUs FromPDUs "  " ToOctets FromOctets;

Brownlee Informational [Page 17] RFC 2123 Traffic Flow Measurement March 1997

 The first rule tests the incoming packet's SourcePeerType to see
 whether it is 'dummy.'  If it is, the packet is ignored, otherwise
 the next rule is executed.
 The second rule tests the Null attribute.  Such a test always
 succeeds, so the rule simply jumps to the action of the next rule.
 (The keyword 'next' is converted by NeMaC into the number of the
 following rule.)
 The third rule pushes the packet's SourcePeerType value, then jumps
 to the action of the next rule.  The user does not know in advance
 what the value of PushPkt rules will be, which is why the value
 appearing in them is always zero.  The user must take care not to
 write rule sets which try to perform the test in a PushPkt rule.
 This is a very common error in a rule set, so NeMaC tests for it and
 displays an error message.
 The following rules push a series of attribute values from the
 packet, and the last rule also Counts the packet, i.e. it tells the
 Packet Matching Engine (PME) that the packet has been successfully
 matched.  The PME responds by searching the flow table to see whether
 the flow is already current (i.e. in the table), creating a new flow
 data record for it should this be necessary, and incrementing its
 packet and byte counters.
 Overall this rule set simply classifies the packet (i.e. decides
 whether or not it is to be counted), then pushes all the Peer and
 Transport attribute values for it.  It makes no attempt to specify a
 direction for the flow - this is left to the PME, as described in
 [1].  The resulting flow data file will have each flow's source and
 destination addresses in the order of the first packet the meter
 observed for the flow.

3.2 Specifying flow direction, using computed attributes

 As indicated above, the Packet Matching Engine will reliably
 determine the flow, and the direction within that flow, for every
 packet seen by a meter.  If the rule set does not specify a direction
 for the flow, the PME simply assumes that the first packet observed
 for a flow is travelling forward, i.e. from source to destination.
 In later analysis of the flow data, however, one is usually
 interested in traffic to or from a particular source.
 One can achieve this in a simple manner by writing a rule set to
 specify the source for flows.  All that is required is to have rules
 which succeed if the packet is travelling in the required direction,
 and which execute a 'Fail' action otherwise.  This is demonstrated in
 the following two examples.

Brownlee Informational [Page 18] RFC 2123 Traffic Flow Measurement March 1997

 (Note that early versions of NeMaC allowed 'Retry' as a synonym for
 'Fail.'  The current version also allows 'NoMatch,' which seems a
 better way to imply "fail, allowing PME to try a second match with
 directions reversed.")
   #  Count IP packets from network
   SourcePeerType & 255 = IP: Pushto, ip_pkt;
   Null & 0 = 0: Ignore, 0;
     SourcePeerAddress & = Goto c_pkt;
     Null & 0 = 0: NoMatch, 0;
     SourcePeerAddress & = 0: PushPkttoAct, Next;
     DestPeerAddress & = 0: CountPkt, 0;
 The rule labelled ip_pkt tests whether the packet came from network
 130.216.  If it did not, the test fails and the following rule
 executes a NoMatch action, causing the PME to retry the match with
 the directions reversed.  If the second match fails the packet did
 not have 130.216 as an end-point, and is ignored.
 The next rule set meters IP traffic on a network segment which
 connects two routers, g1 and g2.  It classifies flows into three
 groups - those travelling from g1 to g2, those whose source is g1 and
 those whose source is g2.

Brownlee Informational [Page 19] RFC 2123 Traffic Flow Measurement March 1997

   #  Count IP packets between two gateways
   #     -------+-------------------+------------------+-------
   #            |                   |                  |
   #       +----+-----+        +----+-----+        +---+---+
   #       |   g1     |        |    g2    |        | meter |
   #       +-+-+-+-+--+        +-+-+-+-+--+        +-------+
   SourcePeerType & 255 = IP: Pushto, ip_pkt;
   Null & 0 = 0: Ignore, 0;
     SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-80-48-81-0E-7C:
         Goto, s1;
     Null & 0 = 0: Goto, s2;
     DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 02-07-01-04-ED-4A
         GotoAct, g3;
     Null & 0 = 0: GotoAct, g1;
     SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 02-07-01-04-ED-4A:
         Goto, s3;
     Null & 0 = 0: NoMatch, 0;
     DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-80-48-81-0E-7C:
         NoMatch, 0;
     Null & 0 = 0: GotoAct, g2;
   g1: FlowClass & 255 = 1:  PushtoAct, c_pkt;  # From g1
   g2: FlowClass & 255 = 2:  PushtoAct, c_pkt;  # From g2
   g3: FlowClass & 255 = 3:  PushtoAct, c_pkt;  # g1 to g2
     SourceAdjacentAddress & FF-FF-FF-FF-FF-FF = 0:
         PushPkttoAct, Next;
     DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 0: PushPkttoAct, Next;
     SourcePeerAddress & = 0: PushPkttoAct, Next;
     DestPeerAddress & = 0: PushPkttoAct, Next;
     Null & 0 = 0:  Count, 0
 The first two rules ignore non-IP packets.  The next two rules Goto
 s1 if the packet's source was g1, or to s2 otherwise.  The rule
 labelled s2 tests whether the packet's source was g2; if not a
 NoMatch action is executed, allowing the PME to try the match with
 the packet's direction reversed.  If the match fails on the second
 try the packet didn't come from (or go to) g1 or g2, and is ignored.

Brownlee Informational [Page 20] RFC 2123 Traffic Flow Measurement March 1997

 Packets which come from g1 are tested by the rule labelled s1, and
 the PME will Goto either g3 or g1.
 Packets which came from g2 are tested by the rule labelled s3.  If
 they are not going to g1 the PME will Goto g2.  If they are going to
 g1 a NoMatch action is executed - we want them counted as backward-
 travelling packets for the g1-g2 flow.
 The rules at g1, g2 and g3 push the value 1, 2 or 3 from their rule
 into the flow's FlowClass attribute.  This value can be used by an
 Analysis Application to separate the flows into the three groups of
 interest.  FlowClass is an example of a 'computed' attribute, i.e.
 one whose value is Pushed by the PME during rule matching.
 The remaining rules Push the values of other attributes required for
 later analysis, then Count the flow.

3.3 Subroutines

 Subroutines are implemented in the PME in much the same way as in
 BASIC.  A subroutine body is just a sequence of statements, supported
 by the GoSub and Return actions.  'GoSub' saves the PME's running
 environment and jumps to the first rule of the subroutine body.
 Subroutine calls may be nested as required - NeTraMet defines the
 maximum nesting at compile time.  'Return n' restores the environment
 and jumps to the action part of the nth rule after the Gosub, where n
 is the index value from the Return rule.
 The Return action provides a way of influencing the flow of control
 in a rule set, rather like a FORTRAN Computed Goto.  This is one way
 in which a subroutine can return a result.  The other way is by
 Pushing a value in either a computed attribute (as demonstrated in
 the preceding section), or in a flow attribute.
 One common use for a subroutine is to test whether a packet attribute
 matches one of a set of values.  Such a subroutine becomes much more
 useful if it can be used to test one of several attributes.  The PME
 architecture provides for this by using 'meter variables' to hold the
 names of the attributes to be tested.  The meter variables are called
 V1, V2, V3, V4 and V5, and the Assign action is provided to set their
 values.  If, for example, we need a subroutine to test either
 SourcePeerAddress or DestPeerAddress, we write its rules to test V1
 instead.  Before calling the subroutine we Assign SourcePeerAddress
 to V1; later tests of V1 are converted by the PME into tests on
 SourcePeerAddress.  Note that since meter variables may be reassigned
 in a subroutine, their values are part of the environment which must
 be saved by a Gosub action.

Brownlee Informational [Page 21] RFC 2123 Traffic Flow Measurement March 1997

 The following rule set demonstrates the use of a subroutine ..
   #  Rule specification file to tally IP packets in three groups:
   #     UA to AIT, UA to elsewhere, AIT to elsewhere
   #     -------+-------------------+-----------------+--------
   #            |                   |                 |
   #       +----+-----+        +----+-----+        +---+---+
   #       |   UA     |        |   AIT    |        | meter |
   #       +-+-+-+-+--+        +-+-+-+-+--+        +-------+
     SourcePeerType & 255 = IP:      PushtoAct, ip_pkt;
     Null & 0 = 0:                   Ignore, 0;
     v1 & 0 = SourcePeerAddress:  AssignAct, Next;
     Null & 0 = 0:  Gosub, classify;
       Null & 0 = 0:  GotoAct, from_ua;    # 1 ua
       Null & 0 = 0:  GotoAct, from_ait;   # 2 ait
       Null & 0 = 0:  NoMatch, 0;          # 3 other
     v1 & 0 = DestPeerAddress:  AssignAct, Next;
     Null & 0 = 0:  Gosub, classify;
       Null & 0 = 0:  Ignore, 0;            # 1 ua-ua
       Null & 0 = 0:  GotoAct, ok_pkt;      # 2 ua-ait
       Null & 0 = 0:  Gotoact, ok_pkt;      # 3 ua-other
     v1 & 0 = DestPeerAddress:  AssignAct, Next;
     Null & 0 = 0:  Gosub, classify;
       Null & 0 = 0:  NoMatch, 0;           # 1 ait-ua
       Null & 0 = 0:  Ignore, 0;            # 2 ait-ait
       Null & 0 = 0:  GotoAct, ok_pkt;      # 3 ait-other
     Null & 0 = 0:                   Count, 0;
 The subroutine begins at the rule labelled classify (shown below).
 It returns to the first, second or third rule after the invoking
 Gosub rule, depending on whether the tested PeerAddress is in the UA,
 AIT, or 'other' group of networks.  In the listing below only one
 network is tested in each of the groups - it is trivial to add more
 rules (one per network) into either of the first two groups.  In this
 example the subroutine Pushes the network number from the packet into
 the tested attribute before returning.

Brownlee Informational [Page 22] RFC 2123 Traffic Flow Measurement March 1997

 The first invocation of classify (above) begins at the rule labelled
 ip_pkt.  It Assigns SourcePeerAddress to V1 then executes a Gosub
 action.  Classify returns to one of the three following rules.  They
 will Goto from_ua or from_ait if the packet came from the UA or AIT
 groups, otherwise the PME will retry the match.  This means that
 matched flows will have a UA or AIT network as their source, and
 flows between other networks will be ignored.
 The next two invocations of 'classify' test the packet's
 DestPeerAddress.  Packets from AIT to UA are Retried, forcing them to
 be counted as AU to AIT flows.  Packets from UA to UA are ignored, as
 are packets from AIT to AIT.
     v1 & =  GotoAct, ua;   # ua
     v1 & =   GotoAct, ait;  # ait
     Null & 0 = 0:                    Return, 3;     # other
     v1 & = 0:            PushPkttoAct, Next;
     Null & 0 = 0:                    Return, 1;
     v1 & = 0:            PushPkttoAct, Next;
     Null & 0 = 0:                    Return, 2;

3.4 More complicated rule sets

 The next example demonstrates a way of grouping IP flows together
 depending on their Transport Address, i.e. their IP port number.
 Simply Pushing every flow's SourceTransAddress and DestTransAddress
 would produce a large number of flows, most of which differ only in
 one of their transport addresses (the one which is not a well-known
 Instead we Push the well-known port number into each flow's
 SourceTransAddress; its DestTransAddress will be zero by default.
   SourcePeerType & 255 = dummy:      Ignore, 0;
   SourcePeerType & 255 = IP:         Pushto, IP_pkt;
   Null & 0 = 0:   GotoAct, Next;
   SourcePeerType & 255 = 0:      PushPkttoAct, Next;
   Null & 0 = 0:  Count, 0;  # Count others by protocol type
  SourceTransType & 255 = tcp:    Pushto, tcp_udp;
  SourceTransType & 255 = udp:    Pushto, tcp_udp;
  SourceTransType & 255 = icmp:   CountPkt, 0;
  SourceTransType & 255 = ospf:   CountPkt, 0;
  Null & 0 = 0:  GotoAct, c_unknown;  #  Unknown transport type

Brownlee Informational [Page 23] RFC 2123 Traffic Flow Measurement March 1997

  SourceTransAddress & 255.255 = domain:  PushtoAct, c_well_known;
  SourceTransAddress & 255.255 = ftp:     PushtoAct, c_well_known;
  SourceTransAddress & 255.255 = 113:     PushtoAct, c_well_known;
  SourceTransAddress & 255.255 = 2049:    PushtoAct, c_well_known;
  SourceTransAddress & 255.255 = 110:     PushtoAct, c_well_known;
  SourceTransAddress & 255.255 = smtp:    PushtoAct, c_well_known;
  SourceTransAddress & 255.255 = telnet:  PushtoAct, c_well_known;
  SourceTransAddress & 255.255 = www:     PushtoAct, c_well_known;
  SourceTransAddress & 255.255 = 6000:    PushtoAct, c_well_known;
  DestTransAddress & 255.255 = domain:    GotoAct, s_domain;
  DestTransAddress & 255.255 = ftp:       GotoAct, s_ftp;
  DestTransAddress & 255.255 = 113:       GotoAct, s_imap;
  DestTransAddress & 255.255 = 2049:      GotoAct, s_nfs;
  DestTransAddress & 255.255 = 110:       GotoAct, s_pop;
  DestTransAddress & 255.255 = smtp:      GotoAct, s_smtp;
  DestTransAddress & 255.255 = telnet:    GotoAct, s_telnet;
  DestTransAddress & 255.255 = www:       GotoAct, s_www;
  DestTransAddress & 255.255 = 6000:      GotoAct, s_xwin;
  Null & 0 = 0:  GotoAct, c_unknown;  #  'Unusual' port
  SourceTransType    & 255 = 0:     PushPkttoAct, Next;
  DestTransType      & 255 = 0:     PushPkttoAct, Next;
  SourceTransAddress & 255.255 = 0: PushPkttoAct, Next;
  DestTransAddress   & 255.255 = 0: CountPkt, 0;
  Null & 0 = 0:  Count, 0
 The first few rules ignore dummy packets, select IP packets for
 further processing, and count packets for other protocols in a single
 flow for each PeerType.  TCP and UDP packets cause the PME to Push
 their TransType and Goto tcp_udp.  ICMP and OSPF packets are counted
 in flows which have only their TransType Pushed.

Brownlee Informational [Page 24] RFC 2123 Traffic Flow Measurement March 1997

 At tcp_udp the packets' SourceTransAddress is tested to see whether
 it is included in a set of 'interesting' port numbers.  If it is, the
 port number is pushed from the rule into the SourceTransAddress
 attribute, and the packet is counted at c_well_known.  (NeMaC accepts
 Pushto as a synonym for PushRuleto).
 This testing is repeated for the packet's DestTransAddress; if one of
 these tests succeeds the PME Goes to the corresponding rule above and
 Pushes the port number into the flow's SourceTransAddress.  If these
 tests fail the packet is counted at c_unknown, where all the flow's
 Trans attributes are pushed.  For production use more well-known
 ports would need to be included in the tests above - c_unknown is
 intended only for little-used exception flows!
 Note that these rules only Push a value into a flow's
 SourceTransAddress, and they don't contain any NoMatch actions.  They
 therefore don't specify a packet's direction, and they could be used
 in other rule sets to group together flows for well-known ports.
 The last example (below) meters flows from a remote router, and
 demonstrates another approach to grouping well-known ports.
  SourceAdjacentAddress & FF-FF-FF-FF-FF-FF =
      00-60-3E-10-E0-A1:  Goto, gateway;  # tmkr2 router
  DestAdjacentAddress & FF-FF-FF-FF-FF-FF = 00-60-3E-10-E0-A1:
      Goto, gateway;  # Source is tmkr2
  Null & 0 = 0:  Ignore, 0;
  SourcePeerType & 255 = IP:  GotoAct, IP_pkt;
  Null & 0 = 0:  GotoAct, Next;
  SourcePeerType & 255 = 0:  CountPkt, 0;
  SourceTransType & 255 = tcp:    PushRuleto, tcp_udp;
  SourceTransType & 255 = udp:    PushRuleto, tcp_udp;
  Null & 0 = 0:  GotoAct, not_wkp;  #  Unknown transport type
  SourceTransAddress & FC-00 = 0:  GotoAct, well_known_port;
  DestTransAddress & FC-00 = 0:  NoMatch, 0;
  Null & 0 = 0:  GotoAct, not_wkp;
  DestTransAddress   & 255.255       = 0: PushPkttoAct, Next;
  SourcePeerType     & 255           = 0: PushPkttoAct, Next;
  DestPeerType       & 255           = 0: PushPkttoAct, Next;

Brownlee Informational [Page 25] RFC 2123 Traffic Flow Measurement March 1997

  SourcePeerAddress  & = 0: PushPkttoAct, Next;
  DestPeerAddress    & = 0: PushPkttoAct, Next;
  SourceTransType    & 255           = 0: PushPkttoAct, Next;
  DestTransType      & 255           = 0: PushPkttoAct, Next;
  SourceTransAddress & 255.255       = 0: CountPkt, 0;
 The first group of rules test incoming packet's AdjacentAddresses to
 see whether they belong to a flow with an end point at the specified
 router.  Any which don't are ignored.  Non-IP packets are counted in
 flows which only have their PeerType Pushed; these will produce one
 flow for each non-IP protocol.  IP packets with TransTypes other than
 UDP and TCP are counted at not_wkp, where all their address
 attributes are pushed.
 The high-order six bits of SourceTransAddress for UDP and TCP packets
 are compared with zero.  If this succeeds their source port number is
 less than 1024, so they are from a well-known port.  The port number
 is pushed from the rule into the flow's SourceTransAddress attribute,
 and the packet is counted at well_known_port.  If the test fails, it
 is repeated on the packet's DestTransAddress.  If the destination is
 a well-known port the match is Retried, and will succeed with the
 well-known port as the flow's source.
 If later analysis were to show that a high proportion of the observed
 flows were from non-well-known ports, further pairs of rules could be
 added to perform a test in each direction for other heavily-used

4 Flow data files

 Although the Architecture document [1] specifies - in great detail -
 how the Traffic Flow Meter works, and how a meter reader should
 collect flow data from a meter, it does not say anything about how
 the collected data should be stored.  NeMaC uses a simple, self-
 documenting file format, which has proved to be very effective in
 There are two kinds of records in a flow data file:  flow records and
 information records.  Each flow record is simply a sequence of
 attribute values with separators (these can be specified in a NeMaC
 rule file) or spaces between them, terminated by a newline.
 Information records all start with a cross-hatch.  The file's first
 record begins with ##, and identifies the file as being a file of
 data from NeTraMet.  It records NeMaC's parameters and the time this
 collection was started.  The file's second record begins with
 #Format: and is a copy of the Format statement used by NeMaC to
 collect the data.

Brownlee Informational [Page 26] RFC 2123 Traffic Flow Measurement March 1997

 The rest of the file is a sequence of collected data sets.  Each of
 these starts with a #Time:  record, giving the time-of-day the
 collection was started, the meter name, and the range of meter times
 this collection represents.  These from and to times are meter
 UpTimes, i.e. they are times in hundredths of seconds since the meter
 commenced operation.  Most analysis applications have simply used the
 collection start times (which are ASCII time-of-day values), but the
 from and to times could be used to convert Uptime values to time-of-
 day.  The flow records which comprise a data set follow the #Time

4.1 Sample flow data file

 A sample flow data file appears below.  Most of the flow records have
 been deleted, but lines of dots show where they were.
##NeTraMet v3.2.  -c300 -r rules.lan -e rules.default
  test_meter -i eth0  4000 flows  starting at 12:31:27 Wed 1 Feb 95
#Format: flowruleset flowindex firsttime  sourcepeertype
  sourcepeeraddress destpeeraddress  topdus frompdus
  tooctets fromoctets
#Time: 12:31:27 Wed  1 Feb 95 Flows
  from 1 to 3642
1 2 13  5  1138 0  121824 0
1 3 13  2  4215 0  689711 0
1 4 13  7  1432 0  411712 0
1 5 13  6  8243 0  4338744 0
3 6 3560  2  0 10  0 1053
3 7 3560  2  59 65  4286 3796
3 8 3560  7  0 4  0 222
3 9 3560  2  118 1  32060 60
3 10 3560  6  782 1  344620 66
3 11 3560  7  0 1  0 73
3 12 3560  5  1 1  60 60
3 13 3560  7  2 2  120 158
3 14 3560  5  2 2  120 120
3 15 3560  5  0 1  0 60
3 16 3560  5  2 2  120 120
     .  .  .  .  .  .  .  .  .
3 42 3560  7  0 1  0 60
3 43 3560  7  0 1  0 60
3 44 3560  7  0 1  0 60
3 45 3560  7  0 1  0 60
3 46 3560  5  2 2  120 120
3 47 3560  5  2 2  120 120
3 48 3560  5  2 2  120 120
3 49 3560  5  0 1  0 60
3 50 3664  5  0 1  0 60

Brownlee Informational [Page 27] RFC 2123 Traffic Flow Measurement March 1997

3 51 3664  5  0 1  0 60
3 52 3664  5  4 4  240 240
#Time: 12:36:25 Wed  1 Feb 95 Flows
  from 3641 to 33420
3 6 3560  2  0 21  0 2378
3 7 3560  2  9586 7148  1111118 565274
3 8 3560  7  0 26  0 1983
3 9 3560  2  10596 1  2792846 60
3 10 3560  6  16589 1  7878902 66
3 11 3560  7  0 87  0 16848
3 12 3560  5  20 20  1200 1200
3 13 3560  7  15 14  900 1144
3 14 3560  5  38 38  2280 2280
3 15 3560  5  0 30  0 1800
3 16 3560  5  20 20  1200 1200
3 17 3560  5  0 11  0 660
    .  .  .  .  .  .  .  .  .
3 476 26162  7  0 1  0 82
3 477 27628  7  1 1  543 543
3 478 27732  7  1 1  543 543
3 479 31048  7  1 1  60 60
3 480 32717  2  0 4  0 240
3 481 32717  2  0 232  0 16240
#Time: 12:41:25 Wed  1 Feb 95 Flows
  from 33419 to 63384
3 6 3560  2  51 180  3079 138195
3 7 3560  2  21842 18428  2467693 1356570
3 8 3560  7  0 30  0 2282
3 9 3560  2  24980 1  5051834 60
3 10 3560  6  20087 1  8800070 66
3 11 3560  7  0 164  0 32608
3 12 3560  5  41 41  2460 2460
3 14 3560  5  82 82  4920 4920
3 15 3560  5  0 60  0 3600
    .  .  .  .  .  .  .  .  .

4.2 Flow data file features

 Several features of NeMaC's flow data files (as indicated above) are
 worthy of note:
  1. Collection times overlap slightly between samples. This allows for

flows which were created after the collection started, and makes

  sure that flows are not missed from a collection.

Brownlee Informational [Page 28] RFC 2123 Traffic Flow Measurement March 1997

  1. The rule set may change during a run. The above shows flows from

rule set 1 - the default set - in the first collection, followed by

  the first flows created by rule set 3 (which has just been
  downloaded by NeMaC).
  1. FlowIndexes may be reused by the meter once their flows have been

recovered by the garbage collector. The combination of

  FlowRuleSet, FlowIndex and StartTime are needed to identify a flow
  1. Packet and Byte counters are 32-bit unsigned integers, and are

never reset by the meter. Computing the counts occurring within a

  collection interval requires taking the difference between the
  collected count and its value when the flow was last collected.
  Note that counter wrap-around can be allowed for by simply
  performing an unsigned subtraction and ignoring any carry.
  1. In the sample flow data file above I have used double spaces as

separators between the flow identifiers, peer addresses, pdu counts

  and packet counts.
  1. The format of addresses in the flow data file depends on the type

of address. NeMaC always displays Adjacent addresses as six hex

  bytes separated by hyphens, and Transport addresses as (16-bit)
  integers.  The format of a Peer address depends on its PeerType,
  e.g. dotted decimal for IP. To facilitate this NeMaC needs to know
  the PeerType for each flow; the user must request NeMaC to collect

4.3 Terminating and restarting meter reading

 When NeMaC first starts collecting from a meter, it reads the flow
 data for all active flows.  This provides a starting point for
 analysis applications to compute the counts between successive
 From time to time the user needs to terminate a flow data file and
 begin a new one.  For example, a user might need to generate a
 separate file for each day of metering.  NeMaC provides for this by
 closing the file after each collection, then opening it and appending
 the data from the next collection.  To terminate a file the user
 simply renames it.  The Unix system will effect the name change
 either immediately (if the file was closed) or as soon as the current
 collection is complete (and the file is closed).
 When NeMaC begins its next collection it observes that the file has
 disappeared, so it creates a new one and writes the # header records
 before writing the collected data.

Brownlee Informational [Page 29] RFC 2123 Traffic Flow Measurement March 1997

 There is one aspect of the above which requires some care on the
 user's part.  The last data set in a file is not duplicated as the
 first data set of the next file.  In other words, analysis
 applications must either look ahead at the first data set of the next
 file, or begin by reading the last data set of the previous file.  If
 they fail to do this they will loose one collection's worth of flow
 data at each change of file.

5 Analysis applications

 Most analysis applications will be unique, taking data produced by a
 locally-developed rule set and producing reports to satisfy specific
 local requirements.  The NeTraMet distribution files include three
 applications which are of general use, as follows:
  1. fd_filter computes data rates, i.e. the differences between

successive data sets in a flow data file. It also allows the user

  to assign a 'tag' number to each flow; these are 'computed'
  attributes similar to FlowClass and FlowKind - the only difference
  is that they are computed from the collected data sets.
  1. fd_extract takes 'tagged' files from fd_filter and produces simple

'column list' files for use by other programs. One common use for

  fd_extract is to produce time-series data files which can be plotted
  by utilities like GNUPlot.
  1. nm_rc is a 'remote console' for a NeTraMet meter. It is a slightly

simplified version of NeMaC combined with fd_filter. It can be used

  to monitor any meter, and will display (as lines of text
  characters) information about the n busiest flows observed during
  each collection interval.
  1. nifty is a traffic flow analyser, which (like nm_rc) displays data

from a NeTraMet meter. nifty is an X/Motif application, which

  produces displays like 'Packet rate (pps) vs Flow lifetime
  (minutes),' so as to highlight those flows which are 'interesting.'
 These applications are useful in themselves, and they provide a good
 starting point for users who wish to write their own analysis

Brownlee Informational [Page 30] RFC 2123 Traffic Flow Measurement March 1997

6 Using NeTraMet in a measurement system

 This section gives a brief summary of the steps involved in setting
 up a traffic measurement system using NeTraMet.  These are:
  1. Decide what is to be measured. One good way to approach this is to

specify exactly which flows are to be measured, and what reports

  will be required.  Specifying the flows should make it obvious
  where meters will have to be placed so that the flows can be
  observed, whether PCs will be adequate for the task, etc..
  1. Install meters. As well as actually placing the meter hosts this

includes making sure that they are configured correctly, with

  appropriate IP addresses, SNMP community strings, etc.
  1. Develop the rule set (and a standby rule set). The degree of

difficulty here depends on how much is known in advance about the

  traffic.  One possible approach is to start with the meter default
  rule set and measure how much traffic there is for each PeerType.
  (This is a good way to verify that NeTraMet and NeMaC are working
  properly).  You can now add rules so as to increase the granularity
  of the flows; this will of course increase the number of flows to
  be collected, and force the meter's garbage collector to work
  harder.  Another approach is to try a rule set with very fine
  granularity (i.e. one which Pushes all the address attributes),
  then observing how many flows are collected every few minutes.
  1. Develop a strategy for controlling meter reader. This means

setting the meter's maximum number of flows, the collection

  interval, how breaks between flow data files will be handled, how
  often NeMaC should check that the meter is running, etc.
  1. Develop application(s) to process the collected flow data and

produce the required files and reports.

  1. Test run. Monitor the system, then refine the rule sets and meter

reading strategy until the overall system performance is

 This process can take quite a long time, but the overall result is
 well worth the effort.

6.1 Examples of NeTraMet in production use

 At the University of Auckland we run two sets of meters.  One of
 these measures the traffic entering and leaving our University
 network, and generates usage reports for all our Internet users.
 This has been in production since early 1994.

Brownlee Informational [Page 31] RFC 2123 Traffic Flow Measurement March 1997

 The other set consists of meters which are distributed at
 Universities throughout New Zealand.  They provide continuous traffic
 flow measurements at five-minute intervals for all the links making
 up the Universities' network (Kawaihiko); this system has been in
 production since January 1996, and has already proved very useful in
 planning the network's development.
 The Kawaihiko Network provides IP connectivity for the New Zealand
 Universities.  They are linked via a Frame Relay cloud, using a
 partial mesh of permanent virtual circuits.  There is a NeTraMet
 meter at each site, metering inward and outward traffic.  All the
 meters are managed from Auckland, and they all run copies of the same
 rule set.
 The rule set has about 650 rules, most of which are in a single
 subroutine which classifies PeerAddresses into three categories -
 'Kawaihiko network,' 'other New Zealand network' and 'non-New Zealand
 network.'  Inside New Zealand IP addresses lie within six CIDR
 blocks, and there are about four hundred older networks which have
 addresses outside those blocks.  The rules are arranged in groups by
 subnet size, i.e. all the /24 networks are tested first, then the /23
 networks, etc, finishing with the /16 networks.  This means that
 although there are about 600 networks, any PeerAddress can be
 classified with only nine tests.
 The Kawaihiko rule set classifies flows, using computed attributes to
 indicate the network 'kind' (Kawaihiko / New Zealand / international)
 for each flow's SourcePeerAddress and DestPeerAddress, and to
 indicate whether the flow is a 'network news' flow or not.
 Flow data is collected from all of the meters every five minutes, and
 is used to produce weekly reports, as follows:
  1. Traffic Plots. Plots of the 5-minute traffic rates for each site,

showing international traffic in and out, news traffic in and out,

  and total traffic in and out of the site.
  1. Traffic Matrices. Two of these are produced, one for news traffic,

the other for total traffic. They show the traffic rates from

  every site (including 'other New Zealand' and 'international') to
  every other site.  The mean, third quartile and maximum are printed
  for every cell in the matrices.

Brownlee Informational [Page 32] RFC 2123 Traffic Flow Measurement March 1997

7 Acknowledgments

 This memo documents the implementation work on traffic flow
 measurement here at the University of Auckland.  Many of my
 University colleagues have contributed significantly to this work,
 especially Russell Fulton (who developed the rules sets, Perl scripts
 and Cron jobs which produce our traffic usage reports automatically
 week after week) and John White (for his patient help in documenting
 the project).

8 References

  [1] Brownlee, N., Mills, C., and G. Ruth, "Traffic Flow
  Measurement: Architecture", RFC 2063, The University of Auckland,
  Bolt Beranek and Newman Inc., GTE Laboratories, Inc, January 1997.
  [2] Brownlee, N., "Traffic Flow Measurement:  Meter MIB",
  RFC 2064, The University of Auckland, January 1997.
  [3] CRYNWR Packer Drivers distribution site:
  [4] Case J., McCloghrie K., Rose M., and Waldbusser S.,
  "Structure of Management Information for version 2 of the
  Simple Network Managemenet Protocol", RFC 1902, SNMP Research
  Inc., Hughes LAN Systems, Dover Beach Consulting, Carnegie
  Mellon University, April 1993.
  [5] IBM Corporation, "IBM PC Technical Reference Manual," 1984.
  [6] Waterloo TCP distribution site:
  [7] CMU SNMP distribution site:
  [8] libpcap distribution site:*.tar.gz

Brownlee Informational [Page 33] RFC 2123 Traffic Flow Measurement March 1997

9 Security Considerations

 Security issues are not discussed in detail in this document.  The
 meter's management and collection protocols are responsible for
 providing sufficient data integrity and confidentiality.

10 Author's Address

 Nevil Brownlee
 The University of Auckland
 Phone: +64 9 373 7599 x8941

Brownlee Informational [Page 34]

/data/webs/external/dokuwiki/data/pages/rfc/rfc2123.txt · Last modified: 1997/03/28 00:26 by

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki