Network Working Group E. Stephan Internet-Draft France Telecom Expires: September 2, 2006 E. Moreau QoSmetrics March 2006 2006IPFIX templates for common ISP usages draft-stephan-ipfix-isp-templates-00.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on September 2, 2006. Copyright Notice Copyright (C) The Internet Society (2006). Abstract Flows observations require several levels of aggregations. Currently switchs and routers analyse flows and export flow information using Netflow. Aggregators are starting to use Netflow or IPFIX to collect basic information and to export aggregated information. In this context, this memo presents potential interoperability issues and proposes to standardize a set of templates to facilitate the Stephan & Moreau Expires September 2, 2006 [Page 1] Internet-Draft IPFIX templates for common ISP usages March 2006 exchange of flows information between ISP. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Motivations for IPFIX templates definitions . . . . . . . . . 3 2.1. Interoperability between IPFIX and Netflow versions . . . 3 2.2. Interoperability between IPFIX Implementations . . . . . . 3 2.3. Collecting Aggregated flows information . . . . . . . . . 4 3. IPFIX and Netflow interoperability . . . . . . . . . . . . . . 4 3.1. Netflow messages headers fields . . . . . . . . . . . . . 5 3.2. Netflow data records fields . . . . . . . . . . . . . . . 6 3.3. IPFIX and Netflow Time Reference . . . . . . . . . . . . . 8 3.4. Converting Netflow flow times to absolute time . . . . . . 8 4. IPFIX and Netflow V5 . . . . . . . . . . . . . . . . . . . . . 8 5. Open issues . . . . . . . . . . . . . . . . . . . . . . . . . 10 5.1. Too many Pseudo Templates in IPFIX specs . . . . . . . . . 10 5.2. Aggregating Netflow using IPFIX . . . . . . . . . . . . . 10 5.3. Data integrity . . . . . . . . . . . . . . . . . . . . . . 10 5.4. Path Template . . . . . . . . . . . . . . . . . . . . . . 10 6. Security Considerations . . . . . . . . . . . . . . . . . . . 10 7. References . . . . . . . . . . . . . . . . . . . . . . . . . . 11 7.1. Normative References . . . . . . . . . . . . . . . . . . . 11 7.2. Informative References . . . . . . . . . . . . . . . . . . 11 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 12 Intellectual Property and Copyright Statements . . . . . . . . . . 13 Stephan & Moreau Expires September 2, 2006 [Page 2] Internet-Draft IPFIX templates for common ISP usages March 2006 1. Introduction This memo defines a set of IPFIX templates for common ISP usages. The content of this memo is built on notions introduced and discussed in documents of the WG IPFIX. The reader should be familiar with these documents. 2. Motivations for IPFIX templates definitions Netflow is massively deployed by ISP. Consequently the potential of usage of IPFIX in the context of an ISP is very large. There are already a lot of contributions at the door of PSAMP and IPFIX WG which are directly related with this document: o to aggregate flows information [I-D.dressler-ipfix-aggregation]; o to use IPFIX in middlebox [I-D.quittek-ipfix-middlebox]; o Bi directional flow aggregation [I-D.boschi-ipfix-biflow] 2.1. Interoperability between IPFIX and Netflow versions To provide the first level of aggregation IFPIX must interoperate with existing Netflow versions. This memo presents the different headers and records format then it presents potential interoperability issues and proposes a set of templates. 2.2. Interoperability between IPFIX Implementations IPFIX documents don't standardize any templates. They specify many kind of pseudo templates with pseudo field ID. That will lead to different templates and consequently to interoperability issues. So it is necessary to define a set of templates to increase interoperability. As an example the intend of the section 4.3" of [I-D.ietf-ipfix- protocol] is to standardize an option template that describe "Exporting Process Reliability Statistics". Actually it doesn't. The fields 'Exporting Process ID', 'time first flow dropped' and 'time last flow dropped' are not fields identifiers. Actually their value must be picked up in a set of 3 or 4 fields. In the real world that will lead to 30 different "Exporting Process Reliability Statistics" templates. Stephan & Moreau Expires September 2, 2006 [Page 3] Internet-Draft IPFIX templates for common ISP usages March 2006 2.3. Collecting Aggregated flows information ISP are using Netflow and IPFIX for different usages. One of them is to exchange aggregated flow information with their costumers or with other ISP. They need standard templates to exchange aggregated information. [I-D.dressler-ipfix-aggregation] presents the aggregation aspect but does not proposes any template. So, despite it is not possible to define every king of flow aggregation, this memo defines templates for existing flow aggregation such as Netflow V8. 3. IPFIX and Netflow interoperability IPFIX and Netflow usages require several level of aggregation. The first level of flows description combines Netflow and IPFIX sources. Aggregators receives these descriptions either to aggregate and reexport them, or to process them locally. it appears that the first level of collector requires the capability to collect flows descriptions from both Netflow and IPFIX implementations. Consequently that requires a strong interoperability between Netflow and IPFIX exporters and collectors. This section compares the headers and the messages of Netflow and IPFIX to identify potential interoperability issues between Netflow and IPFIX exporters and collectors. To identify interoperability issues the study considers a trivial Netflow/IPFIX proxy which collects Netflow packets and reexport them in IPFIX. NOTE WELL: This comparison is based on IPFIX documents available at the beginning of June 2005. They have been updated since this date. This section uses the following conventions. S: Size in byte, or indicated (e.g. 64k) L: location: H: Message Header R: record S: Set header V: Netflow Version *: all Stephan & Moreau Expires September 2, 2006 [Page 4] Internet-Draft IPFIX templates for common ISP usages March 2006 !x: not version x 3.1. Netflow messages headers fields This section classifies each Netflow header field in term of Size, Location and of presence in Netflow Versions. Then it identifies Netflow fields which match directly an IPFIX field. IPFIX header is a sub set of Netflow header. It does not include the fields 'count', 'engine_type', 'SysUptime' and 'unix_nsecs' of Netflow versions. Stephan & Moreau Expires September 2, 2006 [Page 5] Internet-Draft IPFIX templates for common ISP usages March 2006 +------------------------------+--------------------------------+ | Netflow | IPFIX | +-------------------+---+-+----+--------------------------+---+-+ | name | S |L| V | name | S |L| +-------------------+---+-+----+--------------------------+---+-+ | version | 1 |H| * | version | 1 |H| +-------------------+---+-+----+--------------------------+---+-+ | count | 2 |H| * |No field found | - |-| +-------------------+---+-+----+--------------------------+---+-+ | No field found | - |-| - | Length | 2 |H| +-------------------+---+-+----+--------------------------+---+-+ | SysUptime | 4 |H| * |No field found | - |-| +-------------------+---+-+----+--------------------------+---+-+ | unix_secs | s |H| * |Export Time | s |H| +-------------------+---+-+----+--------------------------+---+-+ | unix_nsecs | s |H|!V9 |No field found | ? |?| +-------------------+---+-+----+--------------------------+---+-+ | flow_sequence | 4 |H| !V9|totalFlowCount | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | FLOWS | 4 |F| V9 |totalFlowCount | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | Sequence Number| 4 |H| V9 |Sequence Number | 4 |H| +-------------------+---+-+----+--------------------------+---+-+ | engine_type | 1 |H|V5,8| No field found | - |-| +-------------------+---+-+----+--------------------------+---+-+ | engine_id | 1 |H|V5,8| sourceId | 4 |H| +-------------------+---+-+----+--------------------------+---+-+ | ENGINE_TYPE | 1 |F| V9 |No field found | - |-| +-------------------+---+-+----+--------------------------+---+-+ | ENGINE_ID | 1 |F| V9 | sourceId | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ |sampling mode, | | | | | | | |interval | 2 |H| V5 | No field found | - |-| +-------------------+---+-+----+--------------------------+---+-+ |SAMPLING_INTERVAL | 4 |F| V9 | No field found | - |-| +-------------------+---+-+----+--------------------------+---+-+ |SAMPLING_ALGORITHM | 1 |F| V9 | No field found | - |-| +-------------------+---+-+----+--------------------------+---+-+ 3.2. Netflow data records fields At this step, this section integrates only Netflow V5 record fields. This section classifies each Netflow record field in term of Size, Location and of presence in Netflow Versions. Then it identifies Netflow fields which match directly an IPFIX field. Stephan & Moreau Expires September 2, 2006 [Page 6] Internet-Draft IPFIX templates for common ISP usages March 2006 +------------------------------+--------------------------------+ | Netflow | IPFIX | +-------------------+---+-+----+--------------------------+---+-+ | name | S |L| V | name | S |L| +-------------------+---+-+----+--------------------------+---+-+ | srcaddr | 4 |F| V5 |sourceIpV4Address | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | dstaddr | 4 |F| V5 |destinationIpV4Address | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | nextHop | 4 |F| V5 |ipNextHopIpV4Address | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | input | 4 |F| V5 |ingressInterface | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | output | 4 |F| V5 | egressInterface | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | dPkts | 4 |F| V5 |inPacketTotalCount | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | dOctets | 4 |F| V5 |inOctetTotalCount | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | First | 4 |F| V5 |flowStartMilliSec | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | Last | 4 |F| V5 |flowEndMilliMSec | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | srcport | 2 |F| V5 |sourceTransportPort | 2 |F| +-------------------+---+-+----+--------------------------+---+-+ | dstport | 2 |F| V5 |destinationTransportPort | 2 |F| +-------------------+---+-+----+--------------------------+---+-+ | pad1 | 4 |F| V5 |not applicable | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | TcpFlags | 1 |F| V5 | tcpControlBits | 1 |F| +-------------------+---+-+----+--------------------------+---+-+ | Proto | 1 |F| V5 |protocolIdentifier | 1 |F| +-------------------+---+-+----+--------------------------+---+-+ | Tos | 4 |F| V5 |classOfServiceIpV4 | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | SrcAS | 4 |F| V5 |bgpSourceAsNumber | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | DstAS | 4 |F| V5 |bgpDestinationAsN | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | SrcMask | 4 |F| V5 |sourceIpV4Mask | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | DstMask | 4 |F| V5 |destinationIpV4Mask | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ | Pad2 | 4 |F| V5 | not applicable | 4 |F| +-------------------+---+-+----+--------------------------+---+-+ Stephan & Moreau Expires September 2, 2006 [Page 7] Internet-Draft IPFIX templates for common ISP usages March 2006 3.3. IPFIX and Netflow Time Reference Netflow and IPFIX don't use the same reference of time to describe the begin and the end of the flow. Netflow timestamps the begin of the flow in the field 'First' and the end of a flow in the field 'Last' using 'SysUpTime' relative clock. The fields 'unix_secs' and 'unix_nsecs' of the Netflow V5 header provide a relation between absolute time (since 0000 UTC Jan 1st 1970) and 'sysUptime' relative time (reboot time). IPFIX info model defines the fields types flowStartMilliSeconds and flowEndMilliMSeconds as absolute time (since 0000 UTC Jan 1st 1970) of the begin and of the end of a flow. So the fields 'First' and 'Last' may be converted to flowStartMilliSeconds and flowEndMilliMSeconds. The consequence is that IPFIX encoding will take 2 times more bytes. Netflow V5 header field 'unix_secs' corresponds to IPFIX header field 'Export Time'. 3.4. Converting Netflow flow times to absolute time Following is a proposal to convert Netflow 'First' and 'Last' to the absolute reference of time used by IPFIX: o flowStartMilliSeconds = ('unix_secs' * 1000) + 'unix_nsecs'/ 1000000 - 'sysUptime' + 'First'; o flowEndMilliSeconds = ('unix_secs' * 1000) + 'unix_nsecs'/1000000 - 'sysUptime' + 'Last'. 4. IPFIX and Netflow V5 This section discusses mapping issues and proposes templates to map NetflowV5 on IPFIX. Netflow V5 differs from IPFIX. o It exports sampling method and sampling rate in the message header; o It does not use the same time reference; o It doesn't use option template. So most of the scope data are in the header. Following is trivial template for Netflow V5. Stephan & Moreau Expires September 2, 2006 [Page 8] Internet-Draft IPFIX templates for common ISP usages March 2006 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | FlowSet ID = 0 | Length = 53 bytes | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 257 | Field Count = 18 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | sourceIpV4Address(8) | 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | destinationIpV4Address(12) | 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ipNextHopIpV4Address(15) | 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ingressInterface(10) | 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | egressInterface(14) | 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | inPacketTotalCount(86) | 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | inOctetTotalCount(85) | 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | First(SysUptime)(*) | 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Last(SysUptime) (*) | 4 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | sourceTransportPort(7) | 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | destinationTransportPort(11) | 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | tcpControlBits(6) | 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | protocolIdentifier(4) | 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | classOfServiceIpV4(5) | 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | bgpSourceAsNumber(16) * | 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | bgpDestinationAsNumber(17)* | 2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | sourceIpV4Mask(9) | 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | destinationIpV4Mask(13) | 1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Actually this template is not applicable directly. It needs to be adapted. Several approaches may be used or mixed depending of the transport reliability, the number of sources of flows received...The approach proposed below focus on an NetflowV5 translator which Stephan & Moreau Expires September 2, 2006 [Page 9] Internet-Draft IPFIX templates for common ISP usages March 2006 received flows from several routers and export them using IPFIX. The tranport protocol is UDP: o Before sending the record 'First' and 'Last' should be translated to absolute time using the reference of time of the router; o EngineID and type should be translated in an IPFIX IE: TBD;s o Sampling rate and method should be translated to IPFIX IEs: TBD. 5. Open issues 5.1. Too many Pseudo Templates in IPFIX specs IPFIX documents specify many pseudo templates that will introduce a lot of interoperability issues. To solve this issue a field ID should accept several Field ID in its definition. The template sent will indicate the one used. This approach is close to the size reducing mechanism. 5.2. Aggregating Netflow using IPFIX IPFIX info model should have one Field ID for each field existing in one Netflow header or conversion rules. 5.3. Data integrity How to avoid IPFIX information to be corrupted by the network, by DoS attackers, lost of packets ? Which protocol to use in which case? Is there others mechanisms which are applicable ? Does it make sense to introduce a checksum field ID to protect a data record ? 5.4. Path Template This template is a common need for decribing traceroute and spatial results. 6. Security Considerations Security is a MUST in the context of exchange of information between ISP. As the security of the exchange relies mostly on the protocol used, UDP does not look appropriate to exchange information between ISP. Stephan & Moreau Expires September 2, 2006 [Page 10] Internet-Draft IPFIX templates for common ISP usages March 2006 7. References 7.1. Normative References [I-D.ietf-ipfix-info] Quittek, J., "Information Model for IP Flow Information Export", draft-ietf-ipfix-info-11 (work in progress), September 2005. [I-D.ietf-ipfix-protocol] Claise, B., "IPFIX Protocol Specification", draft-ietf-ipfix-protocol-21 (work in progress), April 2006. [RFC4148] Stephan, E., "IP Performance Metrics (IPPM) Metrics Registry", BCP 108, RFC 4148, August 2005. 7.2. Informative References [I-D.boschi-export-perpktinfo] Boschi, E. and L. Mark, "Use of IPFIX for Export of Per- Packet Information", draft-boschi-export-perpktinfo-01 (work in progress), October 2005. [I-D.boschi-ipfix-biflow] Boschi, E. and B. Trammell, "Biflow implementation support in IPFIX", draft-boschi-ipfix-biflow-01 (work in progress), October 2005. [I-D.dressler-ipfix-aggregation] Dressler, F., "IPFIX Aggregation", draft-dressler-ipfix-aggregation-02 (work in progress), December 2005. [I-D.quittek-ipfix-middlebox] Quittek, J., "Guidelines for IPFIX Implementations on Middleboxes", draft-quittek-ipfix-middlebox-00 (work in progress), February 2004. [NETFLOW_FMT] Cisco, "Netflow format". [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, "Framework for IP Performance Metrics", RFC 2330, May 1998. [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way Delay Metric for IPPM", RFC 2679, September 1999. Stephan & Moreau Expires September 2, 2006 [Page 11] Internet-Draft IPFIX templates for common ISP usages March 2006 Authors' Addresses Stephan Emile France Telecom division R&D 2 avenue Pierre Marzin Lannion, F-22307 Fax: +33 2 96 05 18 52 Email: emile.stephan@francetelecom.com Moreau Eric QoSmetrics EMEA 3-7 Rue du Theatre Massy, F-91300 Fax: +33 1 64 53 27 61 Email: eric_moreau@qosmetrics.net Stephan & Moreau Expires September 2, 2006 [Page 12] Internet-Draft IPFIX templates for common ISP usages March 2006 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Stephan & Moreau Expires September 2, 2006 [Page 13]