RE: [Syslog-sec] RE: Maximum message size

[Chris Lonvick <clonvick@cisco.com>]

Hi All,

Do we have rough consensus on this?  We can still request a slot at the
upcoming IETF to discuss this.

Thanks,
Chris

On Mon, 18 Oct 2004, Rainer Gerhards wrote:

> Anton:
>
> I understand the source of the 480 byte limit you outlined in
> -transport. Even though we have now decided that we can use UDP
> fragementation, I still think the fact that 480 is the only safe number
> still persists.
>
> So I do not see any value in requiring an application to support
> messages of up to 1K when we know this might not work. I agree, it looks
> like it has worked (at least most of the time) so far, but why introduce
> a problem if there is no need to? Maybe we had no issues in the past
> because messages were most often below 480 octets.
>
> Also, I do not think that we can say the minimum to be supported size is
> 1K in protocol and then allow a transport mapping to specify something
> lower. If it is 1K, then it MUST be 1K in all cases. Else I can not see
> how we will be able to ensure interoperability. I am of the strong
> opinion we need to know a maximum size that is guaranteed to be
> received.
>
> I do not see the value in raising the minimum to be supported size to
> 1K. Maybe I am overlooking the obvious, but I think we deliberatley
> introduce problems if we go above that.
>
> And, yes, I am refering to IPv4 only, because I am looking at the
> weakest transport (which, in this case, is also the most commonly used).
>
> Rainer
>
> > -----Original Message-----
> > From: Anton Okmianski [mailto:aokmians@cisco.com]
> > Sent: Thursday, October 14, 2004 7:54 PM
> > To: Rainer Gerhards; ietfdbh@comcast.net; syslog-sec@employees.org
> > Subject: RE: [Syslog-sec] RE: Maximum message size
> >
> > Rainer:
> >
> > The ~480 byte limit was on the size of packet, not datagram or message
> > which can be fragmented into multiple packets.  Besides, for IPv6 it
> > was ~1160.  In order to avoid requiring people to implement
> > "proprietary" transport fragmentation we did say that implementations
> > must only support messages of size which does not require this
> > fragmentation.  But we don't do that fragmentation anymore and rely on
> > IP fragmentation.
> >
> > If the intention of this MUST requirement is a baseline
> > interoperability and we know that existing applications may be sending
> > messages as large as 1K, then it seems to me the minimum required
> > support should be at least that big.
> >
> > The transport may provide its own recommendations.  For example, udp
> > transport will *recommend* that messages do not exceed the smallest
> > MTU size on the network, which could be smaller or larger, but that
> > will be just a recommendation.  For TCP, it is not as big of an issue,
> > for example, because it handles loss of segments.  I don't think such
> > transport layer requirements should be in the protocol which describes
> > the content of the messages.
> >
> > Cheers,
> >
> > Anton.
> >
> > > -----Original Message-----
> > > From: Rainer Gerhards [mailto:rgerhards@hq.adiscon.com]
> > > Sent: Thursday, October 14, 2004 11:00 AM
> > > To: Anton Okmianski; ietfdbh@comcast.net; syslog-sec@employees.org
> > > Subject: RE: [Syslog-sec] RE: Maximum message size
> > >
> > > Hello Anton:
> > >
> > > the 480 octet limit stems back to your work and proposal in
> > > -transport-udp. I thought you found out that more than 480
> > > bytes can not be transmitted "reliably" (whatever this means for
> > UDP).
> > >
> > > Of course, I can remove this and we could simply state that
> > > senders send data and receivers receive data and that the
> > > receiver MAY receive any size or not. But wouldn't it make
> > > sense to have one size that I can count on to be received?
> > >
> > > The intension of the 480 octets was to provide a known
> > > message size that every implementation actually MUST support
> > > - so if an applications intends to send a message that MUST
> > > be received fully, that application knows a maximum size.
> > >
> > > If you mean that with the upcoming version of -transport-udp
> > > the 480 octet can safely be replaced by 1K or 2K then we of
> > > course can change this.
> > >
> > > I, however, think that a minimal burden should be placed on
> > > an implementation. I would tend to call a syslog receiver
> > > that is only capable of receiving 400 octets to be
> > > non-compliant. But, OK, maybe we should say this is OK. So
> > > what limit is it then? Is 200 octets compliant? 100? 50?
> > > Whatever it is, I think we should require one size... If we
> > > do not require this, one could argue writing/parsing a
> > > correct header is also too much of a burden for a given scenario...
> > >
> > > Rainer
> > >
> > > > -----Original Message-----
> > > > From: Anton Okmianski [mailto:aokmians@cisco.com]
> > > > Sent: Thursday, October 14, 2004 4:36 PM
> > > > To: ietfdbh@comcast.net; Rainer Gerhards; syslog-sec@employees.org
> > > > Subject: RE: [Syslog-sec] RE: Maximum message size
> > > >
> > > > Rainer:
> > > >
> > > > I agree with a SHOULD on 64k support on receivers and MAY on
> > larger
> > > > sizes.
> > > >
> > > > I think we need to remove or change this sentence: "A
> > > receiver MUST be
> > > > able to accept messages up to and including 480 octets in length".
> > > >
> > > > 1. Is this an absolute requirement?  We keep thinking about
> > general
> > > > purpose receiver libraries.  But it does not have to be general
> > > > purpose.  If I have two specific applications with their own
> > syslog
> > > > sender/receiver implementations (how hard is it to
> > > send/receive a udp
> > > > message) and they exchange syslog messages which are known not to
> > > > exceed 400 bytes, should my receiver be required to support 8k
> > > > messages?
> > > >
> > > > 2. Why 480?  If this is intended to ensure basic interoperability,
> >
> > > > then at least this should be a size greater than 1K (size previous
> >
> > > > syslog RFC observed).  Maybe 4k or 8k.  At least it is a better
> > > > arbitrary number.
> > > >
> > > > So, I would support either removing the MUST requirement
> > > all together
> > > > or at least changing it to a more reasonable size. The
> > > latter approach
> > > > would be more limiting for implementations, but will insure
> > > > interoperability for general-purpose syslog libraries if
> > > this is the
> > > > overriding goal.
> > > >
> > > > Thanks,
> > > > Anton.
> > > >
> > > > > -----Original Message-----
> > > > > From: syslog-sec-bounces@willers.employees.org
> > > > > [mailto:syslog-sec-bounces@willers.employees.org] On
> > > Behalf Of David
> > > > > B Harrington
> > > > > Sent: Thursday, October 14, 2004 8:53 AM
> > > > > To: 'Rainer Gerhards'; syslog-sec@employees.org
> > > > > Subject: RE: [Syslog-sec] RE: Maximum message size
> > > > >
> > > > > Hi Rainer,
> > > > >
> > > > > Thanks for making the changes.
> > > > >
> > > > > I'd like to make one point a little more strenuously:
> > > > > "For interoperability reasons, all receiver
> > > implementations SHOULD
> > > > > be able to accept messages up to and including 65,535 octets in
> > > > > length."
> > > > >
> > > > > Also, I think the last two sentences could be combined into one
> > -
> > > > > "If a receiver receives a message with a length larger
> > > than 65,535
> > > > > octets, or larger than it supports, the receiver MAY discard the
> >
> > > > > message or truncate the payload."
> > > > >
> > > > > dbh
> > > > >
> > > > > -----Original Message-----
> > > > > From: syslog-sec-bounces@www.employees.org
> > > > > [mailto:syslog-sec-bounces@www.employees.org] On Behalf Of
> > Rainer
> > > > > Gerhards
> > > > > Sent: Thursday, October 14, 2004 7:34 AM
> > > > > To: syslog-sec@employees.org
> > > > > Subject: [Syslog-sec] RE: Maximum message size
> > > > >
> > > > > David and all,
> > > > >
> > > > > many thanks for all the good comments.
> > > > >
> > > > > I agree that 16 MB for a single message is far more than
> > actually
> > > > > should be used. Let's look a bit at the history of this:
> > > > >
> > > > > Initially, we noticed that the 1024 octet limit for
> > > current syslog
> > > > > is too short. I think there is still general agreement on
> > > this. We
> > > > > had a good discussion about the size issue in September
> > > 2003. A link
> > > > > to an archive is here:
> > > > >
> > > > > http://www.syslog.cc/ietf/autoarc/msg00855.html
> > > > >
> > > > > The general issue is that 1K is too short for many
> > > applications. For
> > > > > example, Marshall Glen mentions the healthcare needs
> > > (RFC3881) for a
> > > > > larger message. As far as I know, the IHE initiative currently
> > > > > standardizes on a maximum XML stream size of 32K (which is then
> > > > > supposed to be transmitted via syslog). I think a message
> > > size limit
> > > > > of 64K would probably be suffcient for a long time to come (and
> > > > > leaves some headroom for message content increases).
> > > > >
> > > > > Besides that, I myself identified a need - in very rare
> > > occasions -
> > > > > to send information larger than 64K. In extreme cases, this
> > > > > information could be as large as one megabyte.
> > > > > Again, these were cases that might happen once in a year. All of
> >
> > > > > this, of course, only if an operator
> > > > > *really* wants to submit that large information. So the initial
> > > > > approach and suggestion was to introduce a way to build "message
> >
> > > > > groups". That would have been a way to split "oversize
> > > messages" (as
> > > > > they were called
> > > > > then) into multiple syslog messages. They key to that concept
> > was
> > > > > that each single message could be kept within a
> > > reasonable limit (it
> > > > > would have worked even with 1K max message size). It
> > > would only have
> > > > > been a way to transmit very large messages in those very seldom
> > > > > cases that they would actually be used.
> > > > >
> > > > > Some time later, we came to the conclusion that it should be
> > > > > possible for a single syslog message to be of the largest
> > > imaginable
> > > > > size. It was argued that fragmentation is a transport
> > > issue and as
> > > > > such application-level "fragmentation"
> > > > > (the message groups) would not do good. We agreed to that.
> > > > >
> > > > > Even some time later, we needed to define a maximum message
> > size,
> > > > > just to limit it somewhere (if I recall correctly, that was a
> > > > > transport requirement). We picked 16MB just as a number that we
> > > > > thought would never have be reached and such be
> > > sufficient for all
> > > > > imaginable time to come. As far as I was concerend, that
> > decision
> > > > > was driven by the fact that in the past there were often
> > > limits that
> > > > > nobody expected to reach - and yet they wer quickly reached.
> > > > > Som 16MB was picked more or less as "larger than anybody
> > > will need
> > > > > anytime in the forseeable future". We did not even have a big
> > > > > discussion on that limit.
> > > > >
> > > > > I think this are the key points on how we reached this limit.
> > > > > Now, thanks to your call and looking the whole situation,
> > > I really
> > > > > think we've gone overboard. That 16MB limit is something nobody
> > > > > intends to use. Even a lower - but high limit - is extremely
> > > > > unlikely. I'd expect to see a 1MB "message" in maybe
> > > 0.0001% of the
> > > > > cases - if at all (but I expect to see e.g. 70KB
> > > "message" in maybe
> > > > > 0.001% of the cases).
> > > > >
> > > > > Also, I have read your proposal to set a maxium size for the
> > > > > message, but allow an implementation to go beyond that. Maybe
> > > > > something like
> > > > > this:
> > > > >
> > > > > ####
> > > > > 5.1  Message Length
> > > > >
> > > > >    A receiver MUST be able to accept messages up to and
> > including
> > > > > 480 octets in length.
> > > > >
> > > > >    A receiver SHOULD be able to accept messages up to and
> > > including
> > > > > 65,535 octets in length.
> > > > >    If a receiver receives a message with a length larger than it
> >
> > > > > supports, the receiver MAY
> > > > >    discard the message or truncate the payload.
> > > > >
> > > > >    A receiver MAY accept messages larger than 65,535 octets.
> > > > > ####
> > > > >
> > > > > I've dropped the wording that transports MAY support only
> > > a smaller
> > > > > max size - I think UDP was the most critical in this
> > > regard and it
> > > > > obviously handles 64K. So I assume this clause is not
> > > really needed
> > > > > if we go for 64K as the max size.
> > > > >
> > > > > With that clause, we would limit the size for all interop
> > > cases, but
> > > > > leave a window open so that the message length could be
> > > extended if
> > > > > there is a real need AND the operator and software vendor wants
> > > > > this.
> > > > > I assume nobody will implement this burden if there is no market
> >
> > > > > need to do so. But if it is, it clould at least be
> > > implemented in a
> > > > > standards-compliant way.
> > > > >
> > > > > If I look back at the original requirement - provide a way to
> > > > > transmit very rare oversize messages, I think we can safely
> > > > > implement this the bounds of the current proposals, even
> > > if the size
> > > > > limit does not go beyond 64K. If somebody needs it, this can be
> > > > > easily done via an experimental (or
> > > > > vendor-specific) SD-ID. Again, I do not think somebody
> > > will accept
> > > > > this burden without market need.
> > > > >
> > > > > In the light of this, I, too, would recommend that we limit the
> > > > > message size to 64K. I would prefer, however, that we keep the
> > > > > maximum open as in the suggested wording above.
> > > > >
> > > > > Rainer
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: David B Harrington [mailto:ietfdbh@comcast.net]
> > > > > > Sent: Wednesday, October 13, 2004 5:10 PM
> > > > > > To: Rainer Gerhards; 'Anton Okmianski';
> > syslog-sec@employees.org
> > > > > > Subject: Maximum message size
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am concerned that we are in danger of promoting uses of
> > syslog
> > > > > that
> > > > > > will defeat its current usefulness as a human-readable log of
> > > > system
> > > > >
> > > > > > activity.
> > > > > >
> > > > > > Do network operators really believe the 16,777,216
> > > octet size is a
> > > >
> > > > > > needed improvement to syslog, or are we designing this size in
> >
> > > > > > explicitly for the vendors of applications who want to
> > > use syslog
> > > > as
> > > > > a
> > > > > > programmatic interface? I believe Syslog should be designed to
> > > > meet
> > > > > > the needs of operators. Most of the discussion ssems to be
> > from
> > > > > > vendors of syslog receivers or applications. Other
> > > protocols such
> > > > as
> > > > >
> > > > > > FTP might be better used for such a specialty use case.
> > > > > >
> > > > > > Does allowing messages of 16,777,216 octets to be accumulated
> > > > within
> > > > >
> > > > > > the system log make it harder to use some
> > > widely-available minimal
> > > >
> > > > > > text editors, like Notepad, to view the system log? Will
> > having
> > > > huge
> > > > >
> > > > > > application-specific messages in the system log make it
> > > harder for
> > > > > an
> > > > > > operator to locate useful troubleshooting messages within the
> > > > system
> > > > >
> > > > > > log? Can the information in such a huge message fit within an
> > > > 80x25
> > > > > > screen, when an operator is troubleshooting network problems
> > and
> > > > > needs
> > > > > > to use a bare-bones terminal attached to the serial port of a
> > > > device
> > > > >
> > > > > > to view the system log?
> > > > > >
> > > > > > Syslog was designing to be an operator's tool, and Syslog
> > should
> > > > be
> > > > > > designed to meet the needs of operators.  Will allowing
> > messages
> > > > of
> > > > > > this size negatively impact its usefulness in the primary use
> > > > case?
> > > > > >
> > > > > > What do network **operators** think about the need for
> > > messages of
> > > >
> > > > > > this size? Have we deliberately asked them their opinion on
> > this
> > > > by,
> > > > >
> > > > > > for example, taking a survey or going to NANOG to ask them?
> > > > > >
> > > > > > I don't think we should support such large messages in
> > > the system
> > > > > > logging protocol.
> > > > > >
> > > > > >
> > > > > > David Harrington
> > > > > > dbharrington@comcast.net
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: syslog-sec-bounces@www.employees.org
> > > > > > [mailto:syslog-sec-bounces@www.employees.org] On Behalf
> > > Of Rainer
> > > > > > Gerhards
> > > > > > Sent: Wednesday, October 13, 2004 5:17 AM
> > > > > > To: Anton Okmianski; syslog-sec@employees.org
> > > > > > Subject: RE: [Syslog-sec] UDP message size issue proposal
> > > > > >
> > > > > > Anton,
> > > > > >
> > > > > > I agree to your conclusion.
> > > > > >
> > > > > > Although I am the one who always voted for larger sizes, the
> > > > > > discussion has shown that this yields to unnecessary
> > > complexity. I
> > > > > can
> > > > > > satisfy my needs with a vendor-extension structured
> > > data element,
> > > > > > which I think shows the flexibility of the new drafts.
> > > > > >
> > > > > > So I support your move. I would tend to even remove the
> > > transport
> > > > > > version header. If there is need to, we could always
> > > include that
> > > > in
> > > > > a
> > > > > > later release (e.g. "v1", which would make it clearly
> > > > distingshable
> > > > > > from the current frame format). I see little need to do so,
> > > > though.
> > > > > >
> > > > > > Regarding -protocol, I think we should still keep an upper
> > limit
> > > > in
> > > > > > the spec. Even I don't see any reason why a syslog
> > > message should
> > > > go
> > > > >
> > > > > > above 16MB. So for -protocol, I propose the following new
> > text:
> > > > > >
> > > > > > ####
> > > > > > 5.1  Message Length
> > > > > >
> > > > > >    The maximum length of any syslog message is
> > > 16,777,216 octets.
> > > > > Any
> > > > > >    receiver receiving a larger message MUST discard the
> > message.
> > > > A
> > > > > >    diagnostic message SHOULD be logged in this case.
> > > > > >
> > > > > >    A receiver MUST be able to accept messages that are
> > > 480 octets
> > > > > (or
> > > > > >    less) in length.  A receiver SHOULD be able to
> > > accept messages
> > > > > that
> > > > > >    are 65,535 octets (or less) in length.  It is
> > > RECOMMENDED that
> > > > > >    receivers be able to accept messages up to the
> > > maximum message
> > > > > > length
> > > > > >    of 16,777,216 octets.
> > > > > >
> > > > > >    If a receiver receives a message within the maximum
> > > length, but
> > > >
> > > > > > with
> > > > > >    a length larger than it handles, the receiver MAY discard
> > or
> > > > > > truncate
> > > > > >    it.
> > > > > >
> > > > > >    A transport mapping MAY define a maximum length below the
> > one
> > > > > >    specified here.  Each transport mapping MUST support
> > > a maximum
> > > > > size
> > > > > >    of 480 octets or more.
> > > > > > ####
> > > > > >
> > > > > > If there is agreement on this, I can post a new version of
> > > > > -protocol.
> > > > > > That version will also include some changes made thanks
> > > to Andrew
> > > > > Ross
> > > > > > comments (sent via private mail). I have finished this
> > > version. It
> > > > > is
> > > > > > available for immediate posting, but I would like to wait for
> > > > > > agreement on the size issue (at least if that can be reached
> > > > > quickly).
> > > > > >
> > > > > > Rainer
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: syslog-sec-bounces@www.employees.org
> > > > > > > [mailto:syslog-sec-bounces@www.employees.org] On
> > > Behalf Of Anton
> > > >
> > > > > > > Okmianski
> > > > > > > Sent: Tuesday, October 12, 2004 11:25 PM
> > > > > > > To: syslog-sec@employees.org
> > > > > > > Subject: [Syslog-sec] UDP message size issue proposal
> > > > > > >
> > > > > > > Hi!
> > > > > > >
> > > > > > > Sorry for a long delay on this issue - I was on a 2 week
> > > > vacation.
> > > > > > I
> > > > > > > have spoken with a number of TCP/UDP/IP experts regarding
> > the
> > > > > sizing
> > > > > >
> > > > > > > issue.  I am ready to propose the following changes:
> > > > > > >
> > > > > > > 1. Syslog-protocol will state that the max message will be
> > > > defined
> > > > > > by
> > > > > > > the transport layer.
> > > > > > >
> > > > > > > 2. Syslog-transport-udp will support messages up to
> > > maximum UDP
> > > > > > > datagram size of 64K. UDP is a very bad choice for
> > > large message
> > > >
> > > > > > > transmissions, so it does not make sense for us to
> > > stretch it by
> > > >
> > > > > > > adding our own fragmentation without other
> > > transmission control
> > > > > > > features such as found in TCP.
> > > > > > >
> > > > > > > 3. Syslog-transport-udp will rely on IP fragmentation and we
> > > > will
> > > > > > get
> > > > > > > rid of "proprietary" fragmentation which was designed
> > > to handle
> > > > > > > messages over 64K and deal with various
> > > > > > > non-compliant network hosts.
> > > > > > >
> > > > > > > 4. Syslog-transport-udp will recommend sending messages
> > within
> > > > the
> > > > >
> > > > > > > boundaries of prevalent MTU size on a given network
> > > to be safe.
> > > > It
> > > > >
> > > > > > > will recommend Ethernet's 1500 bytes less headers and
> > > will draw
> > > > > > reader
> > > > > > > attention to the minimum MTU size hosts on the network are
> > > > > required
> > > > > > to
> > > > > > > support for
> > > > > > > IPv6 (576 bytes) and IPv6 (1280 bytes).
> > > > > > >
> > > > > > > 5. Path MTU discovery may not work robustly and some TCP/IP
> > > > stacks
> > > > > > may
> > > > > > > not support UDP packets of full 64K size and truncate
> > > them.  We
> > > > > will
> > > > > >
> > > > > > > mention this and bite this bullet.
> > > > > > > We should not restrict the protocol due to incompliant
> > > > > > implementations
> > > > > > > because it is a moving target and penalizes compliant
> > > > > > implementations
> > > > > > > with extra overhead. The above size recommendation would
> > > > partially
> > > > >
> > > > > > > deal with this, but leave the
> > > > > > > final choice up to the administrator.
> > > > > > >
> > > > > > > 6. We will get rid of most syslog transport headers for UDP
> > as
> > > > > they
> > > > > > > will no longer be needed.  The only thing that will be left
> > is
> > > > the
> > > > >
> > > > > > > transport protocol version prefixed to every syslog message.
> > > > > Should
> > > > > >
> > > > > > > we even bother with that?
> > > > > > >
> > > > > > > This is a major change to the syslog-transport-udp.
> > > I'd like to
> > > > > get
> > > > > >
> > > > > > > positive feedback before I proceed with this update.
> > > > > > > The best part is that if we all agree on the above, the next
> > > > draft
> > > > >
> > > > > > > will be 1/3 of the size -- easier read for you. :)
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Anton.
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Syslog-sec mailing list
> > > > > > > Syslog-sec@www.employees.org
> > > > > > > http://www.employees.org/mailman/listinfo/syslog-sec
> > > > > > >
> > > > > > _______________________________________________
> > > > > > Syslog-sec mailing list
> > > > > > Syslog-sec@www.employees.org
> > > > > > http://www.employees.org/mailman/listinfo/syslog-sec
> > > > > >
> > > > > >
> > > > > >
> > > > > _______________________________________________
> > > > > Syslog-sec mailing list
> > > > > Syslog-sec@www.employees.org
> > > > > http://www.employees.org/mailman/listinfo/syslog-sec
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Syslog-sec mailing list
> > > > > Syslog-sec@www.employees.org
> > > > > http://www.employees.org/mailman/listinfo/syslog-sec
> > > > >
> > > >
> > > >
> > >
> >
> >
> _______________________________________________
> Syslog-sec mailing list
> Syslog-sec@www.employees.org
> http://www.employees.org/mailman/listinfo/syslog-sec
>
_______________________________________________
Syslog-sec mailing list
Syslog-sec@www.employees.org
http://www.employees.org/mailman/listinfo/syslog-sec