[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [Syslog-sec] RE: Maximum message size
Anton:
I understand the source of the 480 byte limit you outlined in
-transport. Even though we have now decided that we can use UDP
fragementation, I still think the fact that 480 is the only safe number
still persists.
So I do not see any value in requiring an application to support
messages of up to 1K when we know this might not work. I agree, it looks
like it has worked (at least most of the time) so far, but why introduce
a problem if there is no need to? Maybe we had no issues in the past
because messages were most often below 480 octets.
Also, I do not think that we can say the minimum to be supported size is
1K in protocol and then allow a transport mapping to specify something
lower. If it is 1K, then it MUST be 1K in all cases. Else I can not see
how we will be able to ensure interoperability. I am of the strong
opinion we need to know a maximum size that is guaranteed to be
received.
I do not see the value in raising the minimum to be supported size to
1K. Maybe I am overlooking the obvious, but I think we deliberatley
introduce problems if we go above that.
And, yes, I am refering to IPv4 only, because I am looking at the
weakest transport (which, in this case, is also the most commonly used).
Rainer
> -----Original Message-----
> From: Anton Okmianski [mailto:aokmians@cisco.com]
> Sent: Thursday, October 14, 2004 7:54 PM
> To: Rainer Gerhards; ietfdbh@comcast.net; syslog-sec@employees.org
> Subject: RE: [Syslog-sec] RE: Maximum message size
>
> Rainer:
>
> The ~480 byte limit was on the size of packet, not datagram or message
> which can be fragmented into multiple packets. Besides, for IPv6 it
> was ~1160. In order to avoid requiring people to implement
> "proprietary" transport fragmentation we did say that implementations
> must only support messages of size which does not require this
> fragmentation. But we don't do that fragmentation anymore and rely on
> IP fragmentation.
>
> If the intention of this MUST requirement is a baseline
> interoperability and we know that existing applications may be sending
> messages as large as 1K, then it seems to me the minimum required
> support should be at least that big.
>
> The transport may provide its own recommendations. For example, udp
> transport will *recommend* that messages do not exceed the smallest
> MTU size on the network, which could be smaller or larger, but that
> will be just a recommendation. For TCP, it is not as big of an issue,
> for example, because it handles loss of segments. I don't think such
> transport layer requirements should be in the protocol which describes
> the content of the messages.
>
> Cheers,
>
> Anton.
>
> > -----Original Message-----
> > From: Rainer Gerhards [mailto:rgerhards@hq.adiscon.com]
> > Sent: Thursday, October 14, 2004 11:00 AM
> > To: Anton Okmianski; ietfdbh@comcast.net; syslog-sec@employees.org
> > Subject: RE: [Syslog-sec] RE: Maximum message size
> >
> > Hello Anton:
> >
> > the 480 octet limit stems back to your work and proposal in
> > -transport-udp. I thought you found out that more than 480
> > bytes can not be transmitted "reliably" (whatever this means for
> UDP).
> >
> > Of course, I can remove this and we could simply state that
> > senders send data and receivers receive data and that the
> > receiver MAY receive any size or not. But wouldn't it make
> > sense to have one size that I can count on to be received?
> >
> > The intension of the 480 octets was to provide a known
> > message size that every implementation actually MUST support
> > - so if an applications intends to send a message that MUST
> > be received fully, that application knows a maximum size.
> >
> > If you mean that with the upcoming version of -transport-udp
> > the 480 octet can safely be replaced by 1K or 2K then we of
> > course can change this.
> >
> > I, however, think that a minimal burden should be placed on
> > an implementation. I would tend to call a syslog receiver
> > that is only capable of receiving 400 octets to be
> > non-compliant. But, OK, maybe we should say this is OK. So
> > what limit is it then? Is 200 octets compliant? 100? 50?
> > Whatever it is, I think we should require one size... If we
> > do not require this, one could argue writing/parsing a
> > correct header is also too much of a burden for a given scenario...
> >
> > Rainer
> >
> > > -----Original Message-----
> > > From: Anton Okmianski [mailto:aokmians@cisco.com]
> > > Sent: Thursday, October 14, 2004 4:36 PM
> > > To: ietfdbh@comcast.net; Rainer Gerhards; syslog-sec@employees.org
> > > Subject: RE: [Syslog-sec] RE: Maximum message size
> > >
> > > Rainer:
> > >
> > > I agree with a SHOULD on 64k support on receivers and MAY on
> larger
> > > sizes.
> > >
> > > I think we need to remove or change this sentence: "A
> > receiver MUST be
> > > able to accept messages up to and including 480 octets in length".
> > >
> > > 1. Is this an absolute requirement? We keep thinking about
> general
> > > purpose receiver libraries. But it does not have to be general
> > > purpose. If I have two specific applications with their own
> syslog
> > > sender/receiver implementations (how hard is it to
> > send/receive a udp
> > > message) and they exchange syslog messages which are known not to
> > > exceed 400 bytes, should my receiver be required to support 8k
> > > messages?
> > >
> > > 2. Why 480? If this is intended to ensure basic interoperability,
>
> > > then at least this should be a size greater than 1K (size previous
>
> > > syslog RFC observed). Maybe 4k or 8k. At least it is a better
> > > arbitrary number.
> > >
> > > So, I would support either removing the MUST requirement
> > all together
> > > or at least changing it to a more reasonable size. The
> > latter approach
> > > would be more limiting for implementations, but will insure
> > > interoperability for general-purpose syslog libraries if
> > this is the
> > > overriding goal.
> > >
> > > Thanks,
> > > Anton.
> > >
> > > > -----Original Message-----
> > > > From: syslog-sec-bounces@willers.employees.org
> > > > [mailto:syslog-sec-bounces@willers.employees.org] On
> > Behalf Of David
> > > > B Harrington
> > > > Sent: Thursday, October 14, 2004 8:53 AM
> > > > To: 'Rainer Gerhards'; syslog-sec@employees.org
> > > > Subject: RE: [Syslog-sec] RE: Maximum message size
> > > >
> > > > Hi Rainer,
> > > >
> > > > Thanks for making the changes.
> > > >
> > > > I'd like to make one point a little more strenuously:
> > > > "For interoperability reasons, all receiver
> > implementations SHOULD
> > > > be able to accept messages up to and including 65,535 octets in
> > > > length."
> > > >
> > > > Also, I think the last two sentences could be combined into one
> -
> > > > "If a receiver receives a message with a length larger
> > than 65,535
> > > > octets, or larger than it supports, the receiver MAY discard the
>
> > > > message or truncate the payload."
> > > >
> > > > dbh
> > > >
> > > > -----Original Message-----
> > > > From: syslog-sec-bounces@www.employees.org
> > > > [mailto:syslog-sec-bounces@www.employees.org] On Behalf Of
> Rainer
> > > > Gerhards
> > > > Sent: Thursday, October 14, 2004 7:34 AM
> > > > To: syslog-sec@employees.org
> > > > Subject: [Syslog-sec] RE: Maximum message size
> > > >
> > > > David and all,
> > > >
> > > > many thanks for all the good comments.
> > > >
> > > > I agree that 16 MB for a single message is far more than
> actually
> > > > should be used. Let's look a bit at the history of this:
> > > >
> > > > Initially, we noticed that the 1024 octet limit for
> > current syslog
> > > > is too short. I think there is still general agreement on
> > this. We
> > > > had a good discussion about the size issue in September
> > 2003. A link
> > > > to an archive is here:
> > > >
> > > > http://www.syslog.cc/ietf/autoarc/msg00855.html
> > > >
> > > > The general issue is that 1K is too short for many
> > applications. For
> > > > example, Marshall Glen mentions the healthcare needs
> > (RFC3881) for a
> > > > larger message. As far as I know, the IHE initiative currently
> > > > standardizes on a maximum XML stream size of 32K (which is then
> > > > supposed to be transmitted via syslog). I think a message
> > size limit
> > > > of 64K would probably be suffcient for a long time to come (and
> > > > leaves some headroom for message content increases).
> > > >
> > > > Besides that, I myself identified a need - in very rare
> > occasions -
> > > > to send information larger than 64K. In extreme cases, this
> > > > information could be as large as one megabyte.
> > > > Again, these were cases that might happen once in a year. All of
>
> > > > this, of course, only if an operator
> > > > *really* wants to submit that large information. So the initial
> > > > approach and suggestion was to introduce a way to build "message
>
> > > > groups". That would have been a way to split "oversize
> > messages" (as
> > > > they were called
> > > > then) into multiple syslog messages. They key to that concept
> was
> > > > that each single message could be kept within a
> > reasonable limit (it
> > > > would have worked even with 1K max message size). It
> > would only have
> > > > been a way to transmit very large messages in those very seldom
> > > > cases that they would actually be used.
> > > >
> > > > Some time later, we came to the conclusion that it should be
> > > > possible for a single syslog message to be of the largest
> > imaginable
> > > > size. It was argued that fragmentation is a transport
> > issue and as
> > > > such application-level "fragmentation"
> > > > (the message groups) would not do good. We agreed to that.
> > > >
> > > > Even some time later, we needed to define a maximum message
> size,
> > > > just to limit it somewhere (if I recall correctly, that was a
> > > > transport requirement). We picked 16MB just as a number that we
> > > > thought would never have be reached and such be
> > sufficient for all
> > > > imaginable time to come. As far as I was concerend, that
> decision
> > > > was driven by the fact that in the past there were often
> > limits that
> > > > nobody expected to reach - and yet they wer quickly reached.
> > > > Som 16MB was picked more or less as "larger than anybody
> > will need
> > > > anytime in the forseeable future". We did not even have a big
> > > > discussion on that limit.
> > > >
> > > > I think this are the key points on how we reached this limit.
> > > > Now, thanks to your call and looking the whole situation,
> > I really
> > > > think we've gone overboard. That 16MB limit is something nobody
> > > > intends to use. Even a lower - but high limit - is extremely
> > > > unlikely. I'd expect to see a 1MB "message" in maybe
> > 0.0001% of the
> > > > cases - if at all (but I expect to see e.g. 70KB
> > "message" in maybe
> > > > 0.001% of the cases).
> > > >
> > > > Also, I have read your proposal to set a maxium size for the
> > > > message, but allow an implementation to go beyond that. Maybe
> > > > something like
> > > > this:
> > > >
> > > > ####
> > > > 5.1 Message Length
> > > >
> > > > A receiver MUST be able to accept messages up to and
> including
> > > > 480 octets in length.
> > > >
> > > > A receiver SHOULD be able to accept messages up to and
> > including
> > > > 65,535 octets in length.
> > > > If a receiver receives a message with a length larger than it
>
> > > > supports, the receiver MAY
> > > > discard the message or truncate the payload.
> > > >
> > > > A receiver MAY accept messages larger than 65,535 octets.
> > > > ####
> > > >
> > > > I've dropped the wording that transports MAY support only
> > a smaller
> > > > max size - I think UDP was the most critical in this
> > regard and it
> > > > obviously handles 64K. So I assume this clause is not
> > really needed
> > > > if we go for 64K as the max size.
> > > >
> > > > With that clause, we would limit the size for all interop
> > cases, but
> > > > leave a window open so that the message length could be
> > extended if
> > > > there is a real need AND the operator and software vendor wants
> > > > this.
> > > > I assume nobody will implement this burden if there is no market
>
> > > > need to do so. But if it is, it clould at least be
> > implemented in a
> > > > standards-compliant way.
> > > >
> > > > If I look back at the original requirement - provide a way to
> > > > transmit very rare oversize messages, I think we can safely
> > > > implement this the bounds of the current proposals, even
> > if the size
> > > > limit does not go beyond 64K. If somebody needs it, this can be
> > > > easily done via an experimental (or
> > > > vendor-specific) SD-ID. Again, I do not think somebody
> > will accept
> > > > this burden without market need.
> > > >
> > > > In the light of this, I, too, would recommend that we limit the
> > > > message size to 64K. I would prefer, however, that we keep the
> > > > maximum open as in the suggested wording above.
> > > >
> > > > Rainer
> > > >
> > > > > -----Original Message-----
> > > > > From: David B Harrington [mailto:ietfdbh@comcast.net]
> > > > > Sent: Wednesday, October 13, 2004 5:10 PM
> > > > > To: Rainer Gerhards; 'Anton Okmianski';
> syslog-sec@employees.org
> > > > > Subject: Maximum message size
> > > > >
> > > > > Hi,
> > > > >
> > > > > I am concerned that we are in danger of promoting uses of
> syslog
> > > > that
> > > > > will defeat its current usefulness as a human-readable log of
> > > system
> > > >
> > > > > activity.
> > > > >
> > > > > Do network operators really believe the 16,777,216
> > octet size is a
> > >
> > > > > needed improvement to syslog, or are we designing this size in
>
> > > > > explicitly for the vendors of applications who want to
> > use syslog
> > > as
> > > > a
> > > > > programmatic interface? I believe Syslog should be designed to
> > > meet
> > > > > the needs of operators. Most of the discussion ssems to be
> from
> > > > > vendors of syslog receivers or applications. Other
> > protocols such
> > > as
> > > >
> > > > > FTP might be better used for such a specialty use case.
> > > > >
> > > > > Does allowing messages of 16,777,216 octets to be accumulated
> > > within
> > > >
> > > > > the system log make it harder to use some
> > widely-available minimal
> > >
> > > > > text editors, like Notepad, to view the system log? Will
> having
> > > huge
> > > >
> > > > > application-specific messages in the system log make it
> > harder for
> > > > an
> > > > > operator to locate useful troubleshooting messages within the
> > > system
> > > >
> > > > > log? Can the information in such a huge message fit within an
> > > 80x25
> > > > > screen, when an operator is troubleshooting network problems
> and
> > > > needs
> > > > > to use a bare-bones terminal attached to the serial port of a
> > > device
> > > >
> > > > > to view the system log?
> > > > >
> > > > > Syslog was designing to be an operator's tool, and Syslog
> should
> > > be
> > > > > designed to meet the needs of operators. Will allowing
> messages
> > > of
> > > > > this size negatively impact its usefulness in the primary use
> > > case?
> > > > >
> > > > > What do network **operators** think about the need for
> > messages of
> > >
> > > > > this size? Have we deliberately asked them their opinion on
> this
> > > by,
> > > >
> > > > > for example, taking a survey or going to NANOG to ask them?
> > > > >
> > > > > I don't think we should support such large messages in
> > the system
> > > > > logging protocol.
> > > > >
> > > > >
> > > > > David Harrington
> > > > > dbharrington@comcast.net
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: syslog-sec-bounces@www.employees.org
> > > > > [mailto:syslog-sec-bounces@www.employees.org] On Behalf
> > Of Rainer
> > > > > Gerhards
> > > > > Sent: Wednesday, October 13, 2004 5:17 AM
> > > > > To: Anton Okmianski; syslog-sec@employees.org
> > > > > Subject: RE: [Syslog-sec] UDP message size issue proposal
> > > > >
> > > > > Anton,
> > > > >
> > > > > I agree to your conclusion.
> > > > >
> > > > > Although I am the one who always voted for larger sizes, the
> > > > > discussion has shown that this yields to unnecessary
> > complexity. I
> > > > can
> > > > > satisfy my needs with a vendor-extension structured
> > data element,
> > > > > which I think shows the flexibility of the new drafts.
> > > > >
> > > > > So I support your move. I would tend to even remove the
> > transport
> > > > > version header. If there is need to, we could always
> > include that
> > > in
> > > > a
> > > > > later release (e.g. "v1", which would make it clearly
> > > distingshable
> > > > > from the current frame format). I see little need to do so,
> > > though.
> > > > >
> > > > > Regarding -protocol, I think we should still keep an upper
> limit
> > > in
> > > > > the spec. Even I don't see any reason why a syslog
> > message should
> > > go
> > > >
> > > > > above 16MB. So for -protocol, I propose the following new
> text:
> > > > >
> > > > > ####
> > > > > 5.1 Message Length
> > > > >
> > > > > The maximum length of any syslog message is
> > 16,777,216 octets.
> > > > Any
> > > > > receiver receiving a larger message MUST discard the
> message.
> > > A
> > > > > diagnostic message SHOULD be logged in this case.
> > > > >
> > > > > A receiver MUST be able to accept messages that are
> > 480 octets
> > > > (or
> > > > > less) in length. A receiver SHOULD be able to
> > accept messages
> > > > that
> > > > > are 65,535 octets (or less) in length. It is
> > RECOMMENDED that
> > > > > receivers be able to accept messages up to the
> > maximum message
> > > > > length
> > > > > of 16,777,216 octets.
> > > > >
> > > > > If a receiver receives a message within the maximum
> > length, but
> > >
> > > > > with
> > > > > a length larger than it handles, the receiver MAY discard
> or
> > > > > truncate
> > > > > it.
> > > > >
> > > > > A transport mapping MAY define a maximum length below the
> one
> > > > > specified here. Each transport mapping MUST support
> > a maximum
> > > > size
> > > > > of 480 octets or more.
> > > > > ####
> > > > >
> > > > > If there is agreement on this, I can post a new version of
> > > > -protocol.
> > > > > That version will also include some changes made thanks
> > to Andrew
> > > > Ross
> > > > > comments (sent via private mail). I have finished this
> > version. It
> > > > is
> > > > > available for immediate posting, but I would like to wait for
> > > > > agreement on the size issue (at least if that can be reached
> > > > quickly).
> > > > >
> > > > > Rainer
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: syslog-sec-bounces@www.employees.org
> > > > > > [mailto:syslog-sec-bounces@www.employees.org] On
> > Behalf Of Anton
> > >
> > > > > > Okmianski
> > > > > > Sent: Tuesday, October 12, 2004 11:25 PM
> > > > > > To: syslog-sec@employees.org
> > > > > > Subject: [Syslog-sec] UDP message size issue proposal
> > > > > >
> > > > > > Hi!
> > > > > >
> > > > > > Sorry for a long delay on this issue - I was on a 2 week
> > > vacation.
> > > > > I
> > > > > > have spoken with a number of TCP/UDP/IP experts regarding
> the
> > > > sizing
> > > > >
> > > > > > issue. I am ready to propose the following changes:
> > > > > >
> > > > > > 1. Syslog-protocol will state that the max message will be
> > > defined
> > > > > by
> > > > > > the transport layer.
> > > > > >
> > > > > > 2. Syslog-transport-udp will support messages up to
> > maximum UDP
> > > > > > datagram size of 64K. UDP is a very bad choice for
> > large message
> > >
> > > > > > transmissions, so it does not make sense for us to
> > stretch it by
> > >
> > > > > > adding our own fragmentation without other
> > transmission control
> > > > > > features such as found in TCP.
> > > > > >
> > > > > > 3. Syslog-transport-udp will rely on IP fragmentation and we
> > > will
> > > > > get
> > > > > > rid of "proprietary" fragmentation which was designed
> > to handle
> > > > > > messages over 64K and deal with various
> > > > > > non-compliant network hosts.
> > > > > >
> > > > > > 4. Syslog-transport-udp will recommend sending messages
> within
> > > the
> > > >
> > > > > > boundaries of prevalent MTU size on a given network
> > to be safe.
> > > It
> > > >
> > > > > > will recommend Ethernet's 1500 bytes less headers and
> > will draw
> > > > > reader
> > > > > > attention to the minimum MTU size hosts on the network are
> > > > required
> > > > > to
> > > > > > support for
> > > > > > IPv6 (576 bytes) and IPv6 (1280 bytes).
> > > > > >
> > > > > > 5. Path MTU discovery may not work robustly and some TCP/IP
> > > stacks
> > > > > may
> > > > > > not support UDP packets of full 64K size and truncate
> > them. We
> > > > will
> > > > >
> > > > > > mention this and bite this bullet.
> > > > > > We should not restrict the protocol due to incompliant
> > > > > implementations
> > > > > > because it is a moving target and penalizes compliant
> > > > > implementations
> > > > > > with extra overhead. The above size recommendation would
> > > partially
> > > >
> > > > > > deal with this, but leave the
> > > > > > final choice up to the administrator.
> > > > > >
> > > > > > 6. We will get rid of most syslog transport headers for UDP
> as
> > > > they
> > > > > > will no longer be needed. The only thing that will be left
> is
> > > the
> > > >
> > > > > > transport protocol version prefixed to every syslog message.
> > > > Should
> > > > >
> > > > > > we even bother with that?
> > > > > >
> > > > > > This is a major change to the syslog-transport-udp.
> > I'd like to
> > > > get
> > > > >
> > > > > > positive feedback before I proceed with this update.
> > > > > > The best part is that if we all agree on the above, the next
> > > draft
> > > >
> > > > > > will be 1/3 of the size -- easier read for you. :)
> > > > > >
> > > > > > Thanks,
> > > > > > Anton.
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Syslog-sec mailing list
> > > > > > Syslog-sec@www.employees.org
> > > > > > http://www.employees.org/mailman/listinfo/syslog-sec
> > > > > >
> > > > > _______________________________________________
> > > > > Syslog-sec mailing list
> > > > > Syslog-sec@www.employees.org
> > > > > http://www.employees.org/mailman/listinfo/syslog-sec
> > > > >
> > > > >
> > > > >
> > > > _______________________________________________
> > > > Syslog-sec mailing list
> > > > Syslog-sec@www.employees.org
> > > > http://www.employees.org/mailman/listinfo/syslog-sec
> > > >
> > > >
> > > > _______________________________________________
> > > > Syslog-sec mailing list
> > > > Syslog-sec@www.employees.org
> > > > http://www.employees.org/mailman/listinfo/syslog-sec
> > > >
> > >
> > >
> >
>
>
_______________________________________________
Syslog-sec mailing list
Syslog-sec@www.employees.org
http://www.employees.org/mailman/listinfo/syslog-sec