[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [Syslog-sec] RE: Maximum message size
Rainer:
The ~480 byte limit was on the size of packet, not datagram or message
which can be fragmented into multiple packets. Besides, for IPv6 it
was ~1160. In order to avoid requiring people to implement
"proprietary" transport fragmentation we did say that implementations
must only support messages of size which does not require this
fragmentation. But we don't do that fragmentation anymore and rely on
IP fragmentation.
If the intention of this MUST requirement is a baseline
interoperability and we know that existing applications may be sending
messages as large as 1K, then it seems to me the minimum required
support should be at least that big.
The transport may provide its own recommendations. For example, udp
transport will *recommend* that messages do not exceed the smallest
MTU size on the network, which could be smaller or larger, but that
will be just a recommendation. For TCP, it is not as big of an issue,
for example, because it handles loss of segments. I don't think such
transport layer requirements should be in the protocol which describes
the content of the messages.
Cheers,
Anton.
> -----Original Message-----
> From: Rainer Gerhards [mailto:rgerhards@hq.adiscon.com]
> Sent: Thursday, October 14, 2004 11:00 AM
> To: Anton Okmianski; ietfdbh@comcast.net; syslog-sec@employees.org
> Subject: RE: [Syslog-sec] RE: Maximum message size
>
> Hello Anton:
>
> the 480 octet limit stems back to your work and proposal in
> -transport-udp. I thought you found out that more than 480
> bytes can not be transmitted "reliably" (whatever this means for
UDP).
>
> Of course, I can remove this and we could simply state that
> senders send data and receivers receive data and that the
> receiver MAY receive any size or not. But wouldn't it make
> sense to have one size that I can count on to be received?
>
> The intension of the 480 octets was to provide a known
> message size that every implementation actually MUST support
> - so if an applications intends to send a message that MUST
> be received fully, that application knows a maximum size.
>
> If you mean that with the upcoming version of -transport-udp
> the 480 octet can safely be replaced by 1K or 2K then we of
> course can change this.
>
> I, however, think that a minimal burden should be placed on
> an implementation. I would tend to call a syslog receiver
> that is only capable of receiving 400 octets to be
> non-compliant. But, OK, maybe we should say this is OK. So
> what limit is it then? Is 200 octets compliant? 100? 50?
> Whatever it is, I think we should require one size... If we
> do not require this, one could argue writing/parsing a
> correct header is also too much of a burden for a given scenario...
>
> Rainer
>
> > -----Original Message-----
> > From: Anton Okmianski [mailto:aokmians@cisco.com]
> > Sent: Thursday, October 14, 2004 4:36 PM
> > To: ietfdbh@comcast.net; Rainer Gerhards; syslog-sec@employees.org
> > Subject: RE: [Syslog-sec] RE: Maximum message size
> >
> > Rainer:
> >
> > I agree with a SHOULD on 64k support on receivers and MAY on
larger
> > sizes.
> >
> > I think we need to remove or change this sentence: "A
> receiver MUST be
> > able to accept messages up to and including 480 octets in length".
> >
> > 1. Is this an absolute requirement? We keep thinking about
general
> > purpose receiver libraries. But it does not have to be general
> > purpose. If I have two specific applications with their own
syslog
> > sender/receiver implementations (how hard is it to
> send/receive a udp
> > message) and they exchange syslog messages which are known not to
> > exceed 400 bytes, should my receiver be required to support 8k
> > messages?
> >
> > 2. Why 480? If this is intended to ensure basic interoperability,
> > then at least this should be a size greater than 1K (size previous
> > syslog RFC observed). Maybe 4k or 8k. At least it is a better
> > arbitrary number.
> >
> > So, I would support either removing the MUST requirement
> all together
> > or at least changing it to a more reasonable size. The
> latter approach
> > would be more limiting for implementations, but will insure
> > interoperability for general-purpose syslog libraries if
> this is the
> > overriding goal.
> >
> > Thanks,
> > Anton.
> >
> > > -----Original Message-----
> > > From: syslog-sec-bounces@willers.employees.org
> > > [mailto:syslog-sec-bounces@willers.employees.org] On
> Behalf Of David
> > > B Harrington
> > > Sent: Thursday, October 14, 2004 8:53 AM
> > > To: 'Rainer Gerhards'; syslog-sec@employees.org
> > > Subject: RE: [Syslog-sec] RE: Maximum message size
> > >
> > > Hi Rainer,
> > >
> > > Thanks for making the changes.
> > >
> > > I'd like to make one point a little more strenuously:
> > > "For interoperability reasons, all receiver
> implementations SHOULD
> > > be able to accept messages up to and including 65,535 octets in
> > > length."
> > >
> > > Also, I think the last two sentences could be combined into one
-
> > > "If a receiver receives a message with a length larger
> than 65,535
> > > octets, or larger than it supports, the receiver MAY discard the
> > > message or truncate the payload."
> > >
> > > dbh
> > >
> > > -----Original Message-----
> > > From: syslog-sec-bounces@www.employees.org
> > > [mailto:syslog-sec-bounces@www.employees.org] On Behalf Of
Rainer
> > > Gerhards
> > > Sent: Thursday, October 14, 2004 7:34 AM
> > > To: syslog-sec@employees.org
> > > Subject: [Syslog-sec] RE: Maximum message size
> > >
> > > David and all,
> > >
> > > many thanks for all the good comments.
> > >
> > > I agree that 16 MB for a single message is far more than
actually
> > > should be used. Let's look a bit at the history of this:
> > >
> > > Initially, we noticed that the 1024 octet limit for
> current syslog
> > > is too short. I think there is still general agreement on
> this. We
> > > had a good discussion about the size issue in September
> 2003. A link
> > > to an archive is here:
> > >
> > > http://www.syslog.cc/ietf/autoarc/msg00855.html
> > >
> > > The general issue is that 1K is too short for many
> applications. For
> > > example, Marshall Glen mentions the healthcare needs
> (RFC3881) for a
> > > larger message. As far as I know, the IHE initiative currently
> > > standardizes on a maximum XML stream size of 32K (which is then
> > > supposed to be transmitted via syslog). I think a message
> size limit
> > > of 64K would probably be suffcient for a long time to come (and
> > > leaves some headroom for message content increases).
> > >
> > > Besides that, I myself identified a need - in very rare
> occasions -
> > > to send information larger than 64K. In extreme cases, this
> > > information could be as large as one megabyte.
> > > Again, these were cases that might happen once in a year. All of
> > > this, of course, only if an operator
> > > *really* wants to submit that large information. So the initial
> > > approach and suggestion was to introduce a way to build "message
> > > groups". That would have been a way to split "oversize
> messages" (as
> > > they were called
> > > then) into multiple syslog messages. They key to that concept
was
> > > that each single message could be kept within a
> reasonable limit (it
> > > would have worked even with 1K max message size). It
> would only have
> > > been a way to transmit very large messages in those very seldom
> > > cases that they would actually be used.
> > >
> > > Some time later, we came to the conclusion that it should be
> > > possible for a single syslog message to be of the largest
> imaginable
> > > size. It was argued that fragmentation is a transport
> issue and as
> > > such application-level "fragmentation"
> > > (the message groups) would not do good. We agreed to that.
> > >
> > > Even some time later, we needed to define a maximum message
size,
> > > just to limit it somewhere (if I recall correctly, that was a
> > > transport requirement). We picked 16MB just as a number that we
> > > thought would never have be reached and such be
> sufficient for all
> > > imaginable time to come. As far as I was concerend, that
decision
> > > was driven by the fact that in the past there were often
> limits that
> > > nobody expected to reach - and yet they wer quickly reached.
> > > Som 16MB was picked more or less as "larger than anybody
> will need
> > > anytime in the forseeable future". We did not even have a big
> > > discussion on that limit.
> > >
> > > I think this are the key points on how we reached this limit.
> > > Now, thanks to your call and looking the whole situation,
> I really
> > > think we've gone overboard. That 16MB limit is something nobody
> > > intends to use. Even a lower - but high limit - is extremely
> > > unlikely. I'd expect to see a 1MB "message" in maybe
> 0.0001% of the
> > > cases - if at all (but I expect to see e.g. 70KB
> "message" in maybe
> > > 0.001% of the cases).
> > >
> > > Also, I have read your proposal to set a maxium size for the
> > > message, but allow an implementation to go beyond that. Maybe
> > > something like
> > > this:
> > >
> > > ####
> > > 5.1 Message Length
> > >
> > > A receiver MUST be able to accept messages up to and
including
> > > 480 octets in length.
> > >
> > > A receiver SHOULD be able to accept messages up to and
> including
> > > 65,535 octets in length.
> > > If a receiver receives a message with a length larger than it
> > > supports, the receiver MAY
> > > discard the message or truncate the payload.
> > >
> > > A receiver MAY accept messages larger than 65,535 octets.
> > > ####
> > >
> > > I've dropped the wording that transports MAY support only
> a smaller
> > > max size - I think UDP was the most critical in this
> regard and it
> > > obviously handles 64K. So I assume this clause is not
> really needed
> > > if we go for 64K as the max size.
> > >
> > > With that clause, we would limit the size for all interop
> cases, but
> > > leave a window open so that the message length could be
> extended if
> > > there is a real need AND the operator and software vendor wants
> > > this.
> > > I assume nobody will implement this burden if there is no market
> > > need to do so. But if it is, it clould at least be
> implemented in a
> > > standards-compliant way.
> > >
> > > If I look back at the original requirement - provide a way to
> > > transmit very rare oversize messages, I think we can safely
> > > implement this the bounds of the current proposals, even
> if the size
> > > limit does not go beyond 64K. If somebody needs it, this can be
> > > easily done via an experimental (or
> > > vendor-specific) SD-ID. Again, I do not think somebody
> will accept
> > > this burden without market need.
> > >
> > > In the light of this, I, too, would recommend that we limit the
> > > message size to 64K. I would prefer, however, that we keep the
> > > maximum open as in the suggested wording above.
> > >
> > > Rainer
> > >
> > > > -----Original Message-----
> > > > From: David B Harrington [mailto:ietfdbh@comcast.net]
> > > > Sent: Wednesday, October 13, 2004 5:10 PM
> > > > To: Rainer Gerhards; 'Anton Okmianski';
syslog-sec@employees.org
> > > > Subject: Maximum message size
> > > >
> > > > Hi,
> > > >
> > > > I am concerned that we are in danger of promoting uses of
syslog
> > > that
> > > > will defeat its current usefulness as a human-readable log of
> > system
> > >
> > > > activity.
> > > >
> > > > Do network operators really believe the 16,777,216
> octet size is a
> >
> > > > needed improvement to syslog, or are we designing this size in
> > > > explicitly for the vendors of applications who want to
> use syslog
> > as
> > > a
> > > > programmatic interface? I believe Syslog should be designed to
> > meet
> > > > the needs of operators. Most of the discussion ssems to be
from
> > > > vendors of syslog receivers or applications. Other
> protocols such
> > as
> > >
> > > > FTP might be better used for such a specialty use case.
> > > >
> > > > Does allowing messages of 16,777,216 octets to be accumulated
> > within
> > >
> > > > the system log make it harder to use some
> widely-available minimal
> >
> > > > text editors, like Notepad, to view the system log? Will
having
> > huge
> > >
> > > > application-specific messages in the system log make it
> harder for
> > > an
> > > > operator to locate useful troubleshooting messages within the
> > system
> > >
> > > > log? Can the information in such a huge message fit within an
> > 80x25
> > > > screen, when an operator is troubleshooting network problems
and
> > > needs
> > > > to use a bare-bones terminal attached to the serial port of a
> > device
> > >
> > > > to view the system log?
> > > >
> > > > Syslog was designing to be an operator's tool, and Syslog
should
> > be
> > > > designed to meet the needs of operators. Will allowing
messages
> > of
> > > > this size negatively impact its usefulness in the primary use
> > case?
> > > >
> > > > What do network **operators** think about the need for
> messages of
> >
> > > > this size? Have we deliberately asked them their opinion on
this
> > by,
> > >
> > > > for example, taking a survey or going to NANOG to ask them?
> > > >
> > > > I don't think we should support such large messages in
> the system
> > > > logging protocol.
> > > >
> > > >
> > > > David Harrington
> > > > dbharrington@comcast.net
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: syslog-sec-bounces@www.employees.org
> > > > [mailto:syslog-sec-bounces@www.employees.org] On Behalf
> Of Rainer
> > > > Gerhards
> > > > Sent: Wednesday, October 13, 2004 5:17 AM
> > > > To: Anton Okmianski; syslog-sec@employees.org
> > > > Subject: RE: [Syslog-sec] UDP message size issue proposal
> > > >
> > > > Anton,
> > > >
> > > > I agree to your conclusion.
> > > >
> > > > Although I am the one who always voted for larger sizes, the
> > > > discussion has shown that this yields to unnecessary
> complexity. I
> > > can
> > > > satisfy my needs with a vendor-extension structured
> data element,
> > > > which I think shows the flexibility of the new drafts.
> > > >
> > > > So I support your move. I would tend to even remove the
> transport
> > > > version header. If there is need to, we could always
> include that
> > in
> > > a
> > > > later release (e.g. "v1", which would make it clearly
> > distingshable
> > > > from the current frame format). I see little need to do so,
> > though.
> > > >
> > > > Regarding -protocol, I think we should still keep an upper
limit
> > in
> > > > the spec. Even I don't see any reason why a syslog
> message should
> > go
> > >
> > > > above 16MB. So for -protocol, I propose the following new
text:
> > > >
> > > > ####
> > > > 5.1 Message Length
> > > >
> > > > The maximum length of any syslog message is
> 16,777,216 octets.
> > > Any
> > > > receiver receiving a larger message MUST discard the
message.
> > A
> > > > diagnostic message SHOULD be logged in this case.
> > > >
> > > > A receiver MUST be able to accept messages that are
> 480 octets
> > > (or
> > > > less) in length. A receiver SHOULD be able to
> accept messages
> > > that
> > > > are 65,535 octets (or less) in length. It is
> RECOMMENDED that
> > > > receivers be able to accept messages up to the
> maximum message
> > > > length
> > > > of 16,777,216 octets.
> > > >
> > > > If a receiver receives a message within the maximum
> length, but
> >
> > > > with
> > > > a length larger than it handles, the receiver MAY discard
or
> > > > truncate
> > > > it.
> > > >
> > > > A transport mapping MAY define a maximum length below the
one
> > > > specified here. Each transport mapping MUST support
> a maximum
> > > size
> > > > of 480 octets or more.
> > > > ####
> > > >
> > > > If there is agreement on this, I can post a new version of
> > > -protocol.
> > > > That version will also include some changes made thanks
> to Andrew
> > > Ross
> > > > comments (sent via private mail). I have finished this
> version. It
> > > is
> > > > available for immediate posting, but I would like to wait for
> > > > agreement on the size issue (at least if that can be reached
> > > quickly).
> > > >
> > > > Rainer
> > > >
> > > > > -----Original Message-----
> > > > > From: syslog-sec-bounces@www.employees.org
> > > > > [mailto:syslog-sec-bounces@www.employees.org] On
> Behalf Of Anton
> >
> > > > > Okmianski
> > > > > Sent: Tuesday, October 12, 2004 11:25 PM
> > > > > To: syslog-sec@employees.org
> > > > > Subject: [Syslog-sec] UDP message size issue proposal
> > > > >
> > > > > Hi!
> > > > >
> > > > > Sorry for a long delay on this issue - I was on a 2 week
> > vacation.
> > > > I
> > > > > have spoken with a number of TCP/UDP/IP experts regarding
the
> > > sizing
> > > >
> > > > > issue. I am ready to propose the following changes:
> > > > >
> > > > > 1. Syslog-protocol will state that the max message will be
> > defined
> > > > by
> > > > > the transport layer.
> > > > >
> > > > > 2. Syslog-transport-udp will support messages up to
> maximum UDP
> > > > > datagram size of 64K. UDP is a very bad choice for
> large message
> >
> > > > > transmissions, so it does not make sense for us to
> stretch it by
> >
> > > > > adding our own fragmentation without other
> transmission control
> > > > > features such as found in TCP.
> > > > >
> > > > > 3. Syslog-transport-udp will rely on IP fragmentation and we
> > will
> > > > get
> > > > > rid of "proprietary" fragmentation which was designed
> to handle
> > > > > messages over 64K and deal with various
> > > > > non-compliant network hosts.
> > > > >
> > > > > 4. Syslog-transport-udp will recommend sending messages
within
> > the
> > >
> > > > > boundaries of prevalent MTU size on a given network
> to be safe.
> > It
> > >
> > > > > will recommend Ethernet's 1500 bytes less headers and
> will draw
> > > > reader
> > > > > attention to the minimum MTU size hosts on the network are
> > > required
> > > > to
> > > > > support for
> > > > > IPv6 (576 bytes) and IPv6 (1280 bytes).
> > > > >
> > > > > 5. Path MTU discovery may not work robustly and some TCP/IP
> > stacks
> > > > may
> > > > > not support UDP packets of full 64K size and truncate
> them. We
> > > will
> > > >
> > > > > mention this and bite this bullet.
> > > > > We should not restrict the protocol due to incompliant
> > > > implementations
> > > > > because it is a moving target and penalizes compliant
> > > > implementations
> > > > > with extra overhead. The above size recommendation would
> > partially
> > >
> > > > > deal with this, but leave the
> > > > > final choice up to the administrator.
> > > > >
> > > > > 6. We will get rid of most syslog transport headers for UDP
as
> > > they
> > > > > will no longer be needed. The only thing that will be left
is
> > the
> > >
> > > > > transport protocol version prefixed to every syslog message.
> > > Should
> > > >
> > > > > we even bother with that?
> > > > >
> > > > > This is a major change to the syslog-transport-udp.
> I'd like to
> > > get
> > > >
> > > > > positive feedback before I proceed with this update.
> > > > > The best part is that if we all agree on the above, the next
> > draft
> > >
> > > > > will be 1/3 of the size -- easier read for you. :)
> > > > >
> > > > > Thanks,
> > > > > Anton.
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Syslog-sec mailing list
> > > > > Syslog-sec@www.employees.org
> > > > > http://www.employees.org/mailman/listinfo/syslog-sec
> > > > >
> > > > _______________________________________________
> > > > Syslog-sec mailing list
> > > > Syslog-sec@www.employees.org
> > > > http://www.employees.org/mailman/listinfo/syslog-sec
> > > >
> > > >
> > > >
> > > _______________________________________________
> > > Syslog-sec mailing list
> > > Syslog-sec@www.employees.org
> > > http://www.employees.org/mailman/listinfo/syslog-sec
> > >
> > >
> > > _______________________________________________
> > > Syslog-sec mailing list
> > > Syslog-sec@www.employees.org
> > > http://www.employees.org/mailman/listinfo/syslog-sec
> > >
> >
> >
>
_______________________________________________
Syslog-sec mailing list
Syslog-sec@www.employees.org
http://www.employees.org/mailman/listinfo/syslog-sec