[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [Syslog-sec] I-D ACTION:draft-ietf-syslog-transport-udp-02.txt
John and all:
First of all, thanks for feeding in the requirements of your project.
I think all of three of them (32k message size, UTF-8 and control
characters) are met by the current drafts. The only issue you raise
is fragmentation.
I dedicate Appendix A in transport-udp draft to "Rational For
Transport Message Size Restrictions", which also explains the reason
for fragmentation. I will try not to repeat everything here, but just
recap and offer one resolution for your concern.
I think that syslog-protocol should have a fairly large message limit.
Where there is a requirement for 32k, we can surely anticipate 128k in
a few years. We could argue on the merits of such large messages and
use of UDP to transmit them, but really it is up to application
developers and administrators to make the call on message appropriate
message size and transport. The next question is how does
transport-udp support messages of this size.
IP/UDP, by definition, does not support datagrams greater than 64k.
And with UDP, obviously, there is no TCP segmentation to help in
partitioning larger messages. So, IP fragmentation won't help us if
we were to deal with messages greater than 64k, anyway. This is the
reason why protocols such as TFTP or TCP do they own partitioning of
messages.
If we assume that 64k max message size is enough (a big if given
people's sentiments on this list in the past), then the next question
is whether or not we can trust the IP layer to fragment it using Path
MTU discovery (or static MTU configuration) or not. The research I
quoted in the aforementioned appendix shows that many UDP stack
implementations do not support full UDP packet size at all. Plus,
Path MTU discovery is often not available and you simply risk losing
packets with size greater than MTU on a lot of networks. This is why
established protocols such as DHCP, DNS, TFTP and RIP all limit UDP
packet size to somewhere in the area of 480-512 bytes.
My opinion is that for huge messages TCP is much better suited. But
if somebody does indeed want to do it with UDP, they will have to use
an implementation which supports fragmentation. The transport-udp
draft defines a standard mechanism for sending pretty much any size
message over UDP. Either way, an application developer will be
shielded from doing fragmentation and re-assembly if he/she uses an
appropriate syslog library.
What I sense people don't like is that with current transport-udp
draft they have to implement fragmentation in syslog library for
messages over ~480 bytes (in IPv4). This may not be needed for them
if they know that their network and all hosts support larger UDP
packets and that they don't send messages over certain size. Maybe we
should consider removing the restriction on packet size where
fragmentation kicks in and make it a recommendation. This way somebody
could send messages of up to ~64k without fragmentation using
transport-udp draft if they know what they are doing. Would this
address your issue? What do others think?
Thanks,
Anton.
> -----Original Message-----
> From: Moehrke, John (GE Healthcare) [mailto:John.Moehrke@med.ge.com]
> Sent: Friday, September 10, 2004 9:57 AM
> To: Rainer Gerhards; ietfdbh@comcast.net; Anton Okmianski;
> syslog-sec@employees.org
> Subject: RE: [Syslog-sec] I-D
> ACTION:draft-ietf-syslog-transport-udp-02.txt
>
>
> I truly don't understand why syslog cares about max-size as
> defined in IP. It can say: "Due to a protocol binding to udp,
> big messages may get fragmented. This fragmentation
> increases the risk of the message not getting delivered." The
> only reason for syslog to use any SHOULD/MUST language on
> this subject is to assure a syslog user that a 'syslog'
> implementation will be able to create messages with a payload
> of n size, and that if this message gets through the gauntlet
> that is udp/ip, it will be consumed without buffer size
> rejects. Going over UDP, you can't give any form of assurance
> that it will be delivered. This lack of assurance is simply
> made worse if the udp packet is big.
>
> On the max-size issue... What IHE is concerned with is
> profiling a use-case that is forbidden by standards or not
> implementable given current systems. Thus we are more
> interested in the syslog-max-size, than the assurance issue.
> If we truly can't expect a syslog implementation (client and
> server) to handle a message bigger than 480 octets; then we
> must abandon the use of udp-syslog. We have no choice but to
> go with Reliable syslog. We will have to go to
> Reliable-syslog, not for the reliability aspect, but because
> we know our payload won't be rejected simply because it is
> big. In our design of the schema we understood that a message
> might not get there if we are bound to udp-syslog. We
> understood that the larger the message the bigger this
> un-reliability is.
>
> OSI layers are GOOD...
>
> John
>
> -----Original Message-----
> From: syslog-sec-bounces@willers.employees.org
> [mailto:syslog-sec-bounces@willers.employees.org] On Behalf
> Of Rainer Gerhards
> Sent: Friday, September 10, 2004 8:19 AM
> To: ietfdbh@comcast.net; Anton Okmianski; syslog-sec@employees.org
> Subject: RE: [Syslog-sec] I-D
> ACTION:draft-ietf-syslog-transport-udp-02.txt
>
> Hi List,
>
> I think there is a lot of wisdom and obviously a lot of
> experience in what David says. So even though I have strong
> concerns that a guaranteed size of just 480 octets is not
> wise, I would accept it if nobody else objects violently.
>
> Rainer
>
> > -----Original Message-----
> > From: David B Harrington [mailto:ietfdbh@comcast.net]
> > Sent: Friday, September 10, 2004 3:06 PM
> > To: Rainer Gerhards; 'Anton Okmianski'; syslog-sec@employees.org
> > Subject: RE: [Syslog-sec] I-D
> > ACTION:draft-ietf-syslog-transport-udp-02.txt
> >
> > Hi,
> >
> > Please keep in mind that MUST is used for an absolute
> requirement of
> > the specification, i.e. the specification won't work if you don't
> > support this. MUST should NOT be used to set some arbitrary size
> > selected by the authors as looking like a nice round number
> that could
> > be useful in many circumstances. It sounds like you are trying to
> > create a MUST.
> >
> > I recommend that you set the 480 as a RECOMMENDED minimum size,
but
> > also RECOMMEND 16k or 64k be supported to make it possible to send
> > larger messages. You could also identify some expected use
> cases, such
> > as the IHE case to demonstrate that potential users might expect
> > support for larger sizes.
> >
> > You express a concern that if 480 is the recommended minimum, then
> > implementors will only implement that size, not a more useful
size.
> > The market can decide whether those implementations that
> only support
> > 480 will actually be used widely. I suspect only free
> > 480-implementations would get much usage, and you get what you pay
> > for. Better implementations, both free and commercial, will
> probably
> > be preferred by operators, and therefore vendors. It
> shouldn't be the
> > author's decision what sizes MUST be supported based on their own
> > preferences and their company's use cases.
> >
> >
> > Dbh
> >
> > -----Original Message-----
> > From: syslog-sec-bounces@www.employees.org
> > [mailto:syslog-sec-bounces@www.employees.org] On Behalf Of Rainer
> > Gerhards
> > Sent: Friday, September 10, 2004 4:10 AM
> > To: Anton Okmianski; syslog-sec@employees.org
> > Subject: RE: [Syslog-sec] I-D
> > ACTION:draft-ietf-syslog-transport-udp-02.txt
> >
> > Anton,
> >
> > > > - In 3.4, the draft says:
> > > >
> > > > ######
> > > > All implementations MUST support sending and receiving
syslog
> > > > messages up to and including the size which does not
require
> > > > fragmentation (507 bytes for IPv4 and 1191 bytes for IPv6).
> > This
> > > > size excludes the overhead of the syslog transport and
UDP/IP
> > > > headers. Support for larger messages is encouraged.
> > > Implementors
> > > > SHOULD clearly state the maximum supported message size in
> > > > documentation.
> > > > ######
> > > >
> > > > Actually, all implementations MUST support a message size of
> > 65,000
> > > > bytes, because this is the required minimum in -protocol 4.1.
We
> > > > agreed on this minimum after long discussions on the
> message size.
> >
> > > > I'd actually prefer not to re-iterate this. But if we
> must, let me
> >
> > > > say that a guaranteed message size of 507 bytes looks
> far too low
> > to
> > > > me.
> > > >
> > > > Please note that the -protcol requirement meant that all
> > > > implementations MUST support fragmentation.
> > > >
> > > > Sorry I didn't spot this in the -01 version...
> > > >
> > > > I've also noticed that in -protocol, I have limited that max
> > message
> > > > size to 16,000,000 bytes in -protocol. It might make
> sense to sync
> >
> > > > this. I can go up with my number.
> > >
> > > Indeed we discussed this at length, but my recollection of
> > conclusions
> > > was different. :) Let's revisit. We decided (I thought) that
we
> > > can't force transport implementors into supporting
> fragmentation as
> > it
> > > was too complicated. Plus, we can't them into supporting
> the maximum
> >
> > > message size as the common denominator may be too small.
> >
> > I've tried to find something definitive in the mailing list
> archive.
> > Actually, I couldn't come up with such a thing... So it looks like
> > there were some private conversations or we both had a wrong
> > understanding of concensus. Anyhow, let's try to finalize it now.
> >
> > I thought that the concensus was that we can not force
> implementors to
> > support very large messages. Thus, the minimum-to-be-supported
size
> > should be just 64K. Please remember that the IHE healthcare
> initiative
> > tries to use syslog for auditing and has a size requirement of at
> > least 32K. So there is some industry need for larger size
messages.
> > >
> > > Hence, only the size that can be transported without
fragmentation
> > > must be supported by transport implementors.
> >
> > This means we are going down with the actual message limit from 1K
> > (which already was very small) to just 480 octets. Given the
> > additional overhead we have introduced, there is not much
> left for the
> > acutal message.
> >
> > Of course, you can argue that it is just the minimum size
> that MUST be
> > supported. But I think if we face reality, that minimum will soon
> > become the maximum supported by most implementations. I am very
> > concerend that such a short guaranteed message size really makes
> > sense... If we really take that route, we should eventually be
bold
> > enough to go back to 1K without fragmentation, even though we have
> > some reliability concerns. At least that seems to have worked for
> > several years now. I don't think we will get much acceptance for a
> > standard that leaves around 400 bytes for the actual message (not
> > counting meta data in structured ids....).
> >
> > > However, a full
> > > transport implementation will support fragmentation and size of
up
> > to
> > > 16777216 bytes.
> > >
> > > I think what we need is a statement in -protocol that it
> supports a
> > > message size up to 16777216, but transports MAY impose a
> different
> > > limit.
> >
> > That's already there ;)
> >
> > >
> > > It was mentioned that some devices will physically not be able
to
> > > handle large messages, where "large" can be quite a small
number.
> > > Restricting all messages to this common denominator is
> not good, so
> > we
> > > have a multi-tier support system. People are not likely to
support
> > > streaming of message parts in their implementations, so
available
> > > memory will be an issue.
> > >
> > > Administrators will choose the appropriate implementations.
Should
> > we
> > > add a statement that transport implementations MUST
> document maximum
> >
> > > supported message size?
> > >
> > > I know it is less than optimal, but requiring all transports to
> > > support some huge size is not a good idea either IMO.
> > >
> > > You also had an idea for doing fragmentation within the
> -protocol by
> >
> > > upper layer. I originally liked it. But later people
> commented that
> >
> > > this was improper layering and shifted responsibilities
> of transport
> >
> > > layer up the stack. I agree with this. It would have
introduced
> > > potential fragmentation on several layers and we would have
still
> > > forced the higher layer to deal with UDP sizing constraints.
> > >
> > > I agree this issue deserves discussion. Let's have it and
> put it bed
> >
> > > for good.
> > >
> > > > I do not fully understand what you mean in 4.2 by:
> > > >
> > > > #####
> > > > No concurrent port reuse on the same host is
> > > > allowed.
> > > > #####
> > > >
> > > > It's probably just my English, but I do not get it. I'd
> appreciate
> >
> > > > if you could elaborate a little.
> > >
> > > What this refers to is the use of SO_REUSEPORT socket option
which
> > > allows multiple sockets to be bound to the same port. I guess
it
> > > would not be a problem if the two different consumers of
> the socket
> > > (maybe threads) use different MessageId numbers. And
> this is stated
> >
> > > as requirement elsewhere: all message parts must be send using
the
> > > same MessageId and source port among other things.
> > >
> > > Following your suggestion, we already allow multiple
> different ports
> >
> > > to be used concurrent within a single process, so I have
> no problem
> > > with removing this restriction. It is too
implementation-specific
> > > anyway. What do you think?
> >
> > I agree. I simply did not fully understand what you were saying ;)
> > Eventually the sentence could be a bit more verbose - but
> this might
> > just have been my personal problem ;)
> >
> > Rainer
> > _______________________________________________
> > Syslog-sec mailing list
> > Syslog-sec@www.employees.org
> > http://www.employees.org/mailman/listinfo/syslog-sec
> >
> >
> >
> _______________________________________________
> Syslog-sec mailing list
> Syslog-sec@www.employees.org
> http://www.employees.org/mailman/listinfo/syslog-sec
>
_______________________________________________
Syslog-sec mailing list
Syslog-sec@www.employees.org
http://www.employees.org/mailman/listinfo/syslog-sec