[Syslog-sec] syslog fragmentation

["Rainer Gerhards" <rgerhards@hq.adiscon.com>]

Hi WG,

I had some private conversation with Anton on the fragmentation, and as
this is now being discussed on-list, I would like to share my points
(Anton: sorry for some re-iteration).

My initial use case for fragmentation is this:
I *need* to support very occasional (1 out of 10,000 if not fewer)
message that get *really* large. I don't think it is appropriate that
any interim system needs to know this data all belongs to a single
message. So I modelled something (the multi-part messages now removed
from -protocol) that allows the sender to indicate that a message is
only part of a larger entity, but nobody except the 

a) sender
b) log ANALYZER (endpoint)

must be aware of the larger entity. This was meant as a way to ensure
that even the largest size message can be pushed to the ANALYZER without
the need for some interim system (even the recording syslogd!) to
actually know everything about it.

Actually, in this scheme each "fragment" of the multi-part message was a
valid syslog message in its own right. Only if somebody wanted/needed
the full message, that somebody could re-assemble it.

So this was meant to provide a way to say "hey, here are more than one
syslog message, but these all somehow belong together". Again, I was
concerned about the FINAL recipient, not (necessarily) an interim
system.

I went into that schema, because many people on-list told us that they
do not see any need at all for large messages (and in reality I very
seldomly see message larger than 128 bytes).

Anton, if I understood correctly, has a totally different use case, that
is he needs to actually feed a single message through all interim
systems and all of them must be aware of the full message. This, of
course forces the interim systems to re-assemble the message, which
obviously means more work to do and much more status keeping. The plus,
of course, is that each interim system can act on the full message and
base routing decision on it (if the administrator desires and the
implementation supports such decisions). As Anton pointed out, there
were also comments from others that such a functionality would be
desirable. As in this approach each system MUST re-assmble the message,
Anton obviously needs a low max message size, as otherwise an receiver
may be easily overrun. So the maximum message size (IPv4) that can
reliably be used is 507 byte, which will be sufficient in most cases. I
remember that we also talked about requiring a transport to support at
least 64K message size in -protocol (not yet in the draft). If so, I
think this will be sufficient in 99,999% of the cases. But if we do
that, we also force all -yet the minimalist - implementations to support
fragmentation (and if we don't, we actually have max size of 507 bytes
for "reliable" delivery).

Anyhow, even with 64K, some of my use cases are in the remaining 0,001%
range, so I need to find a way to use something like the previous
multi-part messaging mechanism in order to deliver my data. Actually,
this is not a big concern: it looks like a single-vendor issue and so we
will most probably go ahead and use a vendor extension for this. Right
now, I think we will actually use what formerly was in -protocol. I'd
preferred to do it in a standard way, but I think it is not a big deal
if a single vendor uses a specific approach for a quite uncommon
problem. After all, this is why extensibility has been build into
-protocol.

Other than that, I have to admit that I do not like the idea of interim
systems needing to re-assemble large-size message. Actually, I don't
like the necessary real-time reassembly at all. I am not sure if this is
in the simple spirit of syslog. This "feeling" is my remaining point on
the current approach. Of course, I can easily go over it - but maybe
somebody has a suggestion in this regard.

Rainer
_______________________________________________
Syslog-sec mailing list
Syslog-sec@www.employees.org
http://www.employees.org/mailman/listinfo/syslog-sec