Heimdal 1.1 telnetd broken under Solaris 10

[Harald Barth <haba@kth.se>]

When tested under Solaris 10, telnetd in Heimdal 1.0.x produces
gibberish in the face of the user, probably some encrypted data.
My hope was that this was fixed in 1.1, but no, so Ian and I had
to fix it ourselves.

The problem is, that under Solaris 10, the configure test for
TIOCSCTTY produces "true", however the call to ioctl(t, TIOCSCTTY,
(char *)0) fails resulting in a call to fatalerror(). That is not
taking down the connection properly. Therefore the gibberish.

However, Solaris 10's terminal implementation is STREAMSPTY, which
means that ptem and ldterm are allready pushed on the stream and the
return value of the TIOCSCTTY ioctl can be ignored. This may even be
true for HPUX, which could remove the !defined(__hpux) from the #ifdef
hell, but I can't test that. Under Solaris 10, telnetd works with the
patch below. I did leave the call to TIOCSCTTY in because there _may_
be architectures that define STREAMSPTY and TIOCSCTTY where the call
to TIOCSCTTY does something usefull.

This still does not address the "not clean shutdown" issue when calling
fatalerror(), but gives a working telnetd.

Harald.

Patch (may not be whitespace compat with the default Heimdal ident rules):

--- heimdal-1.1/appl/telnet/telnetd/sys_term.c.orig     Thu Jan 24 14:10:17 2008
+++ heimdal-1.1/appl/telnet/telnetd/sys_term.c  Fri Jan 25 08:40:57 2008
@@ -1009,8 +1009,13 @@
 int login_tty(int t)
 {
 # if defined(TIOCSCTTY) && !defined(__hpux)
-    if (ioctl(t, TIOCSCTTY, (char *)0) < 0)
-       fatalperror(net, "ioctl(sctty)");
+    if (ioctl(t, TIOCSCTTY, (char *)0) < 0) 
+#  if defined(STREAMSPTY)
+        ; /* not fatal when t is a STREAMSPTY, ignore */
+#  else
+        fatalperror(net, "ioctl(sctty)");
+#  endif
+
 #  ifdef _CRAY
     /*
      * Close the hard fd to /dev/ttypXXX, and re-open through