[DGD] DGD Digest, Vol 106, Issue 2

Raymond Jennings shentino at gmail.com
Sun Oct 18 04:25:14 CEST 2015 

Can I say EWWWW btw?

For the short run I've managed to mitgate problems by delaying the
connection's closure as follows

1.  Return 1 in query_timeout
2.  Block the connection immediately
3.  Start a callout in system::userd that closes the connection by force in
0.1 seconds

Thanks for looking into this for me.  I had absolutely zero clue how to
even begin looking at this.

On Sat, Oct 17, 2015 at 3:34 AM, Dread Quixadhal <quixadhal at gmail.com>
wrote:

> There used to be a nice addon/extension for Firefox which let you see every
> HTTP request or response (and headers) sent or received by the browser.
> Chrome likely has something along those lines too.
>
> I would suggest using that in sync with logging on the server side.  It's
> far too easy to just assume the browser is doing what you THINK it should
> be doing.
>
> On Fri, Oct 16, 2015 at 11:02 PM, Raymond Jennings <shentino at gmail.com>
> wrote:
>
> > Hey list, just had an interesting conversation about RST packets tripping
> > up insta-close connections.
> >
> > Figured it would be useful for LPC dev, so here it is.
> >
> > On Fri, Oct 16, 2015 at 7:07 PM, Andrew Skalski <askalski at gmail.com>
> > wrote:
> >
> > > Sure, feel free to forward to the list.  (I'm not subscribed from this
> > > address, otherwise I would have replied there directly.)
> > >
> > > And yes, the RST is because the client sent data before the server
> could
> > > close the connection.
> > >
> > > If those other immediate closures you mention are for MUD connections,
> > > those are less of a concern because they are stream oriented, as
> opposed
> > to
> > > HTTP which is message-oriented.  Even if a RST happens, the user will
> see
> > > everything that was sent up until that point.  The only concern in that
> > > case is out-of-order delivery.  Also, not all MUD clients will try
> > sending
> > > data soon enough to trigger the issue (the exception would be clients
> > that
> > > immediately attempt to negotiate telnet options.)
> > >
> > > Andy Skalski
> > >
> > > On Fri, Oct 16, 2015 at 9:13 PM, Raymond Jennings <shentino at gmail.com>
> > > wrote:
> > >
> > >> I also use immediate closures to deal with people trying to connect
> when
> > >> the server is under maintenance (system suspended) or if the user
> table
> > is
> > >> too full (I save two slots in the table at all times, firstly to save
> a
> > >> port for the admin login, and secondly to purge and burn off any
> > >> connections piling up in the queue so that they don't hang)
> > >>
> > >> These along with the enforced sitebans are hard coded into system's
> > userd
> > >> and the connection managers asking to handle incoming connections are
> > not
> > >> even given a chance to interfere.
> > >>
> > >> So...basically, the client is getting an RST because the client's data
> > >> got "bounced" after the server closed the connection?
> > >>
> > >> On Fri, Oct 16, 2015 at 6:06 PM, Raymond Jennings <shentino at gmail.com
> >
> > >> wrote:
> > >>
> > >>> may I forward this conversation to the DGD list?  It sounds like
> > >>> important information.
> > >>>
> > >>> Also...EWW.
> > >>>
> > >>> On Fri, Oct 16, 2015 at 5:38 PM, Andrew Skalski <askalski at gmail.com>
> > >>> wrote:
> > >>>
> > >>>> I did some further testing, and was able to rule out NAT as a
> > >>>> contributing factor.
> > >>>>
> > >>>> It all boils down to the order which packets are received and
> > processed
> > >>>> by both peer.  Because the server never reads the HTTP request, it
> > will
> > >>>> send a RST, either (a) when close() is called, if the request was
> > received
> > >>>> already; or (b) immediately, if the request arrives after close().
> > Note
> > >>>> the RST is sent in both cases.
> > >>>>
> > >>>> It behaves intermittently because the following *all* must be true
> for
> > >>>> it to succeed:
> > >>>>
> > >>>>    1. The server responds and closes the socket *before* the kernel
> > >>>>    receives the HTTP request packet.
> > >>>>    2. All response packets, including the FIN must arrive *before*
> the
> > >>>>    client receives the RST.
> > >>>>
> > >>>> Ways to mitigate this:
> > >>>>
> > >>>>    1. Try to read the HTTP request.  Reading the entire request
> means
> > >>>>    no RST will be sent.  Even with a very short timeout (100
> > milliseconds),
> > >>>>    this should solve most of your problems.  The request typically
> > arrives
> > >>>>    immediately (same round-trip interval) after handshake.
> > >>>>    2. Shutdown the "write" half of the socket (SHUT_WR) before
> closing
> > >>>>    it.  This will ensure a FIN is always sent.  (I don't think DGD
> > exposes
> > >>>>    this system call, unfortunately.)
> > >>>>    3. Send a Content-Length header, so the client does not need to
> > >>>>    receive a FIN.  This may still be susceptible to out-of-order
> > delivery,
> > >>>>    because the RST may arrive before all response packets have been
> > received.
> > >>>>    Delaying briefly before closing the socket can help minimize
> > reordering
> > >>>>    (this might be mitigation #1 cleverly disguised as a "delay"...)
> > >>>>    4. Put DGD behind a HTTP proxy such as nginx and let it handle
> > >>>>    shielding you from abusive requests.
> > >>>>
> > >>>> Hope this helps,
> > >>>>
> > >>>> Andy Skalski
> > >>>>
> > >>>> On Fri, Oct 16, 2015 at 5:04 PM, Andrew Skalski <askalski at gmail.com
> >
> > >>>> wrote:
> > >>>>
> > >>>>> That shouldn't be necessary; I was just able to reproduce this a
> few
> > >>>>> minutes ago with a Perl script acting as webserver, running behind
> an
> > >>>>> OpenWRT router doing NAT.
> > >>>>>
> > >>>>> Adding a Content-Length header seems to help significantly.  Apache
> > >>>>> Bench still complains, but web browsers (and 'curl') so far haven't
> > given
> > >>>>> me any errors.
> > >>>>>
> > >>>>>   http://mighty.voltara.org:50080/   (original)
> > >>>>>   http://mighty.voltara.org:50081/   (with Content-Length)
> > >>>>>
> > >>>>> Andy Skalski
> > >>>>>
> > >>>>> On Fri, Oct 16, 2015 at 4:46 PM, Raymond Jennings <
> > shentino at gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Would you like to see the LPC source code involved?
> > >>>>>>
> > >>>>>> On Fri, Oct 16, 2015 at 1:43 PM, Andrew Skalski <
> askalski at gmail.com
> > >
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Oops, didn't mean to copy that to the entire list.  (I'm
> subscribed
> > >>>>>>> from a different address.)
> > >>>>>>>
> > >>>>>>> On Fri, Oct 16, 2015 at 4:40 PM, Andrew Skalski <
> > askalski at gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> I'm having trouble reproducing this in Chrome, except for one
> > >>>>>>>> thing: sometimes the "favicon.ico" request fails with a
> > >>>>>>>> net::ERR_CONNECTION_RESET error code.
> > >>>>>>>>
> > >>>>>>>> I had better luck using Apache Bench to generate concurrent
> > >>>>>>>> requests:
> > >>>>>>>>
> > >>>>>>>> $ ab -l -n4 -c2 http://shentino.mynetgear.com:50080/
> > >>>>>>>> This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
> > >>>>>>>> Copyright 1996 Adam Twiss, Zeus Technology Ltd,
> > >>>>>>>> http://www.zeustech.net/
> > >>>>>>>> Licensed to The Apache Software Foundation,
> > http://www.apache.org/
> > >>>>>>>>
> > >>>>>>>> Benchmarking shentino.mynetgear.com (be
> > >>>>>>>> patient)...apr_socket_recv: Connection reset by peer (104)
> > >>>>>>>> Total of 1 requests completed
> > >>>>>>>>
> > >>>>>>>> Looking at wireshark output, the issue *appears* to be happening
> > >>>>>>>> at the TCP layer, and depends on the timing of the request and
> > response.
> > >>>>>>>> This is mostly guesswork, because I don't have a server-side
> > packet capture
> > >>>>>>>> to compare it to:
> > >>>>>>>>
> > >>>>>>>>    - Browser sends "SYN"
> > >>>>>>>>    - Server receives "SYN", sends "SYN+ACK"
> > >>>>>>>>    - Browser receives "SYN+ACK", sends "ACK" (thus completing
> > >>>>>>>>    3-way handshake) immediately followed by HTTP request.
> > >>>>>>>>    - Server receives "ACK".
> > >>>>>>>>
> > >>>>>>>> At this point, one of two things happen:
> > >>>>>>>>
> > >>>>>>>>    - Server receives HTTP request.
> > >>>>>>>>    - Server sends HTTP response and "FIN" to close the
> connection.
> > >>>>>>>>    - Browser receives HTTP response and "FIN".
> > >>>>>>>>    - Transaction is succesful.
> > >>>>>>>>
> > >>>>>>>> Or...
> > >>>>>>>>
> > >>>>>>>>    - Server sends HTTP response and "FIN" to close the
> > connnection.
> > >>>>>>>>    - Server receives HTTP request.  Because the connection no
> > >>>>>>>>    longer exists, it responds with a "RST".
> > >>>>>>>>    - Browser receives "RST" before either the HTTP response or
> the
> > >>>>>>>>    FIN.
> > >>>>>>>>    - Failure: Browser throws a "connection reset by peer" error.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> This is generally what I suspect is happening.  If you have any
> > >>>>>>>> firewall or NAT devices in front of the server, they may also be
> > playing a
> > >>>>>>>> role.  For example, upon RST from the server, it may immediately
> > remove the
> > >>>>>>>> connection from its NAT table, and discard any frames it might
> > have queued
> > >>>>>>>> for transmit.
> > >>>>>>>>
> > >>>>>>>> One thing that might help (but not necessarily solve the issue)
> is
> > >>>>>>>> to send a Content-Length header.  Without Content-Length, the
> > browser must
> > >>>>>>>> wait for the TCP connection to close (FIN) before it considers
> > the response
> > >>>>>>>> to be complete and successful.  Abnormal termination (RST)
> > prevents this
> > >>>>>>>> from happening.  However, if the server sent a Content-Length
> > header, the
> > >>>>>>>> browser can consider the response complete as soon as it
> receives
> > the
> > >>>>>>>> specified number of bytes.
> > >>>>>>>>
> > >>>>>>>> By the way, another thing I noticed is the HTTP response headers
> > >>>>>>>> are line-terminated improperly.  Header lines (including the
> > final, empty
> > >>>>>>>> line) are required to be CRLF terminated.  (This rule does not
> > apply to the
> > >>>>>>>> response body.)  I doubt this has anything to do with the issue
> > you're
> > >>>>>>>> having, however.
> > >>>>>>>>
> > >>>>>>>> Andy Skalski
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Oct 16, 2015 at 2:50 PM, Raymond Jennings <
> > >>>>>>>> shentino at gmail.com> wrote:
> > >>>>>>>>
> > >>>>>>>>> The site is shentino.mynetgear.com, http is port 50080
> > >>>>>>>>>
> > >>>>>>>>> My suspicious are a race condition of sorts that causes things
> to
> > >>>>>>>>> choke.  Hammering it with telnet repeatedly reliably and
> prompty
> > gets back
> > >>>>>>>>> the error page.
> > >>>>>>>>>
> > >>>>>>>>> If you can find out anything by testing I would be quite
> > intrigued.
> > >>>>>>>>>
> > >>>>>>>>> On Fri, Oct 16, 2015 at 7:47 AM, Andrew Skalski <
> > >>>>>>>>> askalski at gmail.com> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> I'd be interested to see this in action.  My IP is
> > 68.133.31.149,
> > >>>>>>>>>> so
> > >>>>>>>>>> if you siteban that and give me a URL to hit, I can see if I
> can
> > >>>>>>>>>> reproduce it either in Chromium or Chrome (Linux), then use
> > >>>>>>>>>> tcpdump to
> > >>>>>>>>>> see what's going wrong.
> > >>>>>>>>>>
> > >>>>>>>>>> (Assuming you haven't figured it out already.)
> > >>>>>>>>>>
> > >>>>>>>>>> Andy Skalski
> > >>>>>>>>>>
> > >>>>>>>>>> >
> > >>>>>>>>>>
> > ----------------------------------------------------------------------
> > >>>>>>>>>> >
> > >>>>>>>>>> > Message: 1
> > >>>>>>>>>> > Date: Thu, 15 Oct 2015 19:52:43 -0700
> > >>>>>>>>>> > From: Raymond Jennings <shentino at gmail.com>
> > >>>>>>>>>> > To: "All about Dworkin's Game Driver" <dgd at dworkin.nl>
> > >>>>>>>>>> > Subject: [DGD] instant replies to HTTP requests confusing
> > >>>>>>>>>> browser?
> > >>>>>>>>>> > Message-ID:
> > >>>>>>>>>> > <
> > >>>>>>>>>>
> > CAGDaZ_rOUJrg3f+_MG9qao2+rHegxAiq-OLABwxLFv4BVPu4WQ at mail.gmail.com
> > >>>>>>>>>> >
> > >>>>>>>>>> > Content-Type: text/plain; charset=UTF-8
> > >>>>>>>>>> >
> > >>>>>>>>>> > Taking a page out of skotos's book, I added an Http module
> to
> > >>>>>>>>>> kotaka.
> > >>>>>>>>>> >
> > >>>>>>>>>> > I'm having some problems though after having my system userd
> > >>>>>>>>>> check a
> > >>>>>>>>>> > connection's ip against the siteban manager before passing
> the
> > >>>>>>>>>> connection
> > >>>>>>>>>> > to the http userd.
> > >>>>>>>>>> >
> > >>>>>>>>>> > If the IP is sitebanned, the system userd merely asks the
> http
> > >>>>>>>>>> userd for a
> > >>>>>>>>>> > siteban message and then returns that before closing the
> > >>>>>>>>>> connection
> > >>>>>>>>>> > immediately without even waiting for a request.
> > >>>>>>>>>> >
> > >>>>>>>>>> > Sometimes the browser shows the error page, and sometimes
> > shows
> > >>>>>>>>>> nothing at
> > >>>>>>>>>> > all.
> > >>>>>>>>>> >
> > >>>>>>>>>> > Is there any reason this is going on?  Is it buggy behavior
> in
> > >>>>>>>>>> chrome or am
> > >>>>>>>>>> > I violating HTTP/1.1 somehow?
> > >>>>>>>>>> >
> > >>>>>>>>>> >
> > >>>>>>>>>> > ------------------------------
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >
> > ____________________________________________
> > https://mail.dworkin.nl/mailman/listinfo/dgd
> >
> ____________________________________________
> https://mail.dworkin.nl/mailman/listinfo/dgd

More information about the DGD mailing list