[DGD] DGD Digest, Vol 106, Issue 2
Dread Quixadhal
quixadhal at gmail.com
Sat Oct 17 12:34:25 CEST 2015
There used to be a nice addon/extension for Firefox which let you see every
HTTP request or response (and headers) sent or received by the browser.
Chrome likely has something along those lines too.
I would suggest using that in sync with logging on the server side. It's
far too easy to just assume the browser is doing what you THINK it should
be doing.
On Fri, Oct 16, 2015 at 11:02 PM, Raymond Jennings <shentino at gmail.com>
wrote:
> Hey list, just had an interesting conversation about RST packets tripping
> up insta-close connections.
>
> Figured it would be useful for LPC dev, so here it is.
>
> On Fri, Oct 16, 2015 at 7:07 PM, Andrew Skalski <askalski at gmail.com>
> wrote:
>
> > Sure, feel free to forward to the list. (I'm not subscribed from this
> > address, otherwise I would have replied there directly.)
> >
> > And yes, the RST is because the client sent data before the server could
> > close the connection.
> >
> > If those other immediate closures you mention are for MUD connections,
> > those are less of a concern because they are stream oriented, as opposed
> to
> > HTTP which is message-oriented. Even if a RST happens, the user will see
> > everything that was sent up until that point. The only concern in that
> > case is out-of-order delivery. Also, not all MUD clients will try
> sending
> > data soon enough to trigger the issue (the exception would be clients
> that
> > immediately attempt to negotiate telnet options.)
> >
> > Andy Skalski
> >
> > On Fri, Oct 16, 2015 at 9:13 PM, Raymond Jennings <shentino at gmail.com>
> > wrote:
> >
> >> I also use immediate closures to deal with people trying to connect when
> >> the server is under maintenance (system suspended) or if the user table
> is
> >> too full (I save two slots in the table at all times, firstly to save a
> >> port for the admin login, and secondly to purge and burn off any
> >> connections piling up in the queue so that they don't hang)
> >>
> >> These along with the enforced sitebans are hard coded into system's
> userd
> >> and the connection managers asking to handle incoming connections are
> not
> >> even given a chance to interfere.
> >>
> >> So...basically, the client is getting an RST because the client's data
> >> got "bounced" after the server closed the connection?
> >>
> >> On Fri, Oct 16, 2015 at 6:06 PM, Raymond Jennings <shentino at gmail.com>
> >> wrote:
> >>
> >>> may I forward this conversation to the DGD list? It sounds like
> >>> important information.
> >>>
> >>> Also...EWW.
> >>>
> >>> On Fri, Oct 16, 2015 at 5:38 PM, Andrew Skalski <askalski at gmail.com>
> >>> wrote:
> >>>
> >>>> I did some further testing, and was able to rule out NAT as a
> >>>> contributing factor.
> >>>>
> >>>> It all boils down to the order which packets are received and
> processed
> >>>> by both peer. Because the server never reads the HTTP request, it
> will
> >>>> send a RST, either (a) when close() is called, if the request was
> received
> >>>> already; or (b) immediately, if the request arrives after close().
> Note
> >>>> the RST is sent in both cases.
> >>>>
> >>>> It behaves intermittently because the following *all* must be true for
> >>>> it to succeed:
> >>>>
> >>>> 1. The server responds and closes the socket *before* the kernel
> >>>> receives the HTTP request packet.
> >>>> 2. All response packets, including the FIN must arrive *before* the
> >>>> client receives the RST.
> >>>>
> >>>> Ways to mitigate this:
> >>>>
> >>>> 1. Try to read the HTTP request. Reading the entire request means
> >>>> no RST will be sent. Even with a very short timeout (100
> milliseconds),
> >>>> this should solve most of your problems. The request typically
> arrives
> >>>> immediately (same round-trip interval) after handshake.
> >>>> 2. Shutdown the "write" half of the socket (SHUT_WR) before closing
> >>>> it. This will ensure a FIN is always sent. (I don't think DGD
> exposes
> >>>> this system call, unfortunately.)
> >>>> 3. Send a Content-Length header, so the client does not need to
> >>>> receive a FIN. This may still be susceptible to out-of-order
> delivery,
> >>>> because the RST may arrive before all response packets have been
> received.
> >>>> Delaying briefly before closing the socket can help minimize
> reordering
> >>>> (this might be mitigation #1 cleverly disguised as a "delay"...)
> >>>> 4. Put DGD behind a HTTP proxy such as nginx and let it handle
> >>>> shielding you from abusive requests.
> >>>>
> >>>> Hope this helps,
> >>>>
> >>>> Andy Skalski
> >>>>
> >>>> On Fri, Oct 16, 2015 at 5:04 PM, Andrew Skalski <askalski at gmail.com>
> >>>> wrote:
> >>>>
> >>>>> That shouldn't be necessary; I was just able to reproduce this a few
> >>>>> minutes ago with a Perl script acting as webserver, running behind an
> >>>>> OpenWRT router doing NAT.
> >>>>>
> >>>>> Adding a Content-Length header seems to help significantly. Apache
> >>>>> Bench still complains, but web browsers (and 'curl') so far haven't
> given
> >>>>> me any errors.
> >>>>>
> >>>>> http://mighty.voltara.org:50080/ (original)
> >>>>> http://mighty.voltara.org:50081/ (with Content-Length)
> >>>>>
> >>>>> Andy Skalski
> >>>>>
> >>>>> On Fri, Oct 16, 2015 at 4:46 PM, Raymond Jennings <
> shentino at gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Would you like to see the LPC source code involved?
> >>>>>>
> >>>>>> On Fri, Oct 16, 2015 at 1:43 PM, Andrew Skalski <askalski at gmail.com
> >
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Oops, didn't mean to copy that to the entire list. (I'm subscribed
> >>>>>>> from a different address.)
> >>>>>>>
> >>>>>>> On Fri, Oct 16, 2015 at 4:40 PM, Andrew Skalski <
> askalski at gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> I'm having trouble reproducing this in Chrome, except for one
> >>>>>>>> thing: sometimes the "favicon.ico" request fails with a
> >>>>>>>> net::ERR_CONNECTION_RESET error code.
> >>>>>>>>
> >>>>>>>> I had better luck using Apache Bench to generate concurrent
> >>>>>>>> requests:
> >>>>>>>>
> >>>>>>>> $ ab -l -n4 -c2 http://shentino.mynetgear.com:50080/
> >>>>>>>> This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
> >>>>>>>> Copyright 1996 Adam Twiss, Zeus Technology Ltd,
> >>>>>>>> http://www.zeustech.net/
> >>>>>>>> Licensed to The Apache Software Foundation,
> http://www.apache.org/
> >>>>>>>>
> >>>>>>>> Benchmarking shentino.mynetgear.com (be
> >>>>>>>> patient)...apr_socket_recv: Connection reset by peer (104)
> >>>>>>>> Total of 1 requests completed
> >>>>>>>>
> >>>>>>>> Looking at wireshark output, the issue *appears* to be happening
> >>>>>>>> at the TCP layer, and depends on the timing of the request and
> response.
> >>>>>>>> This is mostly guesswork, because I don't have a server-side
> packet capture
> >>>>>>>> to compare it to:
> >>>>>>>>
> >>>>>>>> - Browser sends "SYN"
> >>>>>>>> - Server receives "SYN", sends "SYN+ACK"
> >>>>>>>> - Browser receives "SYN+ACK", sends "ACK" (thus completing
> >>>>>>>> 3-way handshake) immediately followed by HTTP request.
> >>>>>>>> - Server receives "ACK".
> >>>>>>>>
> >>>>>>>> At this point, one of two things happen:
> >>>>>>>>
> >>>>>>>> - Server receives HTTP request.
> >>>>>>>> - Server sends HTTP response and "FIN" to close the connection.
> >>>>>>>> - Browser receives HTTP response and "FIN".
> >>>>>>>> - Transaction is succesful.
> >>>>>>>>
> >>>>>>>> Or...
> >>>>>>>>
> >>>>>>>> - Server sends HTTP response and "FIN" to close the
> connnection.
> >>>>>>>> - Server receives HTTP request. Because the connection no
> >>>>>>>> longer exists, it responds with a "RST".
> >>>>>>>> - Browser receives "RST" before either the HTTP response or the
> >>>>>>>> FIN.
> >>>>>>>> - Failure: Browser throws a "connection reset by peer" error.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> This is generally what I suspect is happening. If you have any
> >>>>>>>> firewall or NAT devices in front of the server, they may also be
> playing a
> >>>>>>>> role. For example, upon RST from the server, it may immediately
> remove the
> >>>>>>>> connection from its NAT table, and discard any frames it might
> have queued
> >>>>>>>> for transmit.
> >>>>>>>>
> >>>>>>>> One thing that might help (but not necessarily solve the issue) is
> >>>>>>>> to send a Content-Length header. Without Content-Length, the
> browser must
> >>>>>>>> wait for the TCP connection to close (FIN) before it considers
> the response
> >>>>>>>> to be complete and successful. Abnormal termination (RST)
> prevents this
> >>>>>>>> from happening. However, if the server sent a Content-Length
> header, the
> >>>>>>>> browser can consider the response complete as soon as it receives
> the
> >>>>>>>> specified number of bytes.
> >>>>>>>>
> >>>>>>>> By the way, another thing I noticed is the HTTP response headers
> >>>>>>>> are line-terminated improperly. Header lines (including the
> final, empty
> >>>>>>>> line) are required to be CRLF terminated. (This rule does not
> apply to the
> >>>>>>>> response body.) I doubt this has anything to do with the issue
> you're
> >>>>>>>> having, however.
> >>>>>>>>
> >>>>>>>> Andy Skalski
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Oct 16, 2015 at 2:50 PM, Raymond Jennings <
> >>>>>>>> shentino at gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> The site is shentino.mynetgear.com, http is port 50080
> >>>>>>>>>
> >>>>>>>>> My suspicious are a race condition of sorts that causes things to
> >>>>>>>>> choke. Hammering it with telnet repeatedly reliably and prompty
> gets back
> >>>>>>>>> the error page.
> >>>>>>>>>
> >>>>>>>>> If you can find out anything by testing I would be quite
> intrigued.
> >>>>>>>>>
> >>>>>>>>> On Fri, Oct 16, 2015 at 7:47 AM, Andrew Skalski <
> >>>>>>>>> askalski at gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> I'd be interested to see this in action. My IP is
> 68.133.31.149,
> >>>>>>>>>> so
> >>>>>>>>>> if you siteban that and give me a URL to hit, I can see if I can
> >>>>>>>>>> reproduce it either in Chromium or Chrome (Linux), then use
> >>>>>>>>>> tcpdump to
> >>>>>>>>>> see what's going wrong.
> >>>>>>>>>>
> >>>>>>>>>> (Assuming you haven't figured it out already.)
> >>>>>>>>>>
> >>>>>>>>>> Andy Skalski
> >>>>>>>>>>
> >>>>>>>>>> >
> >>>>>>>>>>
> ----------------------------------------------------------------------
> >>>>>>>>>> >
> >>>>>>>>>> > Message: 1
> >>>>>>>>>> > Date: Thu, 15 Oct 2015 19:52:43 -0700
> >>>>>>>>>> > From: Raymond Jennings <shentino at gmail.com>
> >>>>>>>>>> > To: "All about Dworkin's Game Driver" <dgd at dworkin.nl>
> >>>>>>>>>> > Subject: [DGD] instant replies to HTTP requests confusing
> >>>>>>>>>> browser?
> >>>>>>>>>> > Message-ID:
> >>>>>>>>>> > <
> >>>>>>>>>>
> CAGDaZ_rOUJrg3f+_MG9qao2+rHegxAiq-OLABwxLFv4BVPu4WQ at mail.gmail.com
> >>>>>>>>>> >
> >>>>>>>>>> > Content-Type: text/plain; charset=UTF-8
> >>>>>>>>>> >
> >>>>>>>>>> > Taking a page out of skotos's book, I added an Http module to
> >>>>>>>>>> kotaka.
> >>>>>>>>>> >
> >>>>>>>>>> > I'm having some problems though after having my system userd
> >>>>>>>>>> check a
> >>>>>>>>>> > connection's ip against the siteban manager before passing the
> >>>>>>>>>> connection
> >>>>>>>>>> > to the http userd.
> >>>>>>>>>> >
> >>>>>>>>>> > If the IP is sitebanned, the system userd merely asks the http
> >>>>>>>>>> userd for a
> >>>>>>>>>> > siteban message and then returns that before closing the
> >>>>>>>>>> connection
> >>>>>>>>>> > immediately without even waiting for a request.
> >>>>>>>>>> >
> >>>>>>>>>> > Sometimes the browser shows the error page, and sometimes
> shows
> >>>>>>>>>> nothing at
> >>>>>>>>>> > all.
> >>>>>>>>>> >
> >>>>>>>>>> > Is there any reason this is going on? Is it buggy behavior in
> >>>>>>>>>> chrome or am
> >>>>>>>>>> > I violating HTTP/1.1 somehow?
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> > ------------------------------
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
> ____________________________________________
> https://mail.dworkin.nl/mailman/listinfo/dgd
>
More information about the DGD
mailing list