Partial downloads not supported leading to multiple requests from download managers
Description
is related to
Activity
Stephen Marquard January 4, 2008 at 12:09 PM
Verified that Accept-Ranges header is output on 2-5-x production build.
Mike Osterman November 13, 2007 at 2:55 PM
Merged to 2.4.x (r38150) and 2.5.x (r38151)
Jim Eng November 13, 2007 at 1:34 PM
Please merge to 2.4.x and 2.5.x. This is a low-risk change (according to the http/1.1 spec). There is no way to reproduce this problem reliably, so there is no way to verify the fix.
Jim Eng November 13, 2007 at 1:09 PM
Just to summarize (to see if I understand correctly): We cannot reproduce the problem, but we believe it may be repeated if somebody uses a particular version of a particular "user agent" (may be a browser or similar tool). It's possible that the problem may be avoided if we set a particular property in returning the bytes, and that change will not have any adverse side-effects. Is that correct?
Stephen Marquard October 21, 2007 at 8:18 AM
So far it doesn't look like Sakai is doing anything wrong. Servers that support partial (range) requests output
Accept-Ranges: bytes
in the response header for a GET request which the /access/ servlet doesn't. However, as per:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.5
we should probably output
Accept-Ranges: none
to be unambiguous until such time as partial requests are handled, e.g.:
Index: content-impl/impl/src/java/org/sakaiproject/content/impl/BaseContentService.java
===================================================================
— content-impl/impl/src/java/org/sakaiproject/content/impl/BaseContentService.java (revision 37151)
+++ content-impl/impl/src/java/org/sakaiproject/content/impl/BaseContentService.java (working copy)
@@ -6109,6 +6109,7 @@
res.setContentType(contentType);
res.addHeader("Content-Disposition", disposition);
+ res.addHeader("Accept-Ranges", "none");
res.setContentLength(len);
// set the buffer of the response to match what we are reading from the request
However, it's still not clear what made this particular FDM 2.x client keep re-requesting the same file (it requested it 416,457 times).
We have seen one particular download manager, identified in the user-agent as "FDM 2.x" (http://www.freedownloadmanager.org/) generate multiple http requests (thousands) for the same file, e.g. a small sample here from a client which has been generating 28 requests/second for the same file:
It appears (though some further investigation is necessary) that there is some protcol misunderstanding between FDM 2.x and Sakai's content hosting (access servlet) which is leading to these repeat requests. A high enough voulme of these or many clients doing this at the same time could lead to a denial of service through excessive traffic.
[20/Oct/2007:17:29:14 +0200] 137.158.108.116 TLSv1 RC4-MD5 "GET /access/content/group/6a058395-ce47-4dc6-004f-c56075774e23/Practical%20sessions/multiple%20forms%20demo.zip HTTP/1.1" 49104 "https://vula.uct.ac.za/portal/site/6a058395-ce47-4dc6-004f-c56075774e23/page/bbb27088-3846-4887-809d-1afa4bd6bb5e" "FDM 2.x" 26068 22430 200
[20/Oct/2007:17:29:14 +0200] 137.158.108.116 TLSv1 RC4-MD5 "GET /access/content/group/6a058395-ce47-4dc6-004f-c56075774e23/Practical%20sessions/multiple%20forms%20demo.zip HTTP/1.1" 16368 "https://vula.uct.ac.za/portal/site/6a058395-ce47-4dc6-004f-c56075774e23/page/bbb27088-3846-4887-809d-1afa4bd6bb5e" "FDM 2.x" 23810 22476 200
[20/Oct/2007:17:29:14 +0200] 137.158.108.116 TLSv1 RC4-MD5 "GET /access/content/group/6a058395-ce47-4dc6-004f-c56075774e23/Practical%20sessions/multiple%20forms%20demo.zip HTTP/1.1" 16368 "https://vula.uct.ac.za/portal/site/6a058395-ce47-4dc6-004f-c56075774e23/page/bbb27088-3846-4887-809d-1afa4bd6bb5e" "FDM 2.x" 24891 22719 200
[20/Oct/2007:17:29:14 +0200] 137.158.108.116 TLSv1 RC4-MD5 "GET /access/content/group/6a058395-ce47-4dc6-004f-c56075774e23/Practical%20sessions/multiple%20forms%20demo.zip HTTP/1.1" 16368 "https://vula.uct.ac.za/portal/site/6a058395-ce47-4dc6-004f-c56075774e23/page/bbb27088-3846-4887-809d-1afa4bd6bb5e" "FDM 2.x" 23475 22135 200
[20/Oct/2007:17:29:14 +0200] 137.158.108.116 TLSv1 RC4-MD5 "GET /access/content/group/6a058395-ce47-4dc6-004f-c56075774e23/Practical%20sessions/multiple%20forms%20demo.zip HTTP/1.1" 16368 "https://vula.uct.ac.za/portal/site/6a058395-ce47-4dc6-004f-c56075774e23/page/bbb27088-3846-4887-809d-1afa4bd6bb5e" "FDM 2.x" 23689 21901 200
[20/Oct/2007:17:29:14 +0200] 137.158.108.116 TLSv1 RC4-MD5 "GET /access/content/group/6a058395-ce47-4dc6-004f-c56075774e23/Practical%20sessions/multiple%20forms%20demo.zip HTTP/1.1" 16368 "https://vula.uct.ac.za/portal/site/6a058395-ce47-4dc6-004f-c56075774e23/page/bbb27088-3846-4887-809d-1afa4bd6bb5e" "FDM 2.x" 23853 22274 200