Video playback connection error with Firefox and load balancer pass-through settings
Description
Attachments
- 25 Sep 2020, 01:01 PM
- 25 Sep 2020, 12:51 PM
- 29 Mar 2019, 01:11 PM
- 29 Mar 2019, 01:10 PM
relates to
Activity
Austin October 29, 2020 at 6:30 PM
earle said this in the sakai-dev forum
No matter how many times we say "Sakai is not a streaming service" people still upload media. IMHO for well known media types we should offer minimal streaming capabilities.
Here is what I was thinking of, For media types like audio and video by default when a user clicks them is to stream them and offer the option for downloading, and yes we would want to use springs streaming api vs rolling our own, which the PR that Miguel references does.
There are likely some downsides to doing this that are not well known like: the overhead that Sam already mentioned, tying up threads for long periods of time so it needs to be well understood and probably configurable.
Austin October 15, 2020 at 6:28 PMEdited
Sam said this about the nightly servers via the dev-forum
Nginx proxying Tomcat. No other load balancer.
Austin October 9, 2020 at 3:11 PM
I'm still NOT able to reproduce the problem on https://qa12-mysql.nightly.sakaiproject.org
what is the hardware setup on that server? Are you using a Load balancer? apache in front of tomcat? etc?
However, when I view the network traffic in Firefox's web developer, I'm getting a "Blocked" response even though the video plays. https://developer.mozilla.org/en-US/docs/Tools/Network_Monitor/request_list says, "Or you might see a Red circle with a diagonal slash for responses that were blocked by the browser or the server". Are the nightly servers blocking responses? (Although I can see the status in Chrome)
We use Load Balancer + Apache (modjk) + Tomcat (ajp connector).
However, I'm still able to reproduce the
error with Tomcat (Http11Nio2Protocol connector) alone.
Sam Ottenhoff September 29, 2020 at 8:05 AM
Re Stephen's link from my PR: I think the idea was that if the user wasn't insisting on a range request, then we should try to avoid loading a big file into JVM memory and then segment it out via a range request. This is especially true if the content is stored in an object storage scheme like Swift or S3 as we certainly don't want to re-load that content back into JVM memory.
Stephen Marquard September 29, 2020 at 3:23 AM
Note that as well as BaseContentService, this code is also in
lessonbuilder/tool/src/java/org/sakaiproject/lessonbuildertool/service/LessonBuilderAccessService.java
(effectively a copy of the content service code).
https://groups.google.com/a/apereo.org/forum/#!topic/sakai-dev/A3XtReNvWT4
Just wondering if anyone has seen any issues with viewing/streaming video via the content resources tool URL in Firefox 60+ behind a load balancer? e.g.
/access/content/group/abc1234d-a123-456c-7e89-123b552b6733/MyVideo.mp4
What's happening is that "some" .mp4 videos will fail with a "Connection Failed" error in Firefox after playing for around 45s.
This problem started happening on our test system after we configured our F5 load balancer with a pass-through* configuration to route incoming and outgoing traffic through the load balancer. Previously, outgoing traffic was sent directly back to the client (*nPath config) and the problem did not happen.
I suspect that there's some combination of Firefox, Sakai, and the new Load Balancer config. that's causing the connection to fail mid stream.
However, what's complicated is that in the past when Firefox 60 first came out, this problem also happened when the .mp4 file was served by apache only and Sakai was not involved at all. But with the most recent Firefox 65.0.2, when the file is served by apache, the problem went away, so Firefox must have improved something in newer versions. But... the problem still happens with the lastest FF when the file is served up by Sakai through the load balancer.
Also, we're using Apache / modjk / tomcat but I've confirmed that those are not involved by running a vanilla tomcat project (no Sakai code at all) with the same apache / modjk config and the videos don't fail. Also note that with this basic setup, the older FF 60.0 also still fails, so that indicates as I mentioned before that FF must have improved something with 65.0. But unfortunately that leaves the Sakai code as being involved.
What's even stranger is that the problem happens with some .mp4s and not others. In particular .mp4s downloaded from youtube fail. But mp4s I created with DaVinci Resolve did not fail. But what I noticed is that the failing .mp4s seem to fail while streaming (I think), but the videos I created, appear to be completely downloaded before it plays, which I suspect is why they don't fail.
What looks like is happening, is that both files are sent to
BaseContentService.java
protected IOException copyRange(InputStream istream, OutputStream ostream, start, long end)and at some point the mp4 from youtube fails in
if (bytesToRead >= len) \{ ostream.write(buffer, 0, len); bytesToRead -= len; }where the ostream.write() fails and throws an error
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
Also, further up the chain (in BaseContentService.java) there's some code that sets the response headers to Accept-Ranges: none
e.g. there's a comment
// If there is a direct link to the asset, no sense streaming it. // Send the asset directly to the load-balancer or to the client URI directLinkUri = m_storage.getDirectLink(resource); ArrayList<Range> ranges = parseRange(req, res, len); if (directLinkUri != null || req.getHeader("Range") == null || (ranges == null) || (ranges.isEmpty())) { res.addHeader("Accept-Ranges", "none");which I think is also involved. But what's stranger, is that despite this header, the failing mp4 still appears to stream... at least it plays right away when it's loaded. vs. the mp4 I created will show a spinner icon while the video is loading before it plays.
(Another note, in Chrome, it will make an initial request where Sakai will set the Accept-Ranges: none, but then Chrome makes another request right away that contains a "Range" header, so Sakai will instead set the response header to Accept-Ranges: bytes and the video streams just fine.)
Thanks and sorry that this is so complicated, but it's been quite an interesting problem! But I'm still confused about where the main point of failure is... the Sakai streaming code, Firefox, or the Load Balancer?