Cannot use valid RSS feed in News

Problem is that it's rejecting the user agent. Even if you set "-Dhttp.agent="Sakai" as described in SAK-10159, it still redirects you around.

curl -v -A "Sakai" https://www.soils.org/news-media/releases/rss/

< HTTP/1.1 302 Found

With some agents (or no agent?) it redirects you to the 403 bad behavior
curl -v -A "MSIE 6.0" https://www.soils.org/news-media/releases/rss/
< HTTP/1.1 403 Bad Behavior

But if you give it the full curl user agent it goes right through?
curl -v -A "curl/7.16.4 (i386-apple-darwin9.0) libcurl/7.16.4 OpenSSL/0.9.7l zlib/1.2.3" https://www.soils.org/news-media/releases/rss/
< HTTP/1.1 200 OK

This is definitely not something we can fix and some security that host has setup. Email their webmaster maybe or find a working user agent.

So, you are suggesting this is an invalid feed that we cannot use? If I view this "soils" URL just using Firefox, it works just fine. It's not like we control the feed...it's external to our University.

I can't replicate the 403 error. It doesn't look like there is any auth on the RSS feed.

Valid HTTPS feeds seem fine. For example:

https://www.eff.org/rss/updates.xml

It looks like a Content-Type issue. Here is what a valid header looks like for RSS:

Content-Type: application/rss+xml; charset=utf-8

Note the Content-Type is HTML for this soils.org feed

$ curl -Ik https://www.soils.org/news-media/releases/rss/
HTTP/1.1 200 OK
Date: Wed, 27 Apr 2011 18:59:52 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.2.12
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Wed, 27 Apr 2011 18:59:52 GMT
Cache-Control:
Pragma:
Connection: close
Content-Type: text/html; charset=utf-8