Using RSS and/or email to monitor the new Statalist Forum

[Phil Schumm]

One of the nice things about the old mailing list format of Statalist was that you could monitor list activity via standard email. Thus, if you kept your mail client open throughout the day, you could automatically stay abreast of discussions taking place on the list. In contrast, accessing the new Statalist Forum through the web requires switching to a web browser and actively navigating to and browsing the various subforums. As a result, some may worry that they'll miss replies or lose touch with what's going on because they're checking the Forum less frequently. It may also seem like it now takes more effort to monitor the Forum than it did the old list.

Fortunately, the forum software includes two features which address this issue. The first is Rich Site Summary (RSS), also known as Really Simple Syndication, which allows a user to subscribe to a web site and receive automatic updates. Some readers will be familiar with RSS as a mechanism for subscribing to online news outlets or blogs. In fact, you can subscribe to both the Stata Headlines (RSS feed) and the Stata Blog (RSS feed) through their own RSS feeds.

Each of the four subforums within the Statalist Forum has its own RSS feed, which you may subscribe to individually by clicking on the corresponding orange RSS icon on the main forums page. If you examine these URLs, you'll see that they have the following form:

http://www.statalist.org/forums/external?type=rss2&nodeid=51

where nodeid 51 corresponds to the "General" subforum, nodeid 52 corresponds to the "Mata" subforum, nodeid 270 corresponds to the "Using the Forum" subforum, and nodeid 269 corresponds to the "Sandbox" subforum.

Now if, say, you wanted to subscribe to a single feed containing posts from both the "General" and "Mata" subforums, you could do so by modifying the URL to include both nodeid 51 and 52, separated by a comma:

http://www.statalist.org/forums/external?type=rss2&nodeid=51,52

Note that the URLs shown above correspond to feeds which are updated with each new topic, but not each subsequent post under that topic. Thus, they work well if you merely want to monitor the appearance of new topics, but not if you want to monitor all forum posts. For the latter, you may add &lastpost=1 to the end of the URL. The resulting feed will be updated with the most recent post under each topic, thereby more closely approximating the experience of the old listserve.

For those new to RSS, there are numerous ways to consume RSS feeds, from dedicated apps (for both desktop and mobile) to cloud-based aggregators (e.g., Feedly) and even ways to hook RSS feeds into your web browser or email client. Wikipedia has a fairly comprehensive list, but you'll probably have to try out a few alternatives before you find a specific configuration that works well for you. Many of these include facilities for hooking into your OS's notification system, if you want to be actively notified of new updates. (Personally, I use a combination of Feedly—both through the web and via their mobile app—and RSS Notifier (for OS X). Feedly is a cloud-based aggregator, which permits me to synchronize my feeds (including their read/unread and saved status) across all my devices. Although they offer a "Pro" plan which requires a subscription fee, their free service is more than capable.)

Finally, note that if you enable email notifications (under User Settings -> Notifications), you will receive an email each time someone posts a new reply on a topic that you started. This is very convenient if you use a new topic to ask a question, and want to make sure that you are notified of all answers to that question.

[Stephen Jenkins]

This is very helpful information, Phil. As an RSS expert, could you (or someone else) please advise if it's possible to set up a feed so that it provides information like the old "Statalist-Digest" format. I'd like to continue the ability to look on a daily basis at posts made on a single day.

[Phil Schumm]

I don't believe that the Forum software itself provides a daily feed (i.e., containing all new topics/last posts per day). You could, in principle, build an external service to provide such a function, but I am not aware of any.

My understanding (correct me if I'm wrong) is that the rationale for digest subscription was to reduce the volume of individual emails. If you're monitoring the Forum via RSS, that issue doesn't arise (i.e., no more mail in your inbox). And if you simply want to see all of the updates from a single day in one place, I presume there are RSS readers that would let you do this (e.g., via a "Smart Folder"). If the concern is falling behind and getting overwhelmed, the reader I use (Feedly) let's me mark as "Read" all items in a given feed that are more than one day old.

Note that while you can use RSS to monitor a single feed (e.g., the Statalist Forum), the real power of the technology is the way it allows you to monitor scores or even hundreds of feeds (that is why RSS readers are sometimes known as aggregators). The trick is to use the technology in your service rather than to be driven by it (i.e., the difference between staying informed versus being overwhelmed). As a result, the makers of RSS readers have become adept at designing interfaces that make it easy to organize and manage feeds, to fly through new items, and generally to separate the wheat from the chaff. I'm sorry for not being able to give a more specific answer to your question, but I'd suggest trying out a few different types of readers first to get a feel for how they work.

[Laura Gibbons]

Sorry to not be up on the latest interfaces, but if someone could spell out how to get the new list fed to my email again I would be very grateful. I can't physically do all the clicking that is required to use this new site, and it takes too long to skim a bunch of messages here. I like to at least glance at all of them because I never know when I might pick up something useful.

Many thanks,

Laura

[Phil Schumm]

What email client do you use? If you use Outlook, here are some instructions.

[Laura Gibbons]

I use Thunderbird, thanks.

[Jorge Eduardo Perez Perez]

Sorry to not be up on the latest interfaces, but if someone could spell out how to get the new list fed to my email again I would be very grateful.
Laura

I am currently doing this:

* Getting Feedly to read the RSS feed following the instructions in the first post. I am using the RSS feed updated with the last post.
* Using IFTTT to send the Feedly feeds to Gmail, following these instructions: http://www.makeuseof.com/tag/how-to-...an-rss-reader/

It doesn't work great though, since it only emails you the last post of a thread. It makes it hard to stay on top of the list if you want to answer questions.
I wish there was a way to email all the posts within a thread.

[Phil Schumm]

I use Thunderbird, thanks.

Here are instructions for using Thunderbird to access RSS feeds. You might also find this list of add-ons helpful. I don't use Thunderbird, so I can't offer any personal experiences.

It doesn't work great though, since it only emails you the last post of a thread. It makes it hard to stay on top of the list if you want to answer questions. I wish there was a way to email all the posts within a thread.

There are several usage patterns for RSS feeds. One is simply to read the feeds as self-contained articles (like a magazine). Some feeds from news organizations or blogs work well this way, allowing you to rely entirely on the feed rather than going to the site. It all depends on how the feed is structured, and what content it includes.

Other feeds are intended primarily as notifications, with only a short summary of the full article/item. The idea is merely to determine whether you are interested. If not, then mark the feed as read and move on. If you are interested, however, then the idea is that you click through to the site to read the full article. With a good reader and a little practice, all of this can happen quickly using only a few keystrokes.

Obviously, with the current feeds provided by the Statalist Forum, you can't rely on the feeds alone. What I have been doing is the following:

1. I keep one set of standard feeds (one per subforum) which contain all new topics. This allows me very quickly to see if there are any topics I'm interested in, or questions I might be able to answer. In the case of the latter, I click through in a new tab in my browser (in Feedly, this is done simply by pressing "v"), answer the question on the site, and close the tab (in Safari on OS X, cmd-w). I can then go back to skimming through the feeds.
2. In addition, I keep a corresponding set of feeds with lastpost set to 1. These can be used to monitor activity on the Forum, since any topic with new activity will appear. If I've answered a question and the OP has a follow-up, it'll appear here. Similarly, if there is new activity on a topic I've been following, it'll appear here too. In both cases, I can again easily click through to the topic in a new tab, read or post, and then close the tab.

I'm certainly not claiming that this will work for everyone—just sharing my own experience. In sum, you can't use the Statalist RSS feeds as a replacement for visiting the site. They can, however, be used as a way of keeping abreast of all new topics (with the first post in its entirety), or a way of monitoring activity on topics you're following. This, together with email notifications of replies to topics you start, can help to keep you engaged and aware of the list without having to keep checking the site.

[Laura Gibbons]

Thank you for the Thunderbird link, I was able to set up the feed that way. As I am at least 10 years from retirement, I guess I'd better adjust to new formats!

[Mark Schaffer]

I'm an Outlook user and am trying the RSS settings helpfully suggested by Phil above and the links he provides. Phil, can you or someone else explain the relevance of the "Update Limit" setting?

Outlook has a tickbox that says: "Use the publisher update recommendation. Send/Receive groups do not update more frequently than the recommended limit to prevent your RSS Feed from being suspended by the content provider. Current provider limit: 1 hour."

... which sounds a bit ominous. And curiously, the example of how to set up RSS that Phil linked to had a current provider limit of 2 minutes.

1 hour seems a bit long to go between checks, but I am new to RSS so I have no idea if it is or not. Personally, I would prefer the immediacy that I had under Old Statalist. (But not if it risks "being suspended by the content provider"!)

[Roberto Ferrer]

Mark,

Some content providers limit the number of requests they receive (e.g. by a specific IP address) in a certain amount of time, in order to avoid network attacks. If an address tries to access the content more times than allowed, then it might get banned.

I don't use Outlook but here is some information:
http://office.microsoft.com/en-001/o...001229947.aspx

It's not clear to me if Statalist.org has limits (is it 2 minutes?). If so, can someone please say.

[Mark Schaffer]

My guess that Outlook is picking up the 1-hour limit from the Statalist.org server....

[Phil Schumm]

I don't know what the maximum polling frequency for statalist.org is set to. I use an aggregator (Feedly), and I believe it polls all feeds on average about once an hour, so I haven't been aware of statalist.org's setting per se. Even if statalist.org were to set it's maximum frequency to a small value (e.g., 15 minutes), RSS is not going to replicate exactly the experience of an email-based list, where new posts are sent as soon as the mail server can process them. For me, hourly (approximately) updates are adequate, and much better than having to remember to visit the website.

[Mark Schaffer]

Phil,

i'm having trouble getting lastpost=1 to work. I'm using Outlook 2010, and the RSS feed for, e.g., the General forum looks like this:

http://www.statalist.org/forums/external?type=rss2&nodeid=51&lastpost=1

but all I get is a notification of when a thread opens. No subsequent posts in the thread.

I suspect I'm doing something wrong that is very simple, but I can't work it out. Is the syntax above correct or is the culprit elsewhere?

[Mark Schaffer]

Actually ... I think what I'm getting is the latest post in the threat. I will post this now and see if the previous one disappears in my RSS feed. I suspect it will, which is too bad - I am trying, and failing, to get New Statalist to mimic Old Statalist in this respect, i.e., having a mail folder that accumulates traffic and makes it easy for me to spot particular topics/contributors.