This article aims to provide some guidance for using NewsRover to retrieve binary files from Usenet newsgroups.
While NewsRover has a conventional article list window, from which you can locate and select the files that you want to download, the easiest way to retrieve multi-part binaries is to set up one or more Autoscan Interest Groups. To learn more about downloading files "conventionally", take a look at the article on downloading.
Before you attempt to set up an AutoScan group, you must first configure NewsRover to use your news server(s). You will be prompted to do this when you first run the program after installing it. NewsRover can handle multiple news servers, so if you have access to more than one then it's worth setting up each of them, particularly as NewsRover can cycle through these to fetch all of the parts of a post. (This is especially useful if the posting comes up short on one server, because NewsRover might be able to complete the post from parts found on the other). Setting up a news server is very simple. Click on the Configure menu item, and you will be presented with the program's configuration dialog.
To add a new server to the list, simply click on the Add Server button, and follow the program's instructions. If you have chosen to setup multiple news servers, you can specify how NewsRover uses these when downloading items in your AutoScan interest groups. Click on the Scanning tab of the dialog to call up these settings.
Each server may have its own scanning priority set. The higher the number, the earlier the server is scanned. In this example, the first server to be scanned will be news.ntlworld.com, and the last chicago.usenetserver.com. Note that west.usenetserver.com will never be scanned, as it has a priority of zero.
So what does this do in practice? In simple terms, if NewsRover is trying to retrieve a 10 part binary from a newsgroup, and it only finds 8 parts on the first news server, it will look for the missing two on each of the other servers in turn.
While you're setting up the news servers, it's worth taking a look at the General tab. Here you can specify how long NewsRover pauses between each Interest Group that it scans. I recommend 60 minutes, as excessive scanning may upset your ISP.
Under Misc. controls, you can configure how much of your available network bandwidth NewsRover is allowed to use. You can also set the maximum number of connections that NewsRover will make per group. If you are planning to have NewsRover running all the time, you might want to try reducing the bandwidth to 75% to get better performance when using your connection for mail checking and web browsing.
Some ISPs limit the maximum number of connections that you are allowed, so the setting for the default number of connections may not have any effect.
One word of warning - when running at 100% utilisation to a fast news server you will likely find that your internet connection becomes unusable for anything else!
Now let's turn to the AutoScan interest groups themselves. You can tag and download files in NewsRover, just like you can in any other newsreader, but an AutoScan group can really help out - especially as you can set it running to repeat at set time intervals, allowing the program to leech everything from the group that it finds matching your criteria almost as soon as it is posted.
To create an AutoScan group, right click on the Autoscan Interest Groups item in the top left pane of NewsRover's three pane window.
Select the option Create New Interest Group from the popup menu, you will be presented with the following dialog:
You can name your interest group whatever you like, but I'd suggest something obvious (you can see some examples in the earlier picture). For this example, I'm creating a group called "Grace's Postings". As these instructions are intended to help you configure NewsRover to retrieve binaries, you should select the radio button Only messages that have attached files under I'm interested in. When you select this, the check box Only scan message if a file name is in the subject becomes available and checked. I recommend leaving it checked. Most posters use software that automatically puts the filename in the subject (such as the excellent PowerPost 2000) so this option helps keep out the garbage.
Now click the Next button to move on to the next stage of the configuration of this interest group. You will be presented with a dialog where you can select which newsgroups are to be scanned. Initially, it looks something like this ...
If you type "alt.binaries" into the text box above the list of newsgroups, the list will be narrowed to just those newsgroups containing alt.binaries. Type another period and an extra x, to get "alt.binaries.x" and you should see the groups that we're interested in for this example.
Click on the group(s) that you want to scan (you can select multiple groups, by pressing CTRL while you click), then click on the Select newsgroup button. For this example, I selected alt.binaries.x-files. The next screen allows you to tell NewsRover how far back in time it should go to start scanning articles. Very few news servers keep binaries longer than 14 days, so I recommend leaving it set at the default (30).
Clicking the Next button brings us to the screen that we really want. This is where we can set up the selection criteria to determine which messages are retrieved.
The trick to making Autoscan Interest Groups work properly is getting these expressions right. Clicking on Help will take you to the detailed instructions for creating these filters, but as an example suppose I wanted to get all of the messages whose subject contained either "Episode" or "Season", but I didn't want episodes 8X10 and 8X11 (remember, we're talking X Files here :)
From personal experience, I would advise against selecting the option to Search message header and message body - unless you want NewsRover to spend a lot of time downloading data that it never saves!
Search expressions can consist of AND, OR, NOT, NEAR and BEFORE operators, along with words, phrases and quoted strings. The syntax of these search expressions is well explained in the NewsRover help file. You can access the topic by clicking on the Help button from the Message Filters screen.
Some useful hints:
Words match at the beginning of a phrase, so car matches car, caravan, carcass.
Words in lower case match any combination of case, while words containing any upper-case characters must match the case exactly. e.g: car matches Car, CAR and CAr, but Car only matches Car.
Two or more words together match a phrase, so blue car matches any message with the word blue followed by any word beginning with car.
Words or phrases enclosed in quotes match whole words, rather than just the beginning of a word. So "blue car" matches blue car, but not blue carcass.
All punctuation symbols, both in the search string, and in the message being searched are treated as white space.
Just as in algebra, you can use parentheses to change the order of evaluation.
Clicking the Next button takes you to the final screen. This is where you tell NewsRover where to put its files. Click on Browse to get a standard directory chooser and select a location for your files. I recommend that you select the option Store incoming files directly into my save directory.
Finally, you can click Finish.
To start NewsRover scanning your new interest group, just click
on the running dog icon
in the toolbar.
That's it! Sit back and wait for NewsRover to pull down all of the files that match your selection criteria.
If you have a DSL or cable connection, you can leave NewsRover running all of the time and it will continue to download messages in the background. As files are downloaded, they will appear in the file list for the interest group. To see the current list of files, just click on the Interest Group's name, or expand the Interest Group and select from Messages, Picture Gallery (useful to preview your files if you're downloading a lot of JPEGs etc), File attachments and HTML attachments.
To find out what NewsRover
is currently doing, you can click on the Autoscan Control Panel button
. The status panel shows you which news server and
which news group is
currently being scanned, amongst other things. You'll see a dialog that
looks like this:
From here, the buttons are self explanatory; clicking Main screen closes the status panel. For more information, try clicking on the Event-Log item in the main menu. This tells you exactly what NewsRover is doing:
While NewsRover is running you will see a lot of network activity as it searches through the message headers looking for files to download. The very first time you run the AutoScan feature, this will continue for some time before you start seeing files appear. This is normal, though a little disconcerting. Before your first run, it's worth checking the current postings on the news server using either NewsRover's own article list window, or another news reader to make certain that there are messages which match your selection criteria.
Once you have the Interest Group working the way you want, you probably won't need to change any settings again, other than the selection criteria.
One useful tip is to create an Interest Group with the same name as a news group that you regularly download from. Set it up to scan that group (and any associated repost group, for example: alt.binaries.multimedia and alt.binaries.multimedia.repost) then change the selection criteria as needed to get specific files that interest you. This is sometimes easier than using the article list to tag and download files, because you get the advantage of scanning across multiple servers for missing parts. If you do this, you should reset the number of days back that NewsRover scans each time you change the selection criteria, this makes certain that all of the articles on the server are scanned each you run AutoScan.
There are two other features of NewsRover's AutoScan mechanism that you should be aware of: The Duplicate Files List and the Multi-Part Postings List.
NewsRover maintains a list of all the files that it has downloaded through the AutoScan mechanism, and uses this list to filter out files that you have already downloaded. This saves a lot of time (and network traffic), though you can fall foul of the feature if a download resulted in a corrupted file and you need to re-get the file. You can access the Duplicate Files list from the Configure menu option. Click on the Duplicate files tab of the configuration dialog to get the following screen.
Note that you can specify how long NewsRover keeps files in the list, and you can also enable/disable the feature altogether. If you want to re-download a file, you should locate its entry in this list, select it, then click on the Delete button. Don't forget to reset the number of days back that NewsRover scans so that it picks up the posting again.
Note: In my experience, if a file was corrupted in download, re-downloading it does not usually fix the problem. NewsRover is very reliable when downloading files, and you will normally find that the source posting was at fault, or there was a problem with the news server. It's usually better to wait for a repost (but remember that to pick up that repost, you still need to delete the filename from the Duplicate Files list).
The Multi-Part Postings list is useful to determine what files are currently in the process of being assembled and have missing parts that NewsRover is still looking for. Files currently in the process of being assembled from the current AutoScan run do not seem to appear here. You can access the list by clicking on the Multipart tab of the configuration dialog.
Notice that you can tell NewsRover how long to keep trying to assemble files with missing parts. It's rarely worth increasing this from the default 14 days, as very few news servers hold binaries longer than this. I can't think of a reason why you would want to disable multi-part file reconstruction (as that is NewsRover's strongest feature) but the option is there if you need it.
NewsRover's AutoScan Interest Groups are a powerful and flexible mechanism for automating the process of large binary file downloads. The ability to scan multiple servers for missing parts, and to specify complex selection criteria can save you a great deal of effort, but you do need to spend some time initially developing the Interest Groups. It's time well spent!
29th July, 2001
