TWiki Site Tools

Utilities for searching, navigation, and monitoring site activity

TWiki Site Tools include utilities for navigating, searching and keeping up with site activity. Preferences can be configured by web or site-wide. You are currently in the TWiki04x02 web. In particular, TWiki provides two highly configurable, automated site monitoring tools, WebNotify, to e-mail alerts when topics are edited, and WebStatistics, to generate detailed activity reports.

WebNotify - recent changes alert

Each TWiki web has an automatic e-mail alert service that sends a list of recent changes on a preset schedule, like once a day. Users can subscribe and unsubscribe using WebNotify in each web. The Perl script mailnotify is called by a background process at regular intervals. The script sends an automated e-mail to subscribed users if topics were changed in a web since the script was last run.

• You can create a WebNotify link using TWikiVariables with %NOTIFYTOPIC%

Web Changes Notification Service

Each TWiki web has an automatic e-mail notification service that sends you an e-mail with links to all of the topics modified since the last alert.

Users subscribe to email notifications using their WikiName or an alternative email address, and can specify the webs/topics they wish to track, Whole groups of users can also be subscribed for notification.

The general format of a subscription is:

three spaces * subscriber [ : topics ]

Where subscriber can be a WikiName, an E-mail address, or a group name. If subscriber contains any characters that are not legal in an email address, then it must be enclosed in 'single' or "double" quotes.

topics is an optional space-separated list of topics:

• ... without a Web. prefix
• ...that exist in this web.

Users may further customize the specific content they will receive using the following controls:

• Using wild-card character in topic names - You can use * in a topic name, where it is treated as a wildcard character. A * will match zero or more other characters - so, for example, Fred* will match all topic names starting with Fred, *Fred will match all topic names ending with Fred, and * will match all topic names.
• Unsubscribing to specific topics - Each topic may optionally be preceded by a '+' or '-' sign. The '+' sign means "subscribe to this topic". The '-' sign means "unsubscribe" or "don't send notifications regarding this particular topic". This allows users to elect to filter out certain topics. Topic filters ('-') take precedence over topic includes ('+') i.e. if you unsubscribe from a topic it will cancel out any subscriptions to that topic.
• Including child-topics in subscription - Each topic may optionally be followed by an integer in parentheses, indicating the depth of the tree of children below that topic. Changes in all these children will be detected and reported along with changes to the topic itself. Note This uses the TWiki "Topic parent" feature.
• Subscribing to entire topic ("news mode") - Each topic may optionally be immediately followed by an exclamation mark ! or a question mark ? with no intervening spaces, indicating that the topic (and children if there is a tree depth specifier as well) should be mailed out as complete topics instead of change summaries. ! causes the topic to be mailed every time even if there have been no changes, and ? will mail the topic only if there have been changes to it. One can limit the content of the subscribed topic to send out by inserting %STARTPUBLISH% and %STOPPUBLISH% markers within the topic. Note that "news mode" subscriptions require a corresponding cron job that includes the "-news" option (see details).

Examples: Subscribe Daisy to all changes to topics in this web.

   * daisy.cutter@flowers.com

Subscribe Daisy to all changes to topics that start with Web.

   * daisy.cutter@flowers.com : Web*

Subscribe Daisy to changes to topics starting with Petal, and their immediate children, WeedKillers and children to a depth of 3, and all topics that match start with Pretty and end with Flowers e.g. PrettyPinkFlowers

   * DaisyCutter: Petal* (1) WeedKillers (3) Pretty*Flowers

Subscribe StarTrekFan to changes to all topics that start with Star except those that end in Wars, sInTheirEyes or shipTroopers.

   * StarTrekFan: Star* - *Wars - *sInTheirEyes - *shipTroopers

Subscribe Daisy to the full content of NewsLetter whenever it has changed

   * daisy@flowers.com: NewsLetter?

Subscribe buttercup to NewsLetter and its immediate children, even if it hasn't changed.

   * buttercup@flowers.com: NewsLetter! (1)

Subscribe GardenGroup (which includes Petunia) to all changed topics under AllnewsLetters to a depth of 3. Then unsubscribe Petunia from the ManureNewsLetter, which she would normally get as a member of GardenGroup? :

   * GardenGroup: AllNewsLetters? (3)
   * petunia@flowers.com: - ManureNewsLetter

Subscribe IT:admins (a non-TWiki group defined by an alternate user mapping) to all changes to Web* topics.

   * 'IT:admins' : Web*

A user may be listed many times in the WebNotify topic. Where a user has several lines in WebNotify that all match the same topic, they will only be notified about changes that topic once (though they will still receive individual mails for news topics).

If a group is listed for notification, the group will be recursively expanded to the e-mail addresses of all members.

__ Warning: Because an email address is not linked to a user name, there is no way for TWiki to check access controls for email addresses. A user identified by an email address will only be sent change notifications if the topic they are asubscribed to is readable by guest users. You can limit what email addresses can be used in WebNotify, or even block use of emails altogther, using the {MailerContrib}{EmailFilterIn} setting in =configure.

Tip: List names in alphabetical order to make it easier to find the names.

Note for System Administrators: Notification is supported by an add-on to the TWiki kernel called the MailerContrib. See the MailerContrib topic for details of how to set up this service.

Note: If you prefer a news feed, point your reader to WebRss (for RSS 1.0 feeds) or WebAtom (for ATOM 1.0 feeds). Learn more at WebRssBase and WebAtomBase, respectively.

You can also use %MAINWEB% instead of Main, but this is not necessary even if you have renamed the main web by configuring {MainWebName} in configure.

WebSearch - search TWiki site

WebSearch is an extremely fast and flexible search facility, part of the core TWiki feature set. WebSearchAdvanced offers more options, including:

• topic title or full-text search
• regular expressions
• search within web or site-wide
• index-style A-Z alphabetical listing sorted topic title
• many more

See also: SearchHelp for help; TWikiVariables and FormattedSearch for including hard-coded searches in text.

WebChanges - what's new

To check for the most recently edited topics while on-site, use the WebChanges link, usually located in the toolbar. It lists the most recently modified topics, newest first, along with the first couple of lines of the page content.

This is simply a preset SEARCH. The number of topics listed by the limit parameter.:

%SEARCH{ ".*" web="TWiki" type="regex" nosearch="on" order="modified"
  reverse="on" 
limit="50" }%

WebRss and WebAtom - news feeds on recent changes

You can point your news reader at WebRss and WebAtom to find out what is new in a TWiki web. WebRssBase and WebAtomBase have the details. Like WebChanges, this is based on a %SEARCH{}%.

WebIndex - list of topics

WebIndex lists all web topics in alphabetical order, with the first couple of lines of text. This is simply a preset SEARCH:

%SEARCH{ "\.*" scope="topic" type="regex" nosearch="on" }%

WebStatistics - site statistics

You can generate a listing manually, or on an automated schedule, of visits to individual pages, on a per web basis. Compiled as a running total on a monthly basis. Includes totals for Topic Views, Topic Saves, Attachment Uploads, Most Popular Topics with number of views, and Top Contributors showing total of saves and attachment uploads. Previous months are saved.

• You can create a WebStatistics link using TWikiVariables with %STATISTICSTOPIC%

Configuring for automatic operation

• You can automatically generate usage statistics for all webs. To enable this:
  • Make sure variable {Log}{view}, {Log}{save} and *{Log}{upload} in are set in configure. This will generate log file entries (see below).
  • The WebStatistics topic must be present in all webs where you want to have statistics. You can use the topic in the Main web as a template.
  • Call the twiki/bin/statistics script from a cron job, once a day is recommended. This will update the WebStatistics topics in all webs.
  • Attention: The script must run as the same user as the CGI scripts are running, which is user nobody on many systems. Example crontab entry:
    0 0 * * * (cd /path/to/twiki/bin; ./statistics >/dev/null 2>&1)
  • There is a workaround in case you can't run the script as user nobody : Run the utility twiki/tools/geturl.pl in your cron job and specify the URL of the twiki/bin/statistics script as a parameter. Example:
    0 0 * * * (cd /path/to/twiki/tools; ./geturl.pl mydomain.com /urlpath/to/twiki/bin/statistics >/dev/null 2>&1)
  • NOTE: geturl.pl will do a TWiki CGI request as the TWikiGuest user, so if you use this workaround, the WebStatistics topics you are updating will have to be writable by TWikiGuest.

When running from the command line or a cron job, you can pass parameters to the script like this:

./statistics -logdate 200605 -webs TWiki,Sandbox

Generating statistics manually by URL

• The twiki/bin/statistics script can also be executed as a CGI script, just enter the URL in your browser. Examples:
  • Update current month for all webs you have access to:
    /cgi-bin/statistics
  • Update current month for Main web only:
    /cgi-bin/statistics/Main
  • Update Jan 2009 for Main web:
    /cgi-bin/statistics/Main?logdate=200901
  • Update Jan 2009 for the ProjectX, ProjectY and ProjectZ webs:
    /cgi-bin/statistics?logdate=200901;webs=ProjectX,ProjectY,ProjectZ

Log Files

TWiki generates monthly log files which are used by the statistics script

• The log file is defined by the {LogFileName} setting in configure
• The file name is log<year><month>.txt
• Example path name: twiki/logs/log200901.txt
• Each access gets logged as:
  | <time> | <wikiusername> | <action> | <web>.<topic> | <extra info> | <IP address> |
• Example log entry:
  | 07 Jan 2009 - 11:38 | TWikiGuest | view | WebRss |  | 66.124.232.02 |
• Actions are logged if enabled in configure by the {Log}{action} flags
• Logged actions:

Script Action name Extra info attach attach when viewing attach screen of previous uploaded attachment: filename changes changes   edit edit when editing non-existing topic: (not exist) rdiff rdiff higher and lower revision numbers: 4 3 register regstart WikiUserName, e-Mail address, LoginName: user attempts to register register register E-mail address: user successfully registers register bulkregister WikiUserName of new, e-mail address, admin ID rename rename when moving topic: moved to Newweb.NewTopic rename move when moving attachment: Attachment filename moved to Newweb.NewTopic passwd resetpasswd LoginName, WikiName, E-mail address, success code from addUserPassword passwd changepasswd LoginName, WikiName save save when replacing existing revision: repRev 3 when user checks the minor changes box: dontNotify when user changes attributes to an exising attachment: filename.ext save cmd special admin parameter used when saving search search search string upload upload filename view view when viewing non-existing topic: (not exist) when viewing previous topic revision: r3

E-mail

Configuring outgoing mail

Outgoing mail is required for TWikiRegistration and for recent changes alert.

TWiki will use the Net::SMTP module if it is installed on your system. Set this with the SMTPMAILHOST variable in TWikiPreferences.

The notify e-mail uses the default changes.tmpl template, or a skin if activated in the TWikiPreferences.

mailnotify also relies on two hidden files in each twiki/data/Web directory: .changes and .mailnotify. Make sure both are writable by your web server process. .changes contains a list of changes; go ahead and make this empty. .mailnotify contains a timestamp of the last time notification was done.

You can use an external mail program, such as sendmail, if the Net::SMTP module is not installed. Set the program path in {MailProgram} in configure.

• Net::SMTP can be easily disabled (if there is an installation error) by setting SMTPMAILHOST in TWikiPreferences to an empty value.

• You can set a separate SMTPSENDERHOST variable to define the mail sender host (some SMTP installations require this).

Setting the automatic e-mail schedule

For Unix platforms: Edit the cron table so that mailnotify is called in an interval of your choice. Please consult man crontab of how to modify the table that schedules program execution at certain intervals. Example:

% crontab -e
0 1 * * * (cd /path/to/twiki/bin; ./mailnotify -q)

The above line will run mailnotify nightly at 01:00. The -q switch suppresses all normal output.

For ISP installations: Many ISPs don't allow hosted accounts direct cron access, as it's often used for things that can heavily load the server. Workaround scripts are available.

On Windows: You can use a scheduled task if you have administrative privileges. TWiki:Codev/CronTabWin is a free scheduler for Windows.

Site Permissions

• TWikiAccessControl describes how to restrict read and write access to topics and webs, by users and groups
• SitePermissions lists the permissions settings of the webs on this TWiki site

Help with crontab

The crontab command is used to schedule commands to be executed periodically.

• Wikipedia.org:Crontab - crontab documentation
• pycron - crontab for Windows

Related Topics: AdminDocumentationCategory, AdminToolsCategory

-- Contributors: TWiki:Main.MikeMannix, TWiki:Main.PeterThoeny, TWiki:Main.CrawfordCurrie, TWiki:Main.KennethLavrsen, TWiki:Main.SvenDowideit

---

Comments & Questions about this Distribution Document Topic

• Need more details for WebChanges, like, how it works? Frequency of updating and all that...

• Need more details on WebNotify: How does it work? Exactly what data is emailed, in what format?

Updated this doc to cover OmittingEmailInWebNotify, and have updated Codev.WebNotify as well, as a first draft. Not sure how to update the template pages for other WebNotify pages - presumably done through TWiki beta installation?

-- RichardDonkin - 10 Nov 2002

According to MailNotifyScanOnlyOnOneWeb Is it possible to pass a web name to the mailnotify script to limit the webs to be handled. So you can add an additional crontab entries with a different schedule for each web. Is it included in the core? PeterKlausner submitted patch 23 Oct 2002

-- PeterMasiar - 02 Jun 2003

It might be appropriate to have references to some of the notification features that are not (yet?) part of the standard TWiki distribution - such as ImmediateNotifyPlugin. Or RSS. Or News. And %MAILLINKTO%.

I.e. the general issue is of interaction of TWiki with notification systems.

-- AndyGlew - 24 Jun 2003

>For ISP installations: Many ISPs don't allow hosted accounts direct cron access, as it's often used for things that can heavily load the server. Workaround scripts are available.

where are these scripts please? thanks. Better yet, how about making MailNotifyCgiScript?

-- MattWilkie - 21 May 2004

The cron example above uses 02:00 as start time. You might want to use a different time in the example, in case people copy it. In general, cron times between 02:00 and 03:00 should be avoided because they might not run or might run double at DST switch days.

-- JoachimSchrod - 26 Apr 2007

Good point. I updated the docs, also in SVN.

-- PeterThoeny - 29 Apr 2007

There is a broken link from Notification section of action tracker, please try to fix that

-- VirajShah - 04 May 2007

Could you please be more specific?

-- PeterThoeny - 08 May 2007

On the list of logged actions ("Log Files" section), the "renameweb" action is missing.

-- GuilhermeGarnier - 11 Jan 2008

Tracked in TWikibug:Item5241

-- PeterThoeny - 12 Jan 2008

Webstatistics is a bit awkward to update automagically per the instructions above, especially when X.509 certificate authentication is enabled. As noted, the geturl.pl script is anonymous.

Here is a shell script that is suitable for use as a cron job to automagically update statistics. Or you can run it by hand (instead of the bin/statistics script.).

Like geturl.pl, it will access the update statistics script. Unlike geturl.pl, it is capable of using host and client X.509 authentiation.

Host authentication ensures that the update goes to the correct host.

Client authentication identifies the script to the wiki. With the X509Plugin? installed, it will automagically log the scrip in under its Wiki username. Thus it will have the correct access privileges. (N.B. TWikiAdminGroup? is usually required for updating the statistics for Main...).

The script should be installed in /twiki/tools. It will determine the configuration from ../lib/LocalSite.cfg. Running it on my site is as simple as tools/runstatistics -- but it has enough options to meet most any requirements.

It would probably help others if it were put in the distribution.

I don't monitor this topic, but can be e-mailed with concerns.

Enjoy

runstatistics

#!/bin/sh
#
# Automagic update for web statistics
#
# Run by URL since we can't login under apache - and in any case, we want
# certficate-based authentication to work
#
# /twiki/bin/statistics - Current month for all webs
# /twiki/bin/statistics/Main - Just the main web
# Can also specify ?logdate=200801;webs=proj1,proj2,... (Would update proj1, proj2 for Jan 2008
#
# Since we use https authentication, wget uses administrator certificates...
# We also must specify our CA so that wget can validate our server
#
# wget can succeed even if the update fails, so we check for the expected
# completion text.
cd `dirname $0`

CFG="../lib/LocalSite.cfg"

#
# Get useful stuff from configuration
#
DefaultUrlHost='http://localhost:80'
ScriptUrlPath='/twiki/binx'
grep -P '^\$TWiki::cfg{(DefaultUrlHost|ScriptUrlPath|SmimeCertificateFile|SmimeKeyFile)}' $CFG | sed -e's|\$TWiki::cfg{\(.*\)} = \(.*\)\;|\1=\2|' >$p.tmp
. $p.tmp
rm $p.tmp
#
WHOST="$DefaultUrlHost$ScriptUrlPath"
#
CCSWS=
if [ -n "$SmimeCertificateFile" ] && [ -n "$SmimeKeyFile" ]; then
    CCSWS="--certificate=$SmimeCertificateFile --private-key=$SmimeKeyFile"
fi
#
# Would be nice to get this from LocalSite.cfg...
HCSWS=
HCSWS='--ca-directory=/etc/pki/tls/ca/'

P="`basename $0`"
WQ='-q'
Q=
T='cat'
WA=

while [ -n "$1" ];  do
    case "$1" in
        -d)
            # debug
            WQ=
            T="tee $P.log"
            ;;
        -l)
            # logfile date to process
            shift
            if [ -z "$Q" ]; then Q="?logdate=$1" ; else Q="$Q;logdate=$1" ; fi
            ;;
        -w)
            # webs to process
            shift
            if [ -z "$Q" ]; then Q="?webs=$1" ; else Q="$Q;webs=$1" ; fi
            ;;
        -c)
            # Client certificate
            shift
            CCSWS="$CCSWS --certificate=$1"
            ;;
        -k)
            # Client private key
            shift
            CCSWS="$CCSWS --private-key=$1"
            ;;
        -X)
            # Suppress X509 client certificate use
            CCSWS=
            ;;
        -a)
            # CA certificate bundle for host verification
            shift
            HCSWS="$HCASWS --ca-certificate=$1"
            ;;
        -A)
            # CA certificate directory for host verification
            shift
            HCSWS="$HCASWS --ca-directory=$1"
            ;;
        -x)
            # Suppress host verification
            HCSWS='--no-check-certificate'
            ;;
        -W)
            # wget advanced usage switches
            shift
            WA="$WA $1"
            ;;
        *)
            cat <<EOF
Usage: $P [-d] [-w web[,web...]] [-l logdate] [-X] [-x] [-c cert] [-k key] [-a cafile] [-A cadir] [-W wgetswitch]
Updates wiki statistics by touching the statistics update url.

The hostname and script url are extracted from LocalSite.cfg.

-w - comma-separated list of webs to update.  Default is all.
-l - logifile dates to update,  Format is yyyymm

https considerations:
  o wget will authenticate the host certificate using the default openssl CA.
    To specify a private CA (or a restricted list),  specify your CA directory with -A.or your bundle with -a
  o To disable host certificate authentication - not a good idea - use -x
  o Your client certificate and key are specified with -c and -k.  They default to the S/MIME signing certificates in LocalSite.cfg (if you have them).
  o Client authentication is enabled if -c and -k are specified or defaulted.  You can disable it with -X,
  o Some environments may need to specify more advanced wget switches.  -W lets you do that.
-d enables debugging and will write $P.log.
EOF
            exit 1;
            ;;

    esac
    shift
done

# After all that, the actual work is quite simple:

/usr/bin/wget $WQ $CCSWS $HCSWS $WA $WHOST/statistics$Q -O - | $T | grep $WQ 'End creating usage statistics'
#
ZZ="$?"
#
if [ $ZZ != 0 ]; then
    echo "wiki update $P failed with status $ZZ on `date -R`"
else
    if [ -z "$WQ" ]; then  echo "wiki update $P succeeded on `date -R`" ; fi
fi
if [ "$T" != 'cat' ]; then echo "Check $P.log for details" ; fi

-- TimotheLitt - 16 Dec 2008

Thanks Timothe for sharing the runstatistics script. The statistics script can be run as a shell script. Your script comes in handy if that is not possible and if auth is required. This is a rare enough case, I do not think the runstatistics should be part of the TWiki distribution since it is a rare case and since it is not Perl (e.g. platform dependent.) Probably best to create a new how-to page in the Codev web.

-- PeterThoeny - 16 Dec 2008

The statistics script can not be run (easily) as a shell script when an installation is secure. For example, files are owned by apache, but logins aren't permitted under apache. selinux is fussy about ownerships, permissions & labelling. While in theory one might create a special group, another account, and deal with the race conditions, it's hardly straightforward.

The distribution provides geturl.pl to provide updates in that case. But it requires TWikiGuest? to have write access to the web statistics pages - hardly a secure environment. And it does not work at all when authentication is required. Authentication is crucial to the enterprise. It should not -- and I predict will not - be a rare case.

TWiki wants to be an enterprise platform - marginalizing secure environments does not seem to be a sound strategy.

Shell scripts of this sort are about as portable as perl. This one would not have been any more portable as a perl script as the engine is wget - a system or backtick call from perl would be just about as platform-bound. Re-engineering wget in perl is theoretically possible - but I do leave that as an exercise to the reader!

I don't understand the notion that this is a Codev web issue - a solution is basic system management, and belongs where someone who is trying to setup a TWiki will find it. There could be other solutions - but this was easy to implement, and works for me.

But it's your project - disposition it however you like.

-- TimotheLitt - 17 Dec 2008