[med-svn] r1047 - in trunk/community/infrastructure: . LibCIA LibCIA/Formatters LibCIA/IRC LibCIA/Stats LibCIA/Web LibCIA/Web/Stats Nouvelle
hanska-guest at alioth.debian.org
hanska-guest at alioth.debian.org
Sun Dec 30 16:44:45 UTC 2007
Author: hanska-guest
Date: 2007-12-30 16:44:44 +0000 (Sun, 30 Dec 2007)
New Revision: 1047
Added:
trunk/community/infrastructure/LibCIA/
trunk/community/infrastructure/LibCIA/Cache.py
trunk/community/infrastructure/LibCIA/Cache.pyc
trunk/community/infrastructure/LibCIA/ColorText.py
trunk/community/infrastructure/LibCIA/Cron.py
trunk/community/infrastructure/LibCIA/Database.py
trunk/community/infrastructure/LibCIA/Database.pyc
trunk/community/infrastructure/LibCIA/Debug.py
trunk/community/infrastructure/LibCIA/Debug.pyc
trunk/community/infrastructure/LibCIA/Files.py
trunk/community/infrastructure/LibCIA/Files.pyc
trunk/community/infrastructure/LibCIA/Formatters/
trunk/community/infrastructure/LibCIA/Formatters/Builder.py
trunk/community/infrastructure/LibCIA/Formatters/Builder.pyc
trunk/community/infrastructure/LibCIA/Formatters/ColorText.py
trunk/community/infrastructure/LibCIA/Formatters/ColorText.pyc
trunk/community/infrastructure/LibCIA/Formatters/Commit.py
trunk/community/infrastructure/LibCIA/Formatters/Commit.pyc
trunk/community/infrastructure/LibCIA/Formatters/Other.py
trunk/community/infrastructure/LibCIA/Formatters/Other.pyc
trunk/community/infrastructure/LibCIA/Formatters/Patch.py
trunk/community/infrastructure/LibCIA/Formatters/Patch.pyc
trunk/community/infrastructure/LibCIA/Formatters/Util.py
trunk/community/infrastructure/LibCIA/Formatters/Util.pyc
trunk/community/infrastructure/LibCIA/Formatters/__init__.py
trunk/community/infrastructure/LibCIA/Formatters/__init__.pyc
trunk/community/infrastructure/LibCIA/IRC/
trunk/community/infrastructure/LibCIA/IRC/Bots.py
trunk/community/infrastructure/LibCIA/IRC/Formatting.py
trunk/community/infrastructure/LibCIA/IRC/Handler.py
trunk/community/infrastructure/LibCIA/IRC/Network.py
trunk/community/infrastructure/LibCIA/IRC/__init__.py
trunk/community/infrastructure/LibCIA/IncomingMail.py
trunk/community/infrastructure/LibCIA/Message.py
trunk/community/infrastructure/LibCIA/Message.pyc
trunk/community/infrastructure/LibCIA/RpcClient.py
trunk/community/infrastructure/LibCIA/RpcServer.py
trunk/community/infrastructure/LibCIA/RpcServer.pyc
trunk/community/infrastructure/LibCIA/Ruleset.py
trunk/community/infrastructure/LibCIA/Ruleset.pyc
trunk/community/infrastructure/LibCIA/Security.py
trunk/community/infrastructure/LibCIA/Security.pyc
trunk/community/infrastructure/LibCIA/Stats/
trunk/community/infrastructure/LibCIA/Stats/Graph.py
trunk/community/infrastructure/LibCIA/Stats/Graph.pyc
trunk/community/infrastructure/LibCIA/Stats/Handler.py
trunk/community/infrastructure/LibCIA/Stats/Handler.pyc
trunk/community/infrastructure/LibCIA/Stats/Interface.py
trunk/community/infrastructure/LibCIA/Stats/Interface.pyc
trunk/community/infrastructure/LibCIA/Stats/Messages.py
trunk/community/infrastructure/LibCIA/Stats/Messages.pyc
trunk/community/infrastructure/LibCIA/Stats/Metadata.py
trunk/community/infrastructure/LibCIA/Stats/Metadata.pyc
trunk/community/infrastructure/LibCIA/Stats/Target.py
trunk/community/infrastructure/LibCIA/Stats/Target.pyc
trunk/community/infrastructure/LibCIA/Stats/__init__.py
trunk/community/infrastructure/LibCIA/Stats/__init__.pyc
trunk/community/infrastructure/LibCIA/TimeUtil.py
trunk/community/infrastructure/LibCIA/TimeUtil.pyc
trunk/community/infrastructure/LibCIA/Units.py
trunk/community/infrastructure/LibCIA/Units.pyc
trunk/community/infrastructure/LibCIA/Web/
trunk/community/infrastructure/LibCIA/Web/Info.py
trunk/community/infrastructure/LibCIA/Web/Info.pyc
trunk/community/infrastructure/LibCIA/Web/Overview.py
trunk/community/infrastructure/LibCIA/Web/Overview.pyc
trunk/community/infrastructure/LibCIA/Web/RegexTransform.py
trunk/community/infrastructure/LibCIA/Web/RegexTransform.pyc
trunk/community/infrastructure/LibCIA/Web/Server.py
trunk/community/infrastructure/LibCIA/Web/Server.pyc
trunk/community/infrastructure/LibCIA/Web/ServerPages.py
trunk/community/infrastructure/LibCIA/Web/ServerPages.pyc
trunk/community/infrastructure/LibCIA/Web/Stats/
trunk/community/infrastructure/LibCIA/Web/Stats/Browser.py
trunk/community/infrastructure/LibCIA/Web/Stats/Browser.pyc
trunk/community/infrastructure/LibCIA/Web/Stats/Catalog.py
trunk/community/infrastructure/LibCIA/Web/Stats/Catalog.pyc
trunk/community/infrastructure/LibCIA/Web/Stats/Columns.py
trunk/community/infrastructure/LibCIA/Web/Stats/Columns.pyc
trunk/community/infrastructure/LibCIA/Web/Stats/Feed.py
trunk/community/infrastructure/LibCIA/Web/Stats/Feed.pyc
trunk/community/infrastructure/LibCIA/Web/Stats/Graph.py
trunk/community/infrastructure/LibCIA/Web/Stats/Graph.pyc
trunk/community/infrastructure/LibCIA/Web/Stats/Link.py
trunk/community/infrastructure/LibCIA/Web/Stats/Link.pyc
trunk/community/infrastructure/LibCIA/Web/Stats/MessageViewer.py
trunk/community/infrastructure/LibCIA/Web/Stats/MessageViewer.pyc
trunk/community/infrastructure/LibCIA/Web/Stats/Metadata.py
trunk/community/infrastructure/LibCIA/Web/Stats/Metadata.pyc
trunk/community/infrastructure/LibCIA/Web/Stats/__init__.py
trunk/community/infrastructure/LibCIA/Web/Stats/__init__.pyc
trunk/community/infrastructure/LibCIA/Web/Template.py
trunk/community/infrastructure/LibCIA/Web/Template.pyc
trunk/community/infrastructure/LibCIA/Web/__init__.py
trunk/community/infrastructure/LibCIA/Web/__init__.pyc
trunk/community/infrastructure/LibCIA/XML.py
trunk/community/infrastructure/LibCIA/XML.pyc
trunk/community/infrastructure/LibCIA/__init__.py
trunk/community/infrastructure/LibCIA/__init__.pyc
trunk/community/infrastructure/Nouvelle/
trunk/community/infrastructure/Nouvelle/BaseHTTP.py
trunk/community/infrastructure/Nouvelle/ModPython.py
trunk/community/infrastructure/Nouvelle/Serial.py
trunk/community/infrastructure/Nouvelle/Table.py
trunk/community/infrastructure/Nouvelle/Twisted.py
trunk/community/infrastructure/Nouvelle/__init__.py
trunk/community/infrastructure/check-static
trunk/community/infrastructure/update-bugs
trunk/community/infrastructure/update-ddtp
trunk/community/infrastructure/update-tasks
trunk/community/infrastructure/update-tasks-wrapper
Log:
Importing scripts into SVN
Added: trunk/community/infrastructure/LibCIA/Cache.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Cache.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Cache.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,166 @@
+""" LibCIA.Cache
+
+A generic object cache. Arbitrary python objects are mapped to files or strings.
+
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.internet import defer
+from twisted.python import log
+import time, md5, os, random, cPickle, traceback, sys
+from LibCIA import Files
+
+class CachePerformance:
+ """Collects performance information about a cache.
+ This is not persistent across server sessions.
+ """
+ def __init__(self, name):
+ self.name = name
+ self.hits = 0
+ self.misses = 0
+
+_cachePerformanceStorage = {}
+
+def getNamedCachePerformance(name):
+ """Retrieve a CachePerformance object with the given name,
+ creating it if necessary.
+ """
+ global _cachePerformanceStorage
+ try:
+ return _cachePerformanceStorage[name]
+ except KeyError:
+ perf = CachePerformance(name)
+ _cachePerformanceStorage[name] = perf
+ return perf
+
+def getCachePerformanceList():
+ """Return an iterator for all registered CachePerformance instances"""
+ global _cachePerformanceStorage
+ return _cachePerformanceStorage.itervalues()
+
+
+class AbstractFileCache:
+ """An abstract cache mapping arbitrary python parameters to
+ a files. Subclasses should define the miss() function to
+ generate the data being cached in the event of a cache miss.
+ """
+ lifespan = None
+
+ def get(self, *args):
+ """Retrieve the item associated with some set of arguments.
+ If the item doesn't exist in the cache, this calls miss()
+ with the same arguments to generate the item, and adds it
+ to the cache.
+
+ Returns a Deferred instance that eventually results in a file object.
+ """
+ filename = self.getFilename(args)
+ perf = getNamedCachePerformance(self.__class__.__name__)
+
+ if os.path.isfile(filename):
+ result = defer.Deferred()
+ try:
+ self._returnHit(filename, result)
+ except:
+ log.msg("Exception occurred in %s._returnHit(%r)\n%s" % (
+ self.__class__.__name__, filename,
+ "".join(traceback.format_exception(*sys.exc_info()))))
+ else:
+ perf.hits += 1
+ return result
+
+ # We need to create the file. Do this atomically by first writing
+ # to a temporary file then moving that.
+ result = defer.Deferred()
+ defer.maybeDeferred(self.miss, *args).addCallback(
+ self._returnMiss, filename, result).addErrback(result.errback)
+ perf.misses += 1
+ return result
+
+ def _returnHit(self, filename, result):
+ result.callback(open(filename))
+
+ def _returnMiss(self, tempFilename, filename, result):
+ os.rename(tempFilename, filename)
+ result.callback(open(filename))
+
+ def miss(self, *args):
+ """Subclasses must implement this to generate the data we're supposed
+ to be caching in the event of a cache miss. Returns a filename where
+ the result can be found- this file will then be moved to the correct
+ place in the cache.
+
+ Implementations are encouraged, but not required, to get their file
+ name from self.getTempFilename().
+
+ The return value can optionally be passed via a Deferred.
+ """
+ pass
+
+ def getFilename(self, args):
+ """Return a filename for a cached object corresponding to the provided args
+ and this data type. This combines a hash of args with the name of this class.
+ We create the cache directory if necessary.
+ """
+ hash = md5.md5(repr(args)).hexdigest()
+ return Files.getCacheFile(self.__class__.__name__, hash)
+
+ def getTempFilename(self):
+ """Return a suggested temporary file name for miss() to use"""
+ return Files.getTempFile()
+
+
+class AbstractStringCache(AbstractFileCache):
+ """Based on AbstractFileCache, this cache provides an interface based on strings
+ rather than files. This still uses the same file-based caching mechanism
+ as AbstractFileCache.
+ """
+ def _returnHit(self, filename, result):
+ result.callback(open(filename, "rb").read())
+
+ def _returnMiss(self, string, filename, result):
+ tempFilename = self.getTempFilename()
+ open(tempFilename, "wb").write(string)
+ os.rename(tempFilename, filename)
+ result.callback(string)
+
+
+class AbstractObjectCache(AbstractFileCache):
+ """Based on AbstractFileCache, this cache provides an interface based on
+ arbitrary Python objects, serialized using cPickle.
+ """
+ def _returnHit(self, filename, result):
+ result.callback(cPickle.load(open(filename, "rb")))
+
+ def _returnMiss(self, obj, filename, result):
+ tempFilename = self.getTempFilename()
+ cPickle.dump(obj, open(tempFilename, "wb"), cPickle.HIGHEST_PROTOCOL)
+ os.rename(tempFilename, filename)
+ result.callback(obj)
+
+
+class Maintenance:
+ """Maintenance operations we run regularly to keep the cache in shape:"""
+
+ def run(self):
+ # FIXME: implement this for the new filesystem-based cache
+ pass
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Cache.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Cache.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/ColorText.py
===================================================================
--- trunk/community/infrastructure/LibCIA/ColorText.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/ColorText.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,266 @@
+""" LibCIA.ColorText
+
+Parses the {color} tag format used by original CIA commit messages,
+generating a <colorText> element tree.
+
+The <colorText> format is much like HTML. Any node can contain text and/or any of the
+nodes defined here. Tags and their effects can be nested like in HTML. The tags are:
+
+ <b> Bold, just like in HTML
+ <u> Underline, just like HTML
+ <br/> Line break, just like HTML
+ <color fg="foo"> Set the foreground color
+ <color bg="foo"> Set the background color
+
+Color names are the same as those allowed in the original {color} format, a complete
+list is in allowedColors.
+
+This code gets awfully fun, since the original format allowed goop like
+{blue}{reverse}{yellow}Hi{red}{bold}There{normal}
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import copy
+from LibCIA import XML
+
+
+allowedColors = (
+ "black",
+ "dark blue",
+ "dark green",
+ "green",
+ "red",
+ "brown",
+ "purple",
+ "orange",
+ "yellow",
+ "light green",
+ "aqua",
+ "light blue",
+ "blue",
+ "violet",
+ "grey",
+ "gray",
+ "light grey",
+ "light gray",
+ "white",
+ )
+
+
+class ColorState:
+ """Represents the current foreground color, background color, bold, and underline
+ state. State changes are generated by tags in the original format, which are
+ used to generate tags in the <colorText> format.
+ """
+ fgColor = None
+ bgColor = None
+ bold = False
+ underline = False
+
+ def reverseVideo(self):
+ """Swap foreground and background colors"""
+ self.fgColor, self.bgColor = self.bgColor, self.fgColor
+
+
+class ColorTextParser:
+ """Parses a commit in the old format (with {color} tags) and generates a <colorText>
+ element containing the message's contents.
+
+ Common usage:
+
+ >>> p = ColorTextParser()
+
+ >>> p.parseToString('hello')
+ '<colorText>hello</colorText>'
+
+ >>> p.parseToString('{bold}hello')
+ '<colorText><b>hello</b></colorText>'
+
+ >>> p.parseToString('{bold}hello{normal} world')
+ '<colorText><b>hello</b> world</colorText>'
+
+ Some pathological examples:
+
+ >>> p.parseToString('{bold}{dark blue}{reverse}{yellow}hello{normal}{underline} world')
+ '<colorText><color bg="dark blue"><color fg="yellow"><b>hello</b></color></color><u> world</u></colorText>'
+
+ >>> p.parseToString('{wiggle}')
+ '<colorText>{wiggle}</colorText>'
+
+ >>> p.parseToString('{blue}')
+ '<colorText/>'
+
+ >>> p.parseToString('<b>')
+ '<colorText><b></colorText>'
+
+ >>> p.parseToString('{blue}{normal}hello')
+ '<colorText>hello</colorText>'
+ """
+ def parse(self, message):
+ """Given a string of text in the original CIA commit format, return a <colorText>
+ element representing it as a DOM tree.
+ """
+ # Initialize our model of the current text format in the original message
+ self.parsedState = ColorState()
+
+ self.document = XML.createRootNode()
+
+ # Initialize our stack of (element, ColorState) tuples representing
+ # the state of the XML document being generated. This starts out with
+ # our root element in it.
+ self.elementStack = [
+ (XML.addElement(self.document, "colorText"), ColorState())
+ ]
+
+ # Break up the message into lines, each with its whitespace stripped.
+ # Run our lexical scanner on each line separately, turning it into
+ # a stream of events. Insert <br/> tags between lines.
+ lines = []
+ for line in message.split("\n"):
+ # Ignore extra whitespace
+ line = line.strip()
+ # Ignore blank lines
+ if line:
+ lines.append(line)
+ for i in xrange(len(lines)):
+ if i != 0:
+ XML.addElement(self.elementStack[-1][0], 'br')
+ self.lex(lines[i])
+ self.closeTags()
+
+ return self.document
+
+ def parseToString(self, message):
+ return XML.toString(self.parse(message).documentElement)
+
+ def lex(self, message):
+ """A simple lexical scanner to convert an incoming message into a stream of text and tag events"""
+ while message:
+ nextSquiggly = message.find("{")
+ if nextSquiggly != 0:
+ # Normal text
+
+ if nextSquiggly > 0:
+ # Process normal text and take it out of the message buffer
+ self.textEvent(message[:nextSquiggly])
+ message = message[nextSquiggly:]
+ else:
+ # This is the last event in the message
+ self.textEvent(message)
+ return
+ else:
+ # Possibly the beginning of a tag
+
+ tagEnd = message.find("}")
+ if tagEnd < 1:
+ # Unclosed tag, the rest of the message as normal text
+ self.textEvent(message)
+ return
+
+ # Chomp up the tag
+ self.tagEvent(message[1:tagEnd])
+ message = message[tagEnd+1:]
+
+ def tagEvent(self, tag):
+ """We just received a color tag. If this is a recognized tag, use it
+ to mutate the current parsedState. If not, put the squiggly brackets
+ back on and treat it as plain text.
+ """
+ if tag == "bold":
+ self.parsedState.bold = not self.parsedState.bold
+ elif tag == "underline":
+ self.parsedState.underline = not self.parsedState.underline
+ elif tag == "reverse":
+ self.parsedState.reverseVideo()
+ elif tag == "normal":
+ self.parsedState = ColorState()
+ elif tag in allowedColors:
+ self.parsedState.fgColor = tag
+ else:
+ # Unrecognized tag- output it unmodified
+ self.textEvent("{" + tag + "}")
+
+ def textEvent(self, text):
+ """We just received some text. Make sure our parsedState matches the
+ state of the current XML node and dump the new text in there.
+ """
+ self.updateState()
+ node = self.elementStack[-1][0]
+ node.appendChild(node.ownerDocument.createTextNode(text))
+
+ def closeTags(self):
+ """Close all currently opened tags"""
+ self.elementStack = self.elementStack[:1]
+
+ def pushTag(self, name, attributes={}, stateChanges={}):
+ """Add a new element to the elementStack, placed at
+ the end of the children list for the tag currently
+ at the top of the stack.
+
+ name: The name of the new element
+ attributes: A dict of attributes to set on the new element
+ stateChanges: A dict of attributes to change in the new tag's state
+ """
+ oldTag, oldState = self.elementStack[-1]
+
+ newTag = XML.addElement(self.elementStack[-1][0], name)
+ for key, value in attributes.iteritems():
+ newTag.setAttributeNS(None, key, value)
+
+ newState = copy.deepcopy(oldState)
+ newState.__dict__.update(stateChanges)
+
+ self.elementStack.append((newTag, newState))
+
+ def updateState(self):
+ """Compare the current parsedState with the ColorState() associated
+ with the XML element topmost on the stack. If they don't match,
+ add and remove elements as necessary to fix this.
+ """
+ if self.elementStack[-1][1] == self.parsedState:
+ # The states match, nothing to do
+ return
+
+ # For now this is pretty unintelligent- just close all tags
+ # and reopen them as necessary to recreate the new state.
+ self.closeTags()
+
+ if self.parsedState.bgColor:
+ self.pushTag('color',
+ {'bg': self.parsedState.bgColor},
+ {'bgColor': self.parsedState.bgColor}
+ )
+ if self.parsedState.fgColor:
+ self.pushTag('color',
+ {'fg': self.parsedState.fgColor},
+ {'fgColor': self.parsedState.fgColor}
+ )
+ if self.parsedState.bold:
+ self.pushTag('b',
+ {},
+ {'bold': True},
+ )
+ if self.parsedState.underline:
+ self.pushTag('u',
+ {},
+ {'underline': True},
+ )
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Cron.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Cron.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Cron.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,123 @@
+""" LibCIA.Cron
+
+A simple scheduler for repeated events. Each event is represented
+by a frequency in seconds, and a callable that handles the event.
+
+Trigger times are always an integer multiple of the period, so
+any period that one day can be evenly divided into will always run
+at the same time each day. An event with a period of an hour will
+always run at the top of the hour, and an event that runs
+every 15 minutes will always run at :00, :15, :30, and :45.
+
+Events are guaranteed not to be triggered too early, but they might
+be late if the system is heavily loaded.
+
+Normally all triggers are run once when they are added, since
+presumably if the server is just starting it will have missed a lot
+of opportunities to trigger events. This makes sense for most
+maintenance tasks. If it doesn't, you can disable that by adding the
+event with runNow=False.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import time
+import TimeUtil
+from twisted.internet import reactor, defer
+from twisted.python import log
+
+# Just some convenience definitions you can use for event periods
+hourly = 60 * 60
+daily = hourly * 24
+
+
+class Event:
+ """An event that can be scheduled by Cron. A callable is run when
+ the event needs to be triggered. Given the current time, an
+ event can report when it needs to run next.
+
+ If the callable isn't expected to complete its tasks right away,
+ it should return a Deferred.
+ """
+ def __init__(self, period, callable, name=None, runNow=False):
+ if not name:
+ name = repr(callable)
+
+ self.period = period
+ self.callable = callable
+ self.name = name
+
+ if runNow:
+ self.triggerTime = 0
+ else:
+ self.triggerTime = self.findNextTriggerTime()
+
+ def findNextTriggerTime(self, now=None):
+ """Find the next time (in seconds since the epoch) that this
+ event should be triggered, after the current time.
+ """
+ if now is None:
+ now = time.time()
+ return (now // self.period + 1) * self.period
+
+ def trigger(self):
+ """Trigger this event, and update the time at which
+ the next triggering should occur.
+ """
+ self.triggerTime = self.findNextTriggerTime()
+ log.msg("Cron: running %s, next run at %s" %
+ (self.name, TimeUtil.formatLogDate(self.triggerTime)))
+ defer.maybeDeferred(self.callable).addCallback(self.triggerFinished)
+
+ def triggerFinished(self, result):
+ log.msg("Cron: finished %s" % self.name)
+
+
+class Scheduler:
+ """Holds events, triggering them when they want to be triggered"""
+ def __init__(self, *events):
+ # Maps from event instance to the DelayedCall that schedules it
+ self.events = {}
+ for event in events:
+ self.schedule(event)
+
+ def cancel(self, event):
+ """Stop triggering the given event"""
+ if event in self.events:
+ if self.events[event].active():
+ self.events[event].cancel()
+ del self.events[event]
+
+ def schedule(self, event):
+ """Delete any stale DelayedCalls we may have for this event, then
+ create one that will trigger our event at the correct time.
+ """
+ self.cancel(event)
+ delay = event.triggerTime - time.time()
+ if delay > 0:
+ self.events[event] = reactor.callLater(event.triggerTime - time.time(), self.trigger, event)
+ else:
+ self.trigger(event)
+
+ def trigger(self, event):
+ """Trigger the given event then reschedule it"""
+ event.trigger()
+ self.schedule(event)
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Database.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Database.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Database.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,224 @@
+""" LibCIA.Database
+
+Utilities for accessing CIA's persistent data stored in an SQL database
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.enterprise import adbapi
+import MySQLdb.cursors
+import _mysql
+import os
+import XML
+
+
+# Import quoting functions for use elsewhere.
+# We use twisted's quote utility most places,
+# but as it doesn't understand MySQL-style binary
+# objects, we use _mysql to define quoteBlob.
+from twisted.enterprise.util import quote
+from _mysql import escape_string as quoteBlob
+
+# Disable the silly BLOB-to-array() conversion
+# in the latest versions of python-mysqldb
+def removeBlobConversions():
+ from MySQLdb.converters import conversions
+ from MySQLdb.constants import FIELD_TYPE
+ del conversions[FIELD_TYPE.BLOB]
+try:
+ removeBlobConversions()
+except:
+ pass
+
+
+class ConnectionPool(adbapi.ConnectionPool):
+ """Our own ConnectionPool subclass, sets the connection
+ encoding for MySQL 5.x automatically.
+ """
+ def connect(self):
+ conn = adbapi.ConnectionPool.connect(self)
+ cur = conn.cursor()
+ cur.execute("SET NAMES UTF8")
+ cur.close()
+ conn.commit()
+ return conn
+
+
+def readDictFile(path):
+ """Read a file formatted as a list of keys and values
+ separated by '=' signs. Any amount of whitespace
+ is allowed around the '=' and at the beginning and
+ end of lines.
+ """
+ d = {}
+ for line in open(path).readlines():
+ line = line.strip()
+ try:
+ key, value = line.split('=', 1)
+ d[key.strip()] = value.strip()
+ except ValueError:
+ pass
+ return d
+
+
+def createPool(overrides={}, filename="~/.cia_db", serverCursor=False):
+ """
+ This creates the global ConnectionPool object that we use to access our database.
+ Note that a ConnectionPool doesn't actually connect to the database, it
+ just imports the database module, validates it, and provides a way to
+ run queries. This is initialized at the module level, so we can use rebuild
+ to modify the database information if necessary.
+
+ The database password is retrieved from ~/.mysql_passwd so it isn't stored
+ in this file. If the file can't be read, an exception is raised.
+ """
+
+ # Defaults
+ info = {
+ 'host': 'localhost',
+ 'db': 'cia',
+ 'user': 'root',
+
+ # This is so we don't splurt our password out to twistd.log...
+ 'cp_noisy': False,
+ }
+
+ # With server-side cursors we can iterate over the result set without
+ # copying it all from mysqld to twistd. Unfortunately this can't
+ # be the default yet- server side cursors require the execute/fetch
+ # cycle to be obeyed strictly, and not all of CIA does this yet.
+ if serverCursor:
+ info['cursorclass'] = MySQLdb.cursors.SSCursor
+
+ # Load user settings from disk
+ if filename:
+ filename = os.path.expanduser(filename)
+ try:
+ info.update(readDictFile(filename))
+ os.chmod(filename, 0600)
+ except IOError:
+ raise Exception("Please create a file %r containing a list of database parameters.\n"
+ "For example:\n"
+ " host = mysql.example.com\n"
+ " user = bob\n"
+ " passwd = widgets\n"
+ % filename)
+
+ # Allow the caller to override settings, and remove Nones
+ info.update(overrides)
+ for key in info.keys():
+ if info[key] is None:
+ del info[key]
+
+ return ConnectionPool('MySQLdb', **info)
+
+
+pool = None
+
+def init(*args, **kwargs):
+ global pool
+ pool = createPool(*args, **kwargs)
+
+
+class Filter(XML.XMLObjectParser):
+ """A Database.Filter is syntactically very similar to a Message.Filter,
+ but describes an SQL expression. This class as-is is a generic SQL expression
+ builder, which is probably too lenient for most apps. Subclasses can
+ define their own variable lookup methods, to restrict the SQL expressions
+ this can generate.
+
+ After parsing, the completed SQL expression is available in the 'sql' attribute.
+
+ >>> Filter('<and> \
+ <or> \
+ <match var="parent_path">project</match> \
+ <match var="parent_path">author</match> \
+ </or> \
+ <not> \
+ <match var="target_path">project/muffin-deluxe</match> \
+ </not> \
+ </and>').sql
+ u"(((parent_path = 'project') OR (parent_path = 'author')) AND (!((target_path = 'project/muffin-deluxe'))))"
+
+ """
+ resultAttribute = 'sql'
+
+ def varLookup(self, var):
+ """Given a variable attribute, return an SQL expression representing it.
+ The default assumes it's already valid SQL, but subclasses may implement
+ this differently.
+ """
+ return var
+
+ def element_match(self, element):
+ """Compare a given variable exactly to the element's content, not including
+ leading and trailing whitespace.
+ """
+ return "(%s = %s)" % (self.varLookup(element.getAttributeNS(None, 'var')),
+ quote(XML.shallowText(element).strip(), 'varchar'))
+
+ def element_like(self, element):
+ """Compare a given variable to the element's content using SQL's 'LIKE' operator,
+ not including leading and trailing whitespace. This is case-insensitive, and includes
+ the '%' wildcard which may be placed at the beginning or end of the string.
+ """
+ return "(%s LIKE %s)" % (self.varLookup(element.getAttributeNS(None, 'var')),
+ quote(XML.shallowText(element).strip(), 'varchar'))
+
+ def element_and(self, element):
+ """Evaluates to True if and only if all child expressions evaluate to True"""
+ return "(%s)" % (" AND ".join([self.parse(node) for node in XML.getChildElements(element)]))
+
+ def element_or(self, element):
+ """Evaluates to True if and only if any child function evaluates to True"""
+ return "(%s)" % (" OR ".join([self.parse(node) for node in XML.getChildElements(element)]))
+
+ def element_not(self, element):
+ """The NOR function, returns false if and only if any child expression evaluates to True.
+ For the reasoning behind calling this 'not', see the doc string for this class.
+ """
+ return "(!%s)" % self.element_or(element)
+
+ def element_true(self, element):
+ """Always evaluates to True"""
+ return "(1)"
+
+ def element_false(self, element):
+ """Always evaluates to False"""
+ return "(0)"
+
+ def __and__(self, other):
+ """Perform a logical 'and' on two Filters without evaluating them"""
+ newFilter = Filter()
+ newFilter.sql = "(%s AND %s)" % (self.sql, other.sql)
+ return newFilter
+
+ def __or__(self, other):
+ """Perform a logical 'or' on two Filters without evaluating them"""
+ newFilter = Filter()
+ newFilter.sql = "(%s OR %s)" % (self.sql, other.sql)
+ return newFilter
+
+ def __invert__(self):
+ """Perform a logical 'not' on this Filter without evaluating it"""
+ newFilter = Filter()
+ newFilter.sql = "(!%s)" % self.sql
+ return newFilter
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Database.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Database.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Debug.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Debug.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Debug.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,284 @@
+""" LibCIA.Debug
+
+Remote interfaces for debugging and dynamic reloading while
+CIA is running.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.python import rebuild, log
+import RpcServer
+from cStringIO import StringIO
+import gc, sys, traceback, code, types
+
+
+class DebugInterface(RpcServer.Interface):
+ """An XML-RPC interface for remote debugging and dynamic reloading of CIA."""
+ def __init__(self):
+ RpcServer.Interface.__init__(self)
+ self.putSubHandler('gc', GcInterface())
+ self.protected_resetInterpreter()
+
+ def newCommandNamespace(self):
+ """Return a dictionary that acts as a fresh namespace for
+ executing debug console commands in.
+ """
+ return dict(globals())
+
+ def protected_rebuild(self, *packageNames):
+ """Use twisted.python.rebuild to reload the given package or module
+ and all loaded packages or modules within it.
+ """
+ for packageName in packageNames:
+ log.msg("Starting a rebuild at the package %r" % packageName)
+ # The non-empty fromlist tells __import__ we want the module referred
+ # to by the given path, not just its top-level module.
+ package = __import__(packageName, globals(), locals(), [''])
+ rebuildPackage(package)
+
+ def protected_resetInterpreter(self):
+ """Reinitialize the debug interpreter, beginning a new session with a new namespace"""
+ self.interpreter = RemoteInterpreter()
+
+ def protected_eval(self, source):
+ """Evaluate arbitrary code in the context of this module.
+ This is meant to be used to provide an interface similar to the python
+ interpreter. For this reason, incomplete code is detected and causes False
+ to be returned. If the code was complete, it is run and a string indicating
+ the result to display is returned.
+
+ NOTE: This is an extremely powerful function, so a 'debug' or 'debug.eval'
+ key should be treated with equivalent respect to a 'universe' key.
+ It is also quite dangerous. Besides being able to execute arbitrary
+ code, any infinite loops or code that takes a long time to run
+ will cause CIA to hang.
+ """
+ log.msg("Executing code in debug.evalCommand: %r" % source)
+ if self.interpreter.runsource(source):
+ # Incomplete
+ return False
+ else:
+ return self.interpreter.collectOutput()
+
+ def protected_getBanner(self):
+ """Return an appropriate banner string to use for interactive interpreters"""
+ return ('Python %s on %s\n'
+ 'Type "help", "copyright", "credits" or "license" for more information.\n'
+ '*** Running remotely in CIA: commands can not be interrupted ***' %
+ (sys.version, sys.platform))
+
+
+class NullDev:
+ """A fake device, like /dev/null"""
+ def write(self, data):
+ pass
+
+ def read(self):
+ return ''
+
+ def readline(self):
+ return ''
+
+
+class RemoteInterpreter(code.InteractiveInterpreter):
+ """An interpreter similar to the Python console that takes acts on commands
+ delivered over XML-RPC and captures what would be stdout into a string.
+ """
+ def __init__(self):
+ # By default, give them a new namespace filled with
+ # all the bare functions from this module.
+ ns = {}
+ for name, value in globals().iteritems():
+ if type(value) == types.FunctionType:
+ ns[name] = value
+
+ code.InteractiveInterpreter.__init__(self, ns)
+ self.capturedOutput = StringIO()
+
+ def runsource(self, source):
+ # Redirect stdout and stderr to our capturedOutput.
+ # We could just override write() to get tracebacks
+ # and such, but we also want commands that generate
+ # output via 'print' to work.
+ #
+ # We override stdin because it's pretty useless, and
+ # commands like help() that try to read from it will stall.
+ #
+ # This could be dangerous if we have other threads
+ # trying to write to them as well, but that would only
+ # happen (currently at least) if there's an error in
+ # one of our database threads. Since this is only used
+ # for debugging and such, that's an acceptable risk.
+ savedStdout = sys.stdout
+ savedStderr = sys.stderr
+ savedStdin = sys.stdin
+ sys.stdout = self.capturedOutput
+ sys.stderr = self.capturedOutput
+ sys.stdin = NullDev()
+ try:
+ result = code.InteractiveInterpreter.runsource(self, source)
+ finally:
+ sys.stdout = savedStdout
+ sys.stderr = savedStderr
+ sys.stdin = savedStdin
+ return result
+
+ def collectOutput(self):
+ """Return a string with all output generated since the last call"""
+ output = self.capturedOutput.getvalue()
+ self.capturedOutput = StringIO()
+ return output
+
+
+def typeInstances(t):
+ """Use gc mojo to return a list of all instances of the given type
+ (as returned by getTypeName(). This is the implementation of
+ debug.gc.typeInstances, and is provided as a module-level function
+ so it can be used in debug.eval
+ """
+ return [object for object in gc.get_objects() if getTypeName(object) == t]
+
+
+def getSingleton(t):
+ """Returns the singleton with the given type name, raising an exception
+ if exactly one object of that type doesn't exist.
+ Meant to be used from inside debug.eval().
+ """
+ insts = typeInstances(t)
+ if len(insts) != 1:
+ raise Exception("Found %d instances of %r, expected it to be a singleton" % (len(insts), t))
+ return insts[0]
+
+
+def rebuildPackage(package):
+ """Recursively rebuild all loaded modules in the given package"""
+ rebuild.rebuild(package)
+ # If this is really a package instead of a module, look for children
+ try:
+ f = package.__file__
+ except AttributeError:
+ return
+ if package.__file__.find("__init__") >= 0:
+ for item in package.__dict__.itervalues():
+ # Is it a module?
+ if type(item) == type(package):
+ rebuildPackage(item)
+
+
+def getTypeName(obj):
+ """Try as hard and as generically as we can to get a useful type/class name"""
+ try:
+ t = obj.__class__
+ except:
+ t = type(obj)
+ try:
+ t = t.__name__
+ except:
+ pass
+ t = str(t)
+ return t
+
+
+_leakTracker = None
+
+def debugLeaks(reset=False, quiet=False, interestedTypes=None):
+ """Show representations of all objects tracked by the garbage collector that
+ were not present at the last invocation.
+ """
+ global _leakTracker
+ gc.collect()
+
+ if _leakTracker is None or reset:
+ _leakTracker = {}
+ quiet = True
+
+ objlist = gc.get_objects()
+ newCount = 0
+ instCount = 0
+ for object in objlist:
+ typename = getTypeName(object)
+ if interestedTypes is not None:
+ if not typename in interestedTypes:
+ continue
+
+ instCount += 1
+ if _leakTracker.get(id(object)) == typename:
+ continue
+ newCount += 1
+ _leakTracker[id(object)] = typename
+
+ if not quiet:
+ print repr(object)
+
+ print "\n-- %d new instances, %d total instances, %d total objects" % (
+ newCount, instCount, len(objlist))
+
+
+class GcInterface(RpcServer.Interface):
+ """Memory debugging and profiling via python's garbage collector interface"""
+ def protected_garbageInfo(self):
+ """Return a string representation of the items in gc.garbage"""
+ return map(repr, gc.garbage)
+
+ def protected_objectsInfo(self):
+ """Return a string representation of the items in gc.get_objects().
+ This can take a while and be very big!
+ """
+ return map(repr, gc.get_objects())
+
+ def protected_typeInstances(self, t):
+ """Return all objects of any one type, using the same type names as typeProfile"""
+ return typeInstances(t)
+
+ def protected_collect(self):
+ """Force the garbage collector to run"""
+ log.msg("Forcing garbage collection")
+ gc.collect()
+
+ def protected_typeProfile(self):
+ """Return a chart showing the most frequently occurring types in memory"""
+ # Create a mapping from type name to frequency,
+ # and a mapping from type name to example instances
+ objects = gc.get_objects()
+ typeFreq = {}
+ typeInstances = {}
+ for object in objects:
+ t = getTypeName(object)
+ # Increment the frequency
+ typeFreq[t] = typeFreq.setdefault(t, 0) + 1
+
+ # Add to our list of example instances if it isn't already too big
+ inst = typeInstances.setdefault(t, [])
+ if len(inst) < 100:
+ inst.append(repr(object))
+
+ # Sort by frequency
+ keys = typeFreq.keys()
+ keys.sort(lambda a,b: cmp(typeFreq[a],typeFreq[b]))
+
+ # And return a nice table
+ lines = []
+ for key in keys:
+ contents = ", ".join(typeInstances[key]).replace("\n", "\\n")
+ if len(contents) > 100:
+ contents = contents[:100] + "..."
+ lines.append("%45s : %-10d %s" % (key, typeFreq[key], contents))
+ return "\n".join(lines)
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Debug.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Debug.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Files.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Files.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Files.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,97 @@
+""" LibCIA.Files
+
+This module is just a centralized place to manage CIA's filesystem
+usage. A top-level data directory can be set here during initialization,
+and this module is later queried for individual directories to use.
+
+The top-level directory ('data' by default) is split into three branches:
+
+ db Persistent data that should be backed up.
+
+ temp Transient files. These should be automatically deleted when they
+ are no longer needed. It's safe to clear this directory when the
+ server isn't running.
+
+ cache Data that is cached to improve performance, but can be rebuilt
+ from 'db' at any time. Files in the cache can be safely removed
+ at any time.
+
+ log Log files written by the server. These are only for the
+ administrator's benefit, and can be removed at any time.
+
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import os, random
+
+def setDataRoot(path):
+ """Change the root 'data' directory. This can be called in
+ your .tac file to override the default location for all
+ data files. The default is set below to 'data', in the
+ same directory as LibCIA.
+ """
+ global dataRoot
+ dataRoot = os.path.realpath(path)
+
+setDataRoot(os.path.join(os.path.split(__file__)[0], '..', 'data'))
+
+# Top-level directories
+dbDir = 'db'
+tempDir = 'temp'
+cacheDir = 'cache'
+logDir = 'log'
+
+def getDir(*seg):
+ """Get a data directory made from the provided path segments.
+ This returns a full path, creating it if necessary.
+ """
+ path = os.path.join(dataRoot, *seg)
+ if not os.path.isdir(path):
+ os.makedirs(path)
+ return path
+
+def tryGetDir(*seg):
+ """Like getDir, but this will not check the directory
+ for existence nor create any new directories.
+ """
+ return os.path.join(dataRoot, *seg)
+
+def getCacheFile(ns, digest):
+ """Get a cache file in the provided namespace (directory)
+ for an item with the provided digest string.
+ """
+ return os.path.join(getDir(cacheDir, ns), digest)
+
+def getTempFile():
+ """Return a new temporary filename, guaranteed
+ not to exist at the moment. This is not designed
+ to be used when our data directory may be shared
+ by other users, so it doesn't deal with the race
+ condition that would result in that situation.
+ """
+ root = getDir(tempDir)
+ for i in xrange(100):
+ path = os.path.join(root, '%d-%d' % (
+ os.getpid(), random.randint(100000, 999999)))
+ if not os.path.isfile(path):
+ return path
+ raise NotImplementedError("getTempFile() appears to be failing")
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Files.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Files.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Formatters/Builder.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Formatters/Builder.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Formatters/Builder.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,125 @@
+""" LibCIA.Formatters.Builder
+
+Formatters for converting builder messages to other formats.
+Builder messages contain a set of success/failure results and messages
+from some automated build or test process.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from LibCIA import Message, XML
+import Nouvelle
+
+__all__ = ['BuilderToPlaintext', 'BuilderToIRC', 'BuilderToXHTML']
+
+
+class BuilderFormatter(Message.Formatter):
+ """Abstract base class for formatters that operate on builder results."""
+ filter = '<find path="/message/body/builder"/>'
+
+ def format(self, args):
+ # Format each package inside each result set
+ packages = []
+ for results in XML.getChildElements(XML.dig(args.message.xml, "message", "body", "builder")):
+ if results.nodeName == 'results':
+ for package in XML.getChildElements(results):
+ if package.nodeName == 'package':
+ packages.append(self.format_package(package))
+ return self.joinMessage(args.message, packages)
+
+ def format_package(self, package):
+ """Format the results associated with one package,
+ given the XML <package> element
+ """
+ pass
+
+ def format_results(self, package):
+ """Given a package, returns a formatted representation of all results for that package"""
+ results = []
+ for element in XML.getChildElements(package):
+ f = getattr(self, 'result_' + element.nodeName, None)
+ if f:
+ results.append(f(element))
+
+ def joinMessage(self, message, packages):
+ """Join the results for each package into a final result"""
+ content = "builder"
+
+ branch = XML.digValue(message.xml, str, "message", "source", "branch")
+ if branch:
+ content += " " + self.format_branch(branch.strip())
+
+ # If we have only one package, put it on the same line as the heading
+ if len(packages) <= 1:
+ content += ": " + packages[0]
+ else:
+ content += "\n" + "\n".join(packages)
+ return content
+
+ def format_branch(self, branch):
+ return branch
+
+
+class BuilderToPlaintext(BuilderFormatter):
+ """Converts builder messages to plain text"""
+ medium = 'plaintext'
+
+ def format_package(self, package):
+ return "%s (%s)" % (package.getAttributeNS(None, 'name'), package.getAttributeNS(None, 'arch'))
+
+
+class BuilderToXHTML(BuilderFormatter):
+ """Converts builder messages to plain text"""
+ medium = 'xhtml'
+
+ def format_package(self, package):
+ return "%s (%s)" % (package.getAttributeNS(None, 'name'), package.getAttributeNS(None, 'arch'))
+
+ def joinMessage(self, message, packages):
+ content = []
+
+ branch = XML.digValue(message.xml, str, "message", "source", "branch")
+ if branch:
+ content.append(Nouvelle.tag('strong')[ self.format_branch(branch.strip()) ])
+
+ for package in packages:
+ if content:
+ content.append(Nouvelle.tag('br'))
+ content.append(package)
+
+ return content
+
+
+class BuilderToIRC(BuilderFormatter):
+ """Converts builder messages to IRC colorized text"""
+ medium = 'irc'
+
+ def __init__(self):
+ """By default, use the IRC color formatter"""
+ from LibCIA.IRC.Formatting import format
+ self.colorFormatter = format
+
+ def format_branch(self, branch):
+ return self.colorFormatter(branch, 'orange')
+
+ def format_package(self, package):
+ return "%s (%s)" % (package.getAttributeNS(None, 'name'),
+ self.colorFormatter(package.getAttributeNS(None, 'arch'), 'green'))
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Formatters/Builder.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Formatters/Builder.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Formatters/ColorText.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Formatters/ColorText.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Formatters/ColorText.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,181 @@
+""" LibCIA.Formatters.ColorText
+
+Formatters for converting colorText messages to other formats.
+This is the legacy format that old non-XML commits are converted to.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from LibCIA import Message, XML
+import Nouvelle, re
+import Util
+
+__all__ = ['ColortextToIRC', 'ColortextTitle', 'ColortextToPlaintext', 'ColortextToXHTML']
+
+
+class ColortextFormatter(Message.Formatter):
+ """Abstract base class for formatters that operate on colorText messages"""
+ filter = '<find path="/message/body/colorText"/>'
+
+
+class ColortextToIRC(ColortextFormatter):
+ """Converts messages with colorText content to plain text
+ with IRC color tags.
+ """
+ medium = 'irc'
+ color = True
+
+ def param_noColor(self, tag):
+ self.color = False
+
+ def __init__(self):
+ from LibCIA.IRC.Formatting import ColortextFormatter
+ self.formatter = ColortextFormatter()
+
+ def format(self, args):
+ colorText = XML.dig(args.message.xml, "message", "body", "colorText")
+ if self.color:
+ return self.formatter.parse(colorText)
+ else:
+ return XML.allText(colorText)
+
+
+class ColortextTitle(ColortextFormatter):
+ """Extracts a title from colorText messages"""
+ medium = 'title'
+
+ def format(self, args):
+ return Util.extractSummary(XML.dig(args.message.xml, "message", "body", "colorText"))
+
+
+class ColortextToPlaintext(ColortextFormatter):
+ """Extracts uncolorized plaintext from colorText messages"""
+ medium = 'plaintext'
+
+ def format(self, args):
+ return self.Parser(XML.dig(args.message.xml, "message", "body", "colorText")).result
+
+ class Parser(XML.XMLObjectParser):
+ requiredRootElement = 'colorText'
+
+ def parseString(self, s):
+ return s
+
+ def element_br(self, element):
+ return "\n"
+
+ def unknownElement(self, element):
+ return ''.join([s for s in self.childParser(element) if s])
+
+
+class ColortextToXHTML(ColortextFormatter):
+ """Converts messages with colorText content to XHTML (using Nouvelle)
+ with inline CSS representing the colorText formatting.
+ Returns an object that can be serialized into XHTML by a Nouvelle.Serializer.
+ """
+ medium = 'xhtml'
+
+ def format(self, args):
+ colorText = XML.dig(args.message.xml, "message", "body", "colorText")
+ return self.Parser(colorText).result
+
+ class Parser(XML.XMLObjectParser):
+ requiredRootElement = 'colorText'
+
+ colorTable = {
+ 'black': "#000000",
+ 'dark-blue': "#0000cc",
+ 'dark-green': "#00cc00",
+ 'green': "#00cc00",
+ 'red': "#cc0000",
+ 'brown': "#aa0000",
+ 'purple': "#bb00bb",
+ 'orange': "#ffaa00",
+ 'yellow': "#eedd22",
+ 'light-green': "#33d355",
+ 'aqua': "#00cccc",
+ 'light-blue': "#33eeff",
+ 'blue': "#0000ff",
+ 'violet': "#ee22ee",
+ 'grey': "#777777",
+ 'gray': "#777777",
+ 'light-grey': "#999999",
+ 'light-gray': "#999999",
+ 'white': "#FFFFFF",
+ }
+
+ def element_colorText(self, element):
+ return list(self.childParser(element))
+
+ def parseString(self, s):
+ return s
+
+ def element_b(self, element):
+ return Nouvelle.tag('b')[ self.element_colorText(element) ]
+
+ def element_u(self, element):
+ # We can't use <u> here, it's been deprecated and XHTML
+ # strict can't contain it. Just use some inline CSS instead.
+ return Nouvelle.tag('span', style="text-decoration: underline;")[ self.element_colorText(element) ]
+
+ def element_br(self, element):
+ return Nouvelle.tag('br')
+
+ def colorQuote(self, color):
+ """Make a color name safe for inclusion into a class attribute.
+ This just replaces any non-alphabetical characters with hyphens.
+ """
+ return re.sub("[^a-zA-Z]", "-", color)
+
+ def element_color(self, element):
+ """Convert the fg and bg attributes, if we have them, to <span> tags"""
+ style = ''
+ for attr, css in (
+ ('fg', 'color'),
+ ('bg', 'background'),
+ ):
+ attrValue = element.getAttributeNS(None, attr)
+ if attrValue:
+ try:
+ style = "%s%s: %s;" % (style, css, self.colorTable[attrValue])
+ except KeyError:
+ pass
+ return Nouvelle.tag('span', style=style)[self.element_colorText(element)]
+
+
+class ColortextToXHTMLLong(ColortextToXHTML):
+ """There isn't much extra we can say about a colortext commit, so put up
+ a notice explaining why. This also might be encouragement to upgrade
+ old clients that still give us colortext messages.
+ """
+ medium = 'xhtml-long'
+
+ def format(self, args):
+ from LibCIA.Web import Template
+ return [
+ Nouvelle.tag('p')[
+ ColortextToXHTML.format(self, args),
+ ],
+ Template.longError[
+ "This message was received in CIA's legacy format, 'colorText'. "
+ "To see much more detailed information about each commit, "
+ "ask this project's administrators to upgrade their client script."],
+ ]
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Formatters/ColorText.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Formatters/ColorText.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Formatters/Commit.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Formatters/Commit.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Formatters/Commit.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,483 @@
+""" LibCIA.Formatters.Commit
+
+Formatters used for converting commit messages to other formats.
+Note that this only handles real XML commit messages. The legacy
+'colorText' messages are handled by a separate module.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from LibCIA import Message, XML
+from Nouvelle import tag
+import re, posixpath
+from twisted.python.util import OrderedDict
+import Util
+
+__all__ = ['CommitToIRC', 'CommitToPlaintext', 'CommitToXHTML',
+ 'CommitTitle', 'CommitToXHTMLLong']
+
+
+class CommitFormatter(Message.ModularFormatter):
+ """Base class for formatters that operate on commit messages.
+ Includes a filter for commit messages, and utilities for
+ extracting useful information from the commits.
+ """
+ filter = '<find path="/message/body/commit"/>'
+ defaultComponentTree = """
+ <format>
+ <author/> <branch/> *
+ <version/><autoHide>r<revision/></autoHide>
+ <module/>/<files/>:
+ <log/>
+ </format>
+ """
+
+ # Subclasses can set this to limit the length of log messages, in lines
+ lineLimit = None
+
+ # Lines in the log longer than this are wrapped to wrapWidth
+ widthLimit = None
+ wrapWidth = None
+
+ # If the list of files ends up longer than this many characters, summarize it
+ filesWidthLimit = 60
+
+ # Instead of using our default pseudo-smart whitespace normalizing algorithm,
+ # we can optionally replace all whitespace with single space characters.
+ crunchWhitespace = False
+
+ def param_crunchWhitespace(self, tag):
+ self.crunchWhitespace = True
+
+ def param_lineLimit(self, tag):
+ self.lineLimit = int(XML.shallowText(tag))
+
+ def param_widthLimit(self, tag):
+ self.widthLimit = int(XML.shallowText(tag))
+ if self.wrapWidth > self.widthLimit:
+ self.wrapWidth = self.widthLimit
+
+ def param_wrapWidth(self, tag):
+ self.wrapWidth = int(XML.shallowText(tag))
+
+ def param_filesWidthLimit(self, tag):
+ self.filesWidthLimit = int(XML.shallowText(tag))
+
+ def component_author(self, element, args):
+ return self.textComponent(element, args, "message", "body", "commit", "author")
+
+ def component_version(self, element, args):
+ return self.textComponent(element, args, "message", "body", "commit", "version")
+
+ def component_revision(self, element, args):
+ return self.textComponent(element, args, "message", "body", "commit", "revision")
+
+ def component_branch(self, element, args):
+ return self.textComponent(element, args, "message", "source", "branch")
+
+ def component_module(self, element, args):
+ return self.textComponent(element, args, "message", "source", "module")
+
+ def component_project(self, element, args):
+ return self.textComponent(element, args, "message", "source", "project")
+
+ def component_files(self, element, args):
+ """Break up our list of files into a common prefix and a sensibly-sized
+ list of filenames after that prefix.
+ """
+ files = XML.dig(args.message.xml, "message", "body", "commit", "files")
+ if not (files and XML.hasChildElements(files)):
+ return [Message.MarkAsHidden()]
+
+ prefix, endings = self.consolidateFiles(files)
+ endingStr = " ".join(endings)
+ if len(endingStr) > self.filesWidthLimit:
+ # If the full file list is too long, give a file summary instead
+ endingStr = self.summarizeFiles(endings)
+ if prefix.startswith('/'):
+ prefix = prefix[1:]
+
+ if endingStr:
+ return ["%s (%s)" % (prefix, endingStr)]
+ else:
+ return [prefix]
+
+ def component_log(self, element, args):
+ log = XML.dig(args.message.xml, "message", "body", "commit", "log")
+ if not log:
+ return [Message.MarkAsHidden()]
+
+ if self.crunchWhitespace:
+ inputLines = [Util.getCrunchedLog(log)]
+ else:
+ inputLines = Util.getNormalizedLog(log)
+
+ # Break the log string into wrapped lines
+ lines = []
+ for line in inputLines:
+ # Ignore blank lines
+ if not line:
+ continue
+
+ # Wrap long lines
+ if self.widthLimit and len(line) > self.widthLimit:
+ lines.extend(Util.wrapLine(line, self.wrapWidth))
+ else:
+ lines.append(line)
+
+ # If our lineLimit is 1, don't bother starting long logs on the
+ # next line since there will be no long logs. Instead of the
+ # long (log message trimmed), just add an ellipsis.
+ if self.lineLimit == 1:
+ if len(lines) > 1:
+ lines[0] += ' ...'
+ del lines[1:]
+
+ # Multiline logs shouldn't start on the same line as the metadata
+ elif len(lines) > 1:
+ lines.insert(0, '')
+
+ # Truncate long log messages if we have a limit
+ if self.lineLimit and len(lines) > self.lineLimit + 1:
+ lines[0] = "(log message trimmed)"
+ del lines[self.lineLimit + 1:]
+
+ # Reassemble the log message and send it to the default formatter
+ return ["\n".join(lines)]
+
+ def summarizeFiles(self, files):
+ """Given a list of strings representing file paths, return
+ a summary of those files and/or directories. This is used
+ in place of a full file list when that would be too long.
+ """
+ # Count the number of distinct directories we have
+ dirs = {}
+ for file in files:
+ dirs[posixpath.split(file)[0]] = True
+
+ if len(dirs) <= 1:
+ return "%d files" % len(files)
+ else:
+ return "%d files in %d dirs" % (len(files), len(dirs))
+
+ def consolidateFiles(self, xmlFiles):
+ """Given a <files> element, find the directory common to all files
+ and return a 2-tuple with that directory followed by
+ a list of files within that directory.
+ """
+ files = []
+ if xmlFiles:
+ for fileTag in XML.getChildElements(xmlFiles):
+ if fileTag.nodeName == 'file':
+ files.append(XML.shallowText(fileTag))
+
+ # If we only have one file, return it as the prefix.
+ # This prevents the below regex from deleting the filename
+ # itself, assuming it was a partial filename.
+ if len(files) == 1:
+ return files[0], []
+
+ # Start with the prefix found by commonprefix,
+ # then actually make it end with a directory rather than
+ # possibly ending with part of a filename.
+ prefix = re.sub("[^/]*$", "", posixpath.commonprefix(files))
+
+ endings = []
+ for file in files:
+ ending = file[len(prefix):].strip()
+ if ending == '':
+ ending = '.'
+ endings.append(ending)
+ return prefix, endings
+
+class CommitToXHTML(CommitFormatter):
+ """Converts commit messages to XHTML, represented as a Nouvelle tag tree."""
+ medium = 'xhtml'
+ defaultComponentTree = """
+ <format xmlns:n='http://www.w3.org/1999/xhtml'>
+ <n:div style='border: 1px solid #888; background-color: #DDD; padding: 0.25em 0.5em; margin: 0em;'>
+ <autoHide> Commit by <n:strong><author/></n:strong></autoHide>
+ <autoHide> on <branch/></autoHide>
+ <n:span style='color: #888;'> :: </n:span>
+ <autoHide><n:b><version/></n:b></autoHide>
+ <autoHide>r<n:b><revision/></n:b></autoHide>
+ <n:b><module/></n:b>/<files/>:
+ <autoHide>(<url/>)</autoHide>
+ </n:div>
+ <n:div style='padding: 0em; margin: 0.5em 0em;'>
+ <log/>
+ </n:div>
+ </format>
+ """
+
+ # Use a lower width limit for HTML- web browsers typically won't wrap
+ # long paths, and it's generally easier to get to the full file tree
+ # on the web.
+ filesWidthLimit = 40
+
+ def __init__(self):
+ from LibCIA.Web import RegexTransform
+ self.hyperlinker = RegexTransform.AutoHyperlink()
+
+ def joinComponents(self, results):
+ """Nouvelle is just fine dealing with lists, don't join anything"""
+ return results
+
+ def walkComponents(self, nodes, args):
+ """Instead of concatenating lists, this implementation of walkComponents
+ nests them. This is more efficient with nouvelle, and lets us detect
+ empty results for <autoHide>.
+ """
+ results = []
+ for node in nodes:
+ results.append(self.evalComponent(node, args))
+ return results
+
+ def component_autoHide(self, element, args):
+ """The standard autoHide component is rewritten to properly recurse
+ into the contents of Nouvelle tags.
+ """
+ results = self.walkComponents(element.childNodes, args)
+ if self._checkVisibility(results):
+ return results
+ else:
+ return []
+
+ def _checkVisibility(self, nodes):
+ """Recursively check visibility for autoHide. Empty lists cause
+ us to return 0, and Nouvelle tags are recursed into.
+ """
+ for node in nodes:
+ if not node:
+ return 0
+ if isinstance(node[0], Message.MarkAsHidden):
+ return 0
+ if isinstance(node[0], tag):
+ if not self._checkVisibility(node[0].content):
+ return 0
+ return 1
+
+ def evalComponent(self, node, args):
+ """Here we convert all components starting with 'n:' into Novuelle tags.
+ FIXME: This should really be using proper DOM namespace manipulation and such
+ """
+ if node.nodeType == node.ELEMENT_NODE and node.nodeName.startswith("n:"):
+ attrs = {}
+ for attr in node.attributes.values():
+ attrs[str(attr.name)] = attr.value
+ return [tag(node.nodeName[2:], **attrs)[ self.walkComponents(node.childNodes, args) ]]
+ return CommitFormatter.evalComponent(self, node, args)
+
+ def component_url(self, element, args):
+ element = XML.dig(args.message.xml, "message", "body", "commit", "url")
+ if element:
+ return [tag('a', href=XML.shallowText(element))[ 'link' ]]
+ else:
+ return [Message.MarkAsHidden()]
+
+ def component_log(self, element, args):
+ """Convert the log message to HTML. If the message seems to be preformatted
+ (it has some lines with indentation) it is stuck into a <pre>. Otherwise
+ it is converted to HTML by replacing newlines with <br> tags and converting
+ bulletted lists.
+ """
+ log = XML.dig(args.message.xml, "message", "body", "commit", "log")
+ if not log:
+ return []
+ content = []
+ lines = Util.getNormalizedLog(log)
+ nonListItemLines = []
+ listItems = []
+
+ if lines:
+ # Scan the message, applying a few heuristics. If we see lines
+ # that are still starting with a space after getNormalizedLog
+ # has done its thing, assume the text is preformatted. Also
+ # look for lines that appear to be list items.
+ isPreFormatted = False
+ for line in lines:
+ if line and line[0] == ' ':
+ isPreFormatted = True
+
+ if line.startswith("* ") or line.startswith("- "):
+ # Assume this is a list item, and convert the bullets to
+ # a proper XHTML list.
+ listItems.append(line[2:])
+ else:
+ if listItems:
+ # It's a continuation of the last item
+ listItems[-1] = listItems[-1] + " " + line.strip()
+ else:
+ # If we haven't seen a list yet, stick this in nonListItemLines.
+ # If this log message isn't a list at all, everything will end
+ # up there but it will be safely ignored
+ nonListItemLines.append(line)
+
+ if listItems:
+ # It looks like a bulleted list. First output the nonListItemLines,
+ # then stick the items inside a list.
+ for line in nonListItemLines:
+ if content:
+ content.append(tag('br'))
+ content.append(line)
+ content = [
+ tag('p')[ content ],
+ tag('ul')[[ tag('li')[item] for item in listItems ]],
+ ]
+
+ elif isPreFormatted:
+ # This is probably a preformatted message, stick it in a <pre>
+ content.append(tag('pre')[ "\n".join(lines) ])
+
+ else:
+ # Plain old text, just stick <br>s between the lines
+ for line in lines:
+ if content:
+ content.append(tag('br'))
+ content.append(line)
+ else:
+ content.append(tag('i')["No log message"])
+
+ return self.hyperlinker.apply(content)
+
+
+class CommitToXHTMLLong(CommitToXHTML):
+ """Builds on the xhtml formatter to generate a longer representation of the commit,
+ suitable for a full page rather than just an item in a listing.
+ """
+ medium = 'xhtml-long'
+ defaultComponentTree = """
+ <format xmlns:n='http://www.w3.org/1999/xhtml'>
+ <n:h1>Commit Message</n:h1>
+ <headers/>
+ <n:div class='messageBody'><log/></n:div>
+ <n:h1>Modified Files</n:h1><files/>
+ </format>
+ """
+
+ _actionIcon = tag('img', width=12, height=12, _class='actionIcon')
+
+ actionIcons = {
+ 'add': _actionIcon(src='/images/file_added.png',
+ title='File Added', alt='Added'),
+ 'remove': _actionIcon(src='/images/file_removed.png',
+ title='File Removed', alt='Removed'),
+ 'rename': _actionIcon(src='/images/file_renamed.png',
+ title='File Renamed', alt='Renamed'),
+ 'modify': _actionIcon(src='/images/file_modified.png',
+ title='File Modified', alt='Modified'),
+ }
+
+ def component_headers(self, element, args):
+ """Format all relevant commit metadata in an email-style header box"""
+ from LibCIA.Web import Template
+
+ message = args.message
+ commit = XML.dig(message.xml, "message", "body", "commit")
+ source = XML.dig(message.xml, "message", "source")
+ author = XML.dig(commit, "author")
+ version = XML.dig(commit, "version")
+ revision = XML.dig(commit, "revision")
+ diffLines = XML.dig(commit, "diffLines")
+ url = XML.dig(commit, "url")
+ log = XML.dig(commit, "log")
+ project = XML.dig(source, "project")
+ module = XML.dig(source, "module")
+ branch = XML.dig(source, "branch")
+ headers = OrderedDict()
+
+ if author:
+ headers['Author'] = XML.shallowText(author)
+ if project:
+ headers['Project'] = XML.shallowText(project)
+ if module:
+ headers['Module'] = XML.shallowText(module)
+ if branch:
+ headers['Branch'] = XML.shallowText(branch)
+ if version:
+ headers['Version'] = XML.shallowText(version)
+ if revision:
+ headers['Revision'] = XML.shallowText(revision)
+ if diffLines:
+ headers['Changed Lines'] = XML.shallowText(diffLines)
+ if url:
+ headers['URL'] = tag('a', href=XML.shallowText(url))[ Util.extractSummary(url) ]
+
+ return [Template.MessageHeaders(headers)]
+
+ def component_files(self, element, args):
+ """Format the contents of our <files> tag as a tree with nested lists"""
+ from LibCIA.Web import Template
+
+ files = XML.dig(args.message.xml, "message", "body", "commit", "files")
+ if not (files and XML.hasChildElements(files)):
+ return []
+
+ # First we organize the files into a tree of nested dictionaries.
+ # The dictionary we ultimately have FileTree render maps each node
+ # (file or directory) to a dictionary of its contents. The keys
+ # in these dictionaries can be any Nouvelle-renderable object
+ # produced by format_file.
+ #
+ # As a first step, we build a dictionary mapping path segment to
+ # [fileTag, children] lists. We then create a visual representation
+ # of each fileTag and generate the final dictionary.
+ fileTree = {}
+ for fileTag in XML.getChildElements(files):
+ if fileTag.nodeName == 'file':
+ # Separate the file into path segments and walk into our tree
+ node = [None, fileTree]
+ for segment in XML.shallowText(fileTag).split('/'):
+ if segment:
+ node = node[1].setdefault(segment, [None, {}])
+ # The leaf node owns this fileTag
+ node[0] = fileTag
+
+ return [Template.FileTree(self.format_file_tree(fileTree))]
+
+ def format_file_tree(self, fileTree):
+ """This is the second half of format_files- it recursively
+ converts a tree of [fileTag,children] style dictionaries
+ into a tree of Template.FileTree() compatible dicts,
+ using format_file() to render each fileTag.
+ """
+ result = {}
+ for name, t in fileTree.iteritems():
+ fileTag, children = t
+ result[self.format_file(name, fileTag)] = self.format_file_tree(children)
+ return result
+
+ def format_file(self, name, fileTag=None):
+ """Given the short name of a file, and optionally its XML tag,
+ return a Nouvelle-serializable representation.
+ """
+ if fileTag:
+ # If we have a 'uri' attribute, make this file a hyperlink
+ uri = fileTag.getAttribute('uri')
+ if uri:
+ name = tag('a', href=uri)[ name ]
+
+ # If we have an 'action' attribute, represent it with an icon
+ actionIcon = self.actionIcons.get(fileTag.getAttribute('action'))
+ if actionIcon:
+ name = (name, actionIcon)
+
+ return name
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Formatters/Commit.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Formatters/Commit.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Formatters/Other.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Formatters/Other.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Formatters/Other.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,60 @@
+""" LibCIA.Formatters.Other
+
+Formatters that don't fit into the other categories- mostly
+formatters that can be referred to by name to modify a message
+rather than being automatically applied.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from LibCIA import Message, XML
+
+
+class IRCProjectName(Message.Formatter):
+ """Prepends the project name to each line of the input message, boldinated for IRC"""
+ medium = 'irc'
+ def format(self, args):
+ if not args.input:
+ return
+ project = XML.dig(args.message.xml, "message", "source", "project")
+ if project:
+ from LibCIA.IRC.Formatting import format
+ prefix = format("%s:" % XML.shallowText(project), 'bold') + " "
+ return "\n".join([prefix + line for line in args.input.split("\n")])
+ else:
+ return args.input
+
+
+class IRCFormat(Message.Formatter):
+ """Apply arbitrary IRC formatting to the input. A color name or other
+ formatting code may be supplied inside this formatter's XML tag.
+ """
+ medium = 'irc'
+ formattingCode = 'normal'
+
+ def format(self, args):
+ if input:
+ from LibCIA.IRC.Formatting import format
+ return format(args.input, self.formattingCode)
+
+ def loadParametersFrom(self, xml):
+ self.formattingCode = XML.shallowText(xml).strip()
+
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Formatters/Other.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Formatters/Other.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Formatters/Patch.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Formatters/Patch.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Formatters/Patch.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,31 @@
+""" LibCIA.Formatters.Patch
+
+Formatters for converting patch messages to other formats.
+Patch messages contain an author, log, and optional URL for new
+patches available from some patch tracker system.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from LibCIA import Message
+
+
+
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Formatters/Patch.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Formatters/Patch.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Formatters/Util.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Formatters/Util.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Formatters/Util.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,119 @@
+""" LibCIA.Formatters.Util
+
+Utilities shared by several formatters
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from LibCIA import XML
+import re
+
+
+def getCrunchedLog(xml):
+ """Given the DOM node for a <log> tag, return the log as
+ a string with all groups of one or more whitespace
+ characters replaced with a single space.
+ """
+ return re.sub("\s+", " ", XML.shallowText(xml)).strip()
+
+
+def getNormalizedLog(xml, tabWidth=8):
+ """Given the DOM node for a <log> tag, return a list of
+ text lines with whitespace normalized appropriately.
+ This strips all whitespace from the right side, and homogeneously
+ strips whitespace from the left side as much as possible.
+ Leading and trailing blank lines are removed, but internal
+ blank lines are not.
+ """
+ if not xml:
+ return []
+
+ lines = []
+ maxLeftStrip = None
+
+ for line in XML.shallowText(xml).split("\n"):
+ # Expand tabs and strip righthand whitespace
+ line = line.replace("\t", " "*tabWidth).rstrip()
+ strippedLine = line.lstrip()
+
+ # Blank lines don't count in determining the left strip amount
+ if strippedLine:
+ # Determine how much we can strip from the left side
+ leftStrip = len(line) - len(strippedLine)
+
+ # Determine the maximum amount of space we can strip
+ # from the left side homogeneously across the whole text
+ if maxLeftStrip is None or leftStrip < maxLeftStrip:
+ maxLeftStrip = leftStrip
+
+ # Skip leading blank lines
+ if lines or strippedLine:
+ lines.append(line)
+
+ # Remove trailing blank lines
+ while lines and not lines[-1].strip():
+ del lines[-1]
+
+ # Homogeneous left strip
+ if maxLeftStrip is None:
+ return lines
+ else:
+ return [line[maxLeftStrip:] for line in lines]
+
+
+def wrapLine(line, width):
+ """Given a long line, wrap it if possible to the given width,
+ returning a list of lines.
+ """
+ lines = []
+ newLine = ''
+ for word in line.split(" "):
+ oldLine = newLine
+ if newLine:
+ newLine = newLine + ' ' + word
+ else:
+ newLine = word
+ if len(newLine) > width:
+ oldLine = oldLine.rstrip()
+ if oldLine:
+ lines.append(oldLine)
+ newLine = word
+ if newLine:
+ lines.append(newLine.rstrip())
+ return lines
+
+
+def extractSummary(element, widthLimit=80):
+ """Extract all text from the given XML element, remove extra
+ whitespace, and truncate it to no longer than the given width.
+ """
+ # Extract all text, eating extra whitespace
+ text = re.sub("\s+", " ", XML.allText(element)).strip()
+
+ # Use wrapLine to cleanly break it if possible, but
+ # truncate it if necessary- wrapLine will not break words in
+ # half if they are longer than the wrap width.
+ lines = wrapLine(text, widthLimit)
+ if lines:
+ summary = lines[0][:widthLimit]
+ if len(summary) < len(text):
+ summary += "..."
+ return summary
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Formatters/Util.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Formatters/Util.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Formatters/__init__.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Formatters/__init__.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Formatters/__init__.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,39 @@
+""" LibCIA.Formatters
+
+A collection of Formatter subclasses that can be searched and
+instantiated via the 'factory' object here.
+
+This is a package holding modules for each category of formatter.
+The modules are aggregated together here and indexed by the factory.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import Commit, ColorText, Builder, Other, Patch
+
+_factory = None
+
+def getFactory():
+ from LibCIA import Message
+ global _factory
+ if not _factory:
+ _factory = Message.FormatterFactory(Commit, ColorText, Builder, Other, Patch)
+ return _factory
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Formatters/__init__.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Formatters/__init__.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/IRC/Bots.py
===================================================================
--- trunk/community/infrastructure/LibCIA/IRC/Bots.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/IRC/Bots.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,1267 @@
+""" LibCIA.IRC.Bots
+
+A small library for managing multiple IRC bots on multiple IRC networks.
+The code in this module runs in a separate daemon, so other CIA components
+can be restarted without effecting the bots.
+
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.protocols import irc
+from twisted.internet import protocol, reactor, defer
+from twisted.python import log, util
+from twisted.spread import pb
+import time, random, Queue
+from LibCIA import TimeUtil
+from LibCIA.IRC import Network
+
+
+class Request(pb.Referenceable):
+ """The Request object specifies a network, optionally a channel, and
+ a number of bots that need to inhabit that network/channel.
+
+ When a request is created, it registers itself with the BotNetwork,
+ which then tries its hardest to keep the request satisfied. The
+ bots satisfying this request are available in its 'bots' member.
+ """
+ def __init__(self, botNet, network, channel=None, numBots=1):
+ self.bots = []
+ self._active = True
+
+ self.botNet = botNet
+ self.network = network
+ self.channel = channel
+ self.numBots = numBots
+
+ log.msg("New %r" % self)
+ botNet.addRequest(self)
+
+ def __repr__(self):
+ if self.numBots == 1:
+ botInfo = "1 bot"
+ else:
+ botInfo = "%d bots" % self.numBots
+ if self.channel:
+ chanInfo = " in %s" % self.channel
+ else:
+ chanInfo = ''
+ return "<Request for %s%s on %s>" % (botInfo, chanInfo, self.network)
+
+ def active(self):
+ """Return True if this request is still active, similar to DelayedCall's interface"""
+ return self._active
+
+ def cancel(self):
+ """Indicate that this request is no longer needed. It is removed from the bot network."""
+ self.botNet.removeRequest(self)
+ self._active = False
+ log.msg("Cancelled %r" % self)
+
+ def findBots(self):
+ """Find bots that match this request, storing them in self.bots"""
+ # Look for our network and channel in the map from networks to bot lists
+ matches = []
+ if self.network in self.botNet.networks:
+ # Look for our channel in the map from channel names to bot lists
+ for bot in self.botNet.networks[self.network]:
+ if (not self.channel) or (self.channel in bot.channels):
+ matches.append(bot)
+
+ # Ignore any bots we don't need
+ self.bots = matches[:self.numBots]
+
+ def isFulfilled(self):
+ return len(self.bots) == self.numBots
+
+ def getUserCount(self):
+ """Return the number of users this request services directly.
+ This only works with requests for a channel, as we can't really
+ tell for channel requests. The returned number is the number of
+ users in the channel minus the number of bots we're using.
+ If the number of users can't be determined, this returns None.
+ """
+ if self.channel and self.bots:
+ nicks = self.bots[0].channels[self.channel].nicks
+ if nicks:
+ return len(nicks) - len(self.bots)
+
+ def remote_getInfoDict(self):
+ return {
+ 'server': str(self.network),
+ 'channel': self.channel,
+ 'user_count': self.getUserCount(),
+ 'is_fulfilled': self.isFulfilled(),
+ 'is_active': self.active(),
+ 'bots': [bot.remote_getInfoDict() for bot in self.bots],
+ }
+
+ def remote_active(self):
+ return self.active()
+
+ def remote_cancel(self):
+ self.cancel()
+
+ def remote_getBots(self):
+ return self.bots
+
+ def remote_getNumBots(self):
+ return self.numBots
+
+ def remote_getBotNicks(self):
+ return [bot.nickname for bot in self.bots]
+
+ def remote_isFulfilled(self):
+ return self.isFulfilled()
+
+ def remote_getChannel(self):
+ return self.channel
+
+ def remote_getNetworkName(self):
+ return str(self.network)
+
+ def remote_getUserCount(self):
+ return self.getUserCount()
+
+ def remote_msg(self, target, line):
+ """Use an arbitrary bot in this request to send a message to a
+ supplied target (channel or nickname). Currently this always
+ uses the first bot if possible and ignores the request if no
+ bots are available. This may be a good place to implement
+ smarter queueing and load balancing later. Currently bots
+ implement their own anti-flood queues.
+ """
+ if self.bots:
+ self.bots[0].queueMessage(target, line)
+
+ def remote_msgList(self, target, lines):
+ """Send multiple msg()es"""
+ for line in lines:
+ self.remote_msg(target, line)
+
+
+class NickAllocator:
+ """A nick allocator is responsible for determining what constitutes a valid nick,
+ and generating a list of valid nicks. This is an abstract base class.
+ """
+ username = 'CIA'
+ realname = 'CIA Bot (http://cia.vc)'
+
+ def isValid(self, nick):
+ """Returns True if the given nickname would be a valid output for our generator"""
+ return False
+
+ def generate(self):
+ """Generate a sequence of valid nicks, starting with the most preferable.
+ This must be an infinite (or nearly infinite) generator.
+ """
+ pass
+
+
+class SequentialNickAllocator(NickAllocator):
+ """Generate sequentially numbered nicks starting with a given prefix"""
+ def __init__(self, prefix):
+ self.prefix = prefix
+
+ def isValid(self, nick):
+ if not nick.startswith(self.prefix):
+ return False
+ numericPart = nick[len(self.prefix):]
+ try:
+ int(numericPart)
+ return True
+ except ValueError:
+ return False
+
+ def generate(self):
+ i = 1
+ while True:
+ yield self.prefix + str(i)
+ i += 1
+
+
+class RandomAcronymNickAllocator(NickAllocator):
+ """A nick allocator that generates random acronyms of a given length.
+ No, it doesn't know what they mean yet :)
+ """
+ def __init__(self, length=3, alphabet="ABCDEFGHIJKLMNOPQRSTUVWXYZ"):
+ self.length = length
+ self.alphabet = alphabet
+
+ def isValid(self, nick):
+ if len(nick) != self.length:
+ return False
+ for letter in nick:
+ if letter not in self.alphabet:
+ return False
+ return True
+
+ def generate(self):
+ while True:
+ yield "".join([random.choice(self.alphabet) for i in xrange(self.length)])
+
+
+
+class MessageLog:
+ """A log that stores a fixed number of messages"""
+ numMessages = 20
+
+ def __init__(self):
+ self.buffer = []
+
+ def log(self, message):
+ self.buffer = self.buffer[-self.numMessages:] + [message]
+
+
+class BotNetwork(pb.Root):
+ """A collection of IRC bots that work to satisfy a collection of Request objects.
+ Users should interact with the BotNetwork via Request instances.
+ """
+ # In addition to checking bot status immediately after changes that*******
+ # are likely to be important, we check bot status periodically, every
+ # botCheckInterval seconds.
+ botCheckInterval = 60
+
+ # Bots are given this many seconds after being marked inactive before they're
+ # disconnected. This way if a request is deleted then immediately replaced
+ # with another that has similar requirements, we don't end up replacing
+ # a bot we just deleted. I'm sure it has other good qualities too.
+ botInactivityTimeout = 60 * 5
+
+ # Maximum acceptable lag, in seconds. After this much the bot is disconnected
+ # This should be a fairly large number, since the bots may experience a minute
+ # or more of lag routinely when trying to join channels on connection.
+ maximumLag = 60 * 3
+
+ botCheckTimer = None
+
+ def __init__(self, nickAllocator):
+ self.nickAllocator = nickAllocator
+ self.requests = []
+ self.unknownMessageLog = MessageLog()
+
+ # A map from Network to list of Bots
+ self.networks = {}
+
+ # A list of all networks for which bots are being created currently.
+ # Maps from BaseNetwork instance to a DelayedCall signalling a timeout.
+ self.newBotNetworks = {}
+
+ # Lists all bots we're thinking about deleting due to inactivity.
+ # Maps from Bot instance to a DelayedCall.
+ self.inactiveBots = {}
+
+ # Start the bot checking cycle
+ self.checkBots()
+
+ def findBot(self, host, nickname, port=None):
+ """Find the bot currently on the given network with the given nick.
+ This is mostly for use with the debug console. Note that for
+ convenience, the network is specified as a host and port here.
+ A BaseNetwork instance will be created.
+ """
+ network = Network.find(host, port)
+ try:
+ bots = self.networks[network]
+ except KeyError:
+ return None
+ for bot in bots:
+ if bot.nickname == nickname:
+ return bot
+
+ def findRequest(self, network, channel):
+ """Find a request matching the given network, and channel"""
+ for req in self.requests:
+ if req.network == network and req.channel == channel:
+ return req
+
+ def addRequest(self, request):
+ """Add a request to be serviced by this bot network. This should
+ only be called by the Request class, as it automatically registers
+ itself with the botNet it was constructed for.
+ """
+ self.requests.append(request)
+ self.checkBots()
+
+ def removeRequest(self, request):
+ """Indicates that a request is no longer needed"""
+ self.requests.remove(request)
+ self.checkBots()
+
+ def checkBots(self):
+ """Search for unfulfilled requests, trying to satisfy them, then search
+ for unused bots and channels, deleting them or scheduling them for deletion.
+ """
+ # Scan through all requests, trying to satisfy those that aren't.
+ # Make note of which bots and which channels are actually being used.
+ # activeBots is a map from Bot instance to a map of channels that are being used.
+ usedBots = {}
+ for request in self.requests:
+ request.findBots()
+
+ if not request.isFulfilled():
+ # This request isn't fulfilled, try to change that
+ self.tryToFulfill(request)
+
+ for reqBot in request.bots:
+ # Make note of the bots and channels this request needs,
+ # and if the bot is already marked as inactive, cancel that.
+ usedBots.setdefault(reqBot, util.InsensitiveDict())[request.channel] = True
+
+ # Make sure that any referenced bots are no longer marked inactive
+ if reqBot in self.inactiveBots:
+ timer = self.inactiveBots[reqBot]
+ if timer.active():
+ timer.cancel()
+ del self.inactiveBots[reqBot]
+
+ # Now look for unused bots and/or channels
+ for network in self.networks.itervalues():
+ for bot in network:
+ if bot in usedBots:
+ usedChannels = usedBots[bot]
+
+ # We need this bot.. but are all the channels necessary?
+ for channel in bot.channels.iterkeys():
+ if channel not in usedChannels:
+ bot.part(channel)
+
+ # Since we need this bot, make sure it's still responsive. If its lag
+ # is too high, force it to give up. IF we have to disconnect the bot,
+ # give up this checkBots() and start over when botDisconnected() calls
+ # us again.
+ if bot.getLag() > self.maximumLag:
+ bot.quit()
+ self.botDisconnected(bot)
+ return
+
+ else:
+ # We don't need this bot. Tell it to part all of its channels,
+ # and if it isn't already, schedule it for deletion.
+ for channel in bot.channels.iterkeys():
+ bot.part(channel)
+ if bot not in self.inactiveBots:
+ self.inactiveBots[bot] = reactor.callLater(
+ self.botInactivityTimeout, self.botInactivityCallback, bot)
+
+ # Set up the next round of bot checking
+ if self.botCheckTimer and self.botCheckTimer.active():
+ self.botCheckTimer.cancel()
+ self.botCheckTimer = reactor.callLater(self.botCheckInterval, self.checkBots)
+
+ def tryToFulfill(self, request):
+ """Given an unfulfilled request, try to take actions to fulfill it"""
+ # How many more bots do we need?
+ neededBots = request.numBots - len(request.bots)
+
+ # Do we have any existing bots that can join a channel to fulfill the request?
+ if request.channel and request.network in self.networks:
+ for bot in self.networks[request.network]:
+ if bot not in request.bots:
+ # If the bot's already trying to connect to our channel,
+ # decrease the needed bots count so we don't end up asking
+ # all our bots to join this channel before the first one succeeds
+ if request.channel in bot.requestedChannels:
+ neededBots -= 1
+ elif not bot.isFull():
+ bot.join(request.channel)
+ neededBots -= 1
+ if neededBots <= 0:
+ return
+
+ # Nope... how about asking more bots to join the request's network?
+ if not request.network in self.newBotNetworks:
+ self.createBot(request.network)
+
+ def createBot(self, network):
+ """Create a new bot for the given network, retrying if necessary"""
+ # We're not already trying to connect a bot, or our previous attempt failed.
+ # Start trying to connect a bot, and set a timeout.
+ log.msg("Creating a new IRC bot for %s" % network)
+ BotFactory(self, network)
+ self.newBotNetworks[network] = reactor.callLater(network.newBotTimeout, self.newBotTimedOut, network)
+
+ def newBotTimedOut(self, network):
+ """We just timed out waiting for a new bot connection. Try again."""
+ log.msg("Timed out waiting for an IRC bot to connect to %s" % network)
+ del self.newBotNetworks[network]
+ # Don't immediately assume that we need to try again, but give us a chance to check
+ self.checkBots()
+
+ def botInactivityCallback(self, bot):
+ """Bots that have been unused for a while eventually end up here, and are disconnected"""
+ log.msg("Disconnecting inactive bot %r" % bot)
+ bot.quit()
+
+ def botConnected(self, bot):
+ """Called by a bot when it has been successfully connected."""
+ self.networks.setdefault(bot.network, []).append(bot)
+
+ try:
+ timer = self.newBotNetworks[bot.network]
+ if timer.active():
+ timer.cancel()
+ del self.newBotNetworks[bot.network]
+ except KeyError:
+ # Hmm, we weren't waiting for this bot to connect?
+ # Oh well, this bot will be garbage collected soon if that's really the case.
+ pass
+ self.checkBots()
+
+ def botDisconnected(self, bot):
+ """Called by a bot when it has been disconnected for any reason. Since
+ this might have been a disconnection we didn't intend, do another bot
+ check right away.
+ """
+ try:
+ self.networks[bot.network].remove(bot)
+ if not self.networks[bot.network]:
+ del self.networks[bot.network]
+ except:
+ # The bot might have not been in our network list in the first
+ # place, if it got disconnected before becoming fully connected
+ # or if its disconnection gets detected in multiple ways (socket
+ # closed, ping timeout, etc)
+ pass
+
+ # The maximum lag autodisconnect code in checkBots relies on this being
+ # called whether or not we actually removed the bot- the above 'pass'
+ # can not be safely replaced with 'return'.
+ self.checkBots()
+
+ def botJoined(self, bot, channel):
+ self.checkBots()
+
+ def botLeft(self, bot, channel):
+ self.checkBots()
+
+ def botKicked(self, bot, channel, kicker, message):
+ # FIXME: remove the ruleset
+ pass
+
+ def remote_findRequest(self, host, port, channel, create=True):
+ """Find and return a request, optionally creating it if necessary"""
+ network = Network.find(host, port)
+ r = self.findRequest(network, channel)
+ if r:
+ return r
+ elif create:
+ return Request(self, network, channel)
+ else:
+ return None
+
+ def remote_getRequests(self):
+ """Return a dictionary mapping (host, port, channel) to request instances"""
+ d = {}
+ for request in self.requests:
+ d[ request.network.getIdentity() + (request.channel,) ] = request
+ return d
+
+ def remote_getAllRequestInfo(self):
+ """Return a list of dictionaries describing each request."""
+ return [req.remote_getInfoDict() for req in self.requests]
+
+ def remote_findRequestInfo(self, host, port, channel):
+ """Return an info dictionary for a single request"""
+ req = self.remote_findRequest(host, port, channel, create=False)
+ if req:
+ return req.remote_getInfoDict()
+
+ def remote_getTotals(self):
+ """Return a dictionary of impressive-looking totals related to the bots"""
+ totals = dict(
+ networks = 0,
+ bots = 0,
+ channels = 0,
+ users = 0,
+ requests = 0,
+ unfulfilled = 0,
+ )
+
+ for network in self.networks.iterkeys():
+ totals['networks'] += 1
+ for bot in self.networks[network]:
+ totals['bots'] += 1
+ totals['channels'] += len(bot.channels)
+
+ for request in self.requests:
+ totals['requests'] += 1
+ totals['users'] += request.getUserCount() or 0
+ if not request.isFulfilled():
+ totals['unfulfilled'] += 1
+
+ return totals
+
+ def remote_getBots(self):
+ bots = []
+ for networkBots in self.networks.itervalues():
+ bots.extend(networkBots)
+ return bots
+
+ def remote_getMessageLog(self):
+ return self.unknownMessageLog.buffer
+
+ def remote_getNewBots(self):
+ """Return a list of bots that are currently trying to
+ connect, represented as (network, deadline) tuples.
+ """
+ newBots = []
+ for network, timer in self.newBotNetworks.iteritems():
+ newBots.append((str(network), timer.getTime()))
+ return newBots
+
+
+class ChannelInfo:
+ """A container for information about an IRC channel. Currently
+ this just holds things we will be told without asking, like the
+ channel's occupants and its topic.
+ """
+ def __init__(self, name):
+ self.name = name
+ self.nicks = []
+ self.topic = None
+
+ # This list collects names mentioned in an RPL_NAMREPLY.
+ # Upon receiving an RPL_ENDOFNAMES this is transferred to
+ # 'nicks' and replaced with a new empty list.
+ self.nickCollector = []
+
+
+class MessageQueue:
+ """A single message FIFO with a fixed maximum size. If messages
+ are dropped, this emits a warning at the proper time.
+
+ Normally, this acts just like a bare Queue object, except that
+ it never blocks. Once it fills up, each new message increments
+ a 'dropped' counter.
+
+ Logically, the dropped messages will always be at the back of
+ the queue. After the queue empties, a new message is added
+ indicating how many lines were dropped. Normal queueing resumes
+ after this.
+ """
+ def __init__(self, maxSize):
+ self._fifo = Queue.Queue(maxSize)
+ self._numDropped = 0
+ self._next = None
+
+ def put(self, message):
+ """Push a new message onto the Queue. Never blocks, but it
+ may drop messages if the queue is full. The message
+ should never be None.
+ """
+ if self._numDropped:
+ self._numDropped += 1
+ else:
+ try:
+ self._fifo.put(message, False)
+ except Queue.Full:
+ self._numDropped = 1
+
+ def get(self):
+ """Return the next message, discarding it from the queue."""
+ try:
+ return self._fifo.get(False)
+ except Queue.Empty:
+ if self._numDropped:
+ n = self._numDropped
+ self._numDropped = 0
+ return "(%d lines omitted)" % n
+
+
+class FairQueue:
+ """A FairQueue tracks pending messages to any number of targets,
+ each of which own their own independent queue. This prevents
+ floods to one target from affecting timely delivery to other
+ targets. All queues are checked in round-robin order.
+ """
+ def __init__(self, queueSize):
+ self._queueSize = queueSize
+ self._targetDict = {}
+ self._pendingQueue = Queue.Queue()
+ self._next = None
+
+ def put(self, target, message):
+ """Queue up a new message to the supplied target"""
+ if target not in self._targetDict:
+ queue = MessageQueue(self._queueSize)
+ self._targetDict[target] = queue
+ self._pendingQueue.put(target, False)
+ else:
+ queue = self._targetDict[target]
+ queue.put(message)
+
+ def peek(self):
+ """Return the next message to be processed, without discarding
+ it. A subsequent call to peek() will return the same message.
+ Returns a (target, message) tuple if a message is available,
+ or None if all queues are empty.
+ """
+ if self._next is None:
+ while 1:
+ # Get the next queue target in round-robin order
+ try:
+ target = self._pendingQueue.get(False)
+ except Queue.Empty:
+ break
+
+ queue = self._targetDict[target]
+ message = queue.get()
+
+ if message:
+ # We got a message. Reschedule this queue for later.
+ self._next = (target, message)
+ self._pendingQueue.put(target, False)
+ break
+
+ else:
+ # This queue is empty. Discard it.
+ del self._targetDict[target]
+
+ return self._next
+
+ def flush(self):
+ """Discard the last message returned by peek()"""
+ self._next = None
+
+
+class Bot(irc.IRCClient, pb.Referenceable):
+ """An IRC bot connected to one network any any number of channels,
+ sending messages on behalf of the BotController.
+
+ The Bot class is responsible for keeping track of the timers and
+ limits associated with joining channels, but it doesn't map itself
+ onto Requests, nor does it manage bot connection and disconnection.
+ """
+ # Timeout, in seconds, for joining channels
+ joinTimeout = 60
+
+ # Important timestamps
+ lastPingTimestamp = None
+ lastPongTimestamp = None
+ signonTimestamp = None
+ lastPingTransmitTimestamp = None
+
+ # Byte counters. We maintain a perpetually incrementing byte counter,
+ # tracking the amount of data sent over the life of the connection.
+ # This byte counter is sent with PING, and we record the value returned
+ # with the last PONG. Taking (txByteCount - txConfirmedBytes) gives us
+ # the number of bytes transmitted but not confirmed: the maximum amount
+ # of data the server may be buffering on our behalf. This buffer
+ # level must be controlled, to avoid being kicked for flooding.
+ txByteCount = 0
+ txConfirmedBytes = 0
+
+ # Maximum number of lines to queue per target (channel/user)
+ maxQueueSize = 20
+
+ # Unhandled commands to ignore, rather than log
+ ignoredCommands = [
+ "ERR_NOCHANMODES", # Freenode spamming us to register
+ 506, # PLD spamming us to register
+ 333, # Freenode sends these with channel registration info
+ ]
+
+ def __init__(self):
+ self.emptyChannels()
+ self._messageQueue = FairQueue(self.maxQueueSize)
+ self.pendingWhoisTests = {}
+ self.connectTimestamp = None
+
+ def emptyChannels(self):
+ """Called when we know we're not in any channels and we shouldn't
+ be trying to join any yet.
+ """
+ # Map from channel name to ChannelInfo instance. This only holds
+ # channels we're actually in, not those we've been asked to join
+ # but aren't in yet.
+ self.channels = util.InsensitiveDict()
+
+ # clear stale timers
+ if hasattr(self, 'requestedChannels'):
+ for timer in self.requestedChannels.itervalues():
+ if timer.active():
+ timer.cancel()
+
+ # A map from channel name to a DelayedCall instance representing its timeout
+ self.requestedChannels = util.InsensitiveDict()
+
+ def __repr__(self):
+ return "<Bot %r on network %s>" % (self.nickname, self.network)
+
+ def isFull(self):
+ return len(self.channels) + len(self.requestedChannels) >= self.network.maxChannels
+
+ def connectionMade(self):
+ """Called by IRCClient when we have a socket connection to the server."""
+ self.emptyChannels()
+ self.connectTimestamp = time.time()
+ self.network = self.factory.network
+ self.botNet = self.factory.botNet
+
+ # Start picking an initial nickname. This is really only expected to work
+ # on servers where this is the only CIA bot. If this one is in use, we get
+ # an ERR_NICKNAMEINUSE which we handle by picking a temporary nick we can
+ # use to search for a better one.
+ self.nickname = self.findNickQuickly()
+ self.nicknames = [self.nickname]
+ irc.IRCClient.connectionMade(self)
+
+ def irc_ERR_NICKNAMEINUSE(self, prefix, params):
+ """An alternate nickname-in-use error handler that generates a random
+ temporary nick. This will let us at least connect to the server and
+ issue WHOIS queries to efficiently find a better nick.
+
+ As soon as we get the nickChanged back from this operation, we will
+ realize this nick doesn't match those allowed by nickAllocator and
+ start looking for a better one.
+ """
+ tempNick = "CIA-temp%03d" % random.randint(0, 999)
+ self.nicknames.append(tempNick)
+ self.setNick(tempNick)
+
+ def sendLine(self, line):
+ # Override sendLine() to update txByteCount.
+ # Note that the text of 'line' doesn't count a CRLF,
+ # but we should include that in our buffer estimates.
+ self.txByteCount += len(line) + 2
+ irc.IRCClient.sendLine(self, line)
+
+ def nickChanged(self, newname):
+ irc.IRCClient.nickChanged(self, newname)
+ if self.botNet.nickAllocator.isValid(newname):
+ # The nick was valid. If we aren't completely connected yet, fix that
+ if self.signonTimestamp is None:
+ self.finishConnection()
+ else:
+ # We got a bad nick, try to find a better one. If it doesn't
+ # work within 1 minute, self-destruct.
+ log.msg("%r starting nick negotiation" % self)
+ reactor.callLater(60, self.enforceNickDeadline)
+ self.findNick().addCallback(self.foundBetterNick)
+
+ def enforceNickDeadline(self):
+ """If this bot doesn't have a valid nick yet, kill it."""
+ if not self.botNet.nickAllocator.isValid(self.nickname):
+ self.sendLine("QUIT")
+
+ def register(self, nickname, hostname='foo', servername='bar'):
+ """Twisted's default register() is silly in that it doesn't let us
+ specify a new username. We want all the usernames to be the
+ same, so filters can be written and things are just in general
+ better when bots are renaming themselves dynamically.
+ """
+ if self.password is not None:
+ self.sendLine("PASS %s" % self.password)
+ self.setNick(nickname)
+ self.sendLine("USER %s %s %s :%s" % (self.botNet.nickAllocator.username,
+ hostname,
+ servername,
+ self.botNet.nickAllocator.realname))
+
+ def foundBetterNick(self, nick):
+ log.msg("%r found a better nick, renaming to %r" % (self, nick))
+ self.nicknames.append(nick)
+ self.setNick(nick)
+
+ def findNickQuickly(self):
+ """This is used to get an initial nick during registration, before
+ we're allowed to make WHOIS queries. It only checks whether a nick
+ is already in use by one of our bots. If we happened to grab a nick
+ that's already in use, the server will rename us and our nickChanged()
+ handler will try to find a better nick.
+ """
+ for nick in self.botNet.nickAllocator.generate():
+ if not nick in self.botNet.networks.get(self.network, []):
+ return nick
+
+ def findNick(self):
+ """Find a new unused nickname for this bot. As this requires
+ testing whether each desired nickname is in use, it returns a Deferred.
+ """
+ result = defer.Deferred()
+ self._findNick(True, None, self.botNet.nickAllocator.generate(), result)
+ return result
+
+ def _findNick(self, isUsed, testedNick, generator, result):
+ """Callback implementing the guts of findNick.
+ On the first call, isUsed is True and testedNick is None.
+ We grab the next nick from the provided generator and
+ start testing it, providing this function as the deferred's
+ callback. If the nick isn't used, we send the nick to
+ our result callback. Otherwise, the cycle continues.
+ """
+ if isUsed:
+ nextNick = generator.next()
+ self.isNickUsed(nextNick).addCallback(self._findNick, nextNick, generator, result)
+ else:
+ result.callback(testedNick)
+
+ def isNickUsed(self, nick):
+ """Determine if the given nick is in use, using WHOIS.
+ Returns a Deferred that eventually resolves to a boolean.
+
+ If the server doesn't respond to the WHOIS, we assume the nick
+ isn't in use. This way if we're on a server that somehow has a broken
+ WHOIS, we end up with an ugly nick rather than sitting in an infinite loop.
+ """
+ result = defer.Deferred()
+
+ # First check whether any of our own bots are using this nick
+ for bot in self.botNet.networks.get(self.network, []):
+ if nick == bot.nickname:
+ result.callback(True)
+ return result
+
+ # It's not that easy- try a WHOIS query
+ if nick in self.pendingWhoisTests:
+ # We already have a request on the wire, tack another deferred onto it
+ self.pendingWhoisTests[nick].append(result)
+
+ else:
+ # Send a new request
+ self.pendingWhoisTests[nick] = [result]
+ self.sendLine("WHOIS %s" % nick)
+
+ return result
+
+ def irc_RPL_WHOISUSER(self, prefix, params):
+ """Reply to the WHOIS command we use to evaluate if a nick is used or not.
+ This one would indicate that the nick is indeed used.
+ """
+ nick = params[1]
+ if nick in self.pendingWhoisTests:
+ for result in self.pendingWhoisTests[nick]:
+ result.callback(True)
+ del self.pendingWhoisTests[nick]
+
+ def irc_ERR_NOSUCHNICK(self, prefix, params):
+ """Reply to the WHOIS command we use to evaluate if a nick is used or not.
+ This indicates that the nick is available.
+ """
+ nick = params[1]
+ if nick in self.pendingWhoisTests:
+ for result in self.pendingWhoisTests[nick]:
+ result.callback(False)
+ del self.pendingWhoisTests[nick]
+
+ # These are several other WHOIS replies that we want to ignore
+ def irc_RPL_WHOISSERVER(self, prefix, params):
+ pass
+ def irc_RPL_WHOISIDLE(self, prefix, params):
+ pass
+ def irc_RPL_WHOISCHANNELS(self, prefix, params):
+ pass
+ def irc_RPL_ENDOFWHOIS(self, prefix, params):
+ pass
+
+ def signedOn(self):
+ """IRCClient is notifying us that we've finished connecting to
+ the IRC server and can finally start joining channels.
+ """
+ self.emptyChannels()
+
+ # Check our initial nick, finish our connection if it was good
+ self.nickChanged(self.nickname)
+
+ def finishConnection(self):
+ log.msg("%r connected" % self)
+ self.botNet.botConnected(self)
+ self.signonTimestamp = time.time()
+
+ # Start the cycle of pinging the server to ensure our connection
+ # is still up and measure lag. IRC servers seem to often fail in
+ # ways that leave clients' sockets connected but ignore all data
+ # from them, and this lets us measure lag for free.
+ self._lagPingLoop()
+
+ def sendServerPing(self):
+ """Send a ping stamped with the current time and byte count"""
+ self.lastPingTransmitTimestamp = time.time()
+ self.sendLine("PING %s-%f" % (self.txByteCount, self.lastPingTransmitTimestamp))
+
+ def _lagPingLoop(self):
+ self.sendServerPing()
+ reactor.callLater(self.network.pingInterval, self._lagPingLoop)
+
+ def irc_PONG(self, prefix, params):
+ """Handle the responses to pings sent with sendServerPing. This compares
+ the timestamp in the pong (from when the ping was sent) and the current
+ time, storing the lag and the current time.
+ """
+ try:
+ # Most IRC servers send back a server name as params[0] then the
+ # ping argument as params[1].. but some (broken?) ones send back
+ # only a single argument, with the ping parameter.
+ if len(params) >= 2:
+ pingParam = params[1]
+ else:
+ pingParam = params[0]
+
+ byteCount, timestamp = pingParam.split("-")
+ self.txConfirmedBytes = int(byteCount)
+ self.lastPingTimestamp = float(timestamp)
+
+ except (ValueError, IndexError):
+ # This must be some broken IRC server that's not preserving our ping timestamp.
+ # The best we can do is assume this is the pong for the most recent ping we sent.
+ log.msg("%r received bad PONG reply: %r" % (self, params))
+ self.lastPingTimestamp = self.lastPingTransmitTimestamp
+ self.txConfirmedBytes = self.txByteCount
+
+ now = time.time()
+ self.lastPongTimestamp = now
+ self._pollMessageQueue(now)
+
+ def connectionLost(self, reason):
+ self.emptyChannels()
+ log.msg("%r disconnected" % self)
+ self.botNet.botDisconnected(self)
+ irc.IRCClient.connectionLost(self)
+
+ def getLag(self):
+ """Calculate a single figure for the lag between us and the server.
+ If pings have been coming back on time this is just the raw lag,
+ but if our latest ping has been particularly late, it's the average
+ of the latest successful ping's lag and the amount of time we've been
+ waiting for this late ping.
+ """
+ if self.lastPongTimestamp is None:
+ # We've never received a successful pong
+ if self.signonTimestamp is None:
+ # Hmm, we've also never signed on. Nothing more we can do
+ return None
+ else:
+ # We've signed on, but never received a good pong. Let's pretend
+ # the time since the last pong is just the time since signon- if
+ # we really have that funky of a connection this will at least eventually
+ # let us detect that.
+ timeSincePong = time.time() - self.signonTimestamp
+ lag = None
+ else:
+ timeSincePong = time.time() - self.lastPongTimestamp
+ if self.lastPingTimestamp is None:
+ lag = None
+ else:
+ lag = self.lastPongTimestamp - self.lastPingTimestamp
+
+ if timeSincePong < self.network.pingInterval * 2:
+ # We're doing fine, report the raw lag
+ return lag
+ else:
+ # Yikes, it's been a while since we've had a good pong.
+ # Weigh that in to the returned lag figure as described above.
+ return ((lag or 0) + (timeSincePong - self.network.pingInterval)) / 2
+
+ def join(self, channel):
+ """Called by the bot's owner to request that a channel be joined.
+ If this channel isn't already on our requests list, we send a join
+ command and set up a timeout.
+ """
+ if channel not in self.requestedChannels and channel not in self.channels:
+ self.requestedChannels[channel] = reactor.callLater(self.joinTimeout, self.joinTimedOut, channel)
+ irc.IRCClient.join(self, channel)
+
+ def cancelRequest(self, channel):
+ """Cancels a request to join the given channel if we have one"""
+ if channel in self.requestedChannels:
+ timer = self.requestedChannels[channel]
+ if timer.active():
+ timer.cancel()
+ del self.requestedChannels[channel]
+
+ def joinTimedOut(self, channel):
+ """Our join() timed out, remove the channel from our request list"""
+ self.cancelRequest(channel)
+
+ def part(self, channel):
+ """Called to request that a bot leave the given channel.
+ This removes the channel from our requests list if necessary
+ before sending the part command.
+ """
+ self.cancelRequest(channel)
+ irc.IRCClient.part(self, channel)
+
+ def joined(self, channel):
+ """A channel has successfully been joined"""
+ log.msg("%r joined %r" % (self, channel))
+ self.cancelRequest(channel)
+ self.channels[channel] = ChannelInfo(channel)
+ self.factory.botNet.botJoined(self, channel)
+
+ def left(self, channel):
+ """Called when part() is successful and we've left a channel.
+ Implicitly also called when we're kicked via kickedFrom().
+ """
+ log.msg("%r left %r" % (self, channel))
+ del self.channels[channel]
+ self.factory.botNet.botLeft(self, channel)
+
+ def kickedFrom(self, channel, kicker, message):
+ log.msg("%r was kicked from %r by %r: %r" % (self, channel, kicker, message))
+ self.left(channel)
+ self.factory.botNet.botKicked(self, channel, kicker, message)
+
+ def ctcpUnknownQuery(self, user, channel, tag, data):
+ """Ignore unknown queries, so if someone sends a CTCP BAGEL to
+ the channel CIA doesn't respond needlessly.
+ """
+ pass
+
+ def irc_unknown(self, prefix, command, params):
+ """Log unknown commands, making debugging easier. This also lets
+ us see responses in the log for commands sent via debug_tool.
+ """
+ if command in self.ignoredCommands:
+ return
+
+ log.msg("%r received unknown IRC command %s: %r" % (self, command, params))
+
+ self.factory.botNet.unknownMessageLog.log((
+ time.time(),
+ self.nickname,
+ str(self.network),
+ command,
+ repr(params)[1:-1],
+ ))
+
+ def topicUpdated(self, user, channel, newTopic):
+ self.channels[channel].topic = newTopic
+
+ def irc_JOIN(self, prefix, params):
+ """This is a modified implementation that checks the nick against
+ both our current nickname and our previous one. This hopefully
+ avoids a race condition when we're joining a channel and changing
+ our nick at nearly the same time.
+ """
+ nick = prefix.split('!')[0]
+ channel = params[-1]
+ if nick in self.nicknames:
+ self.joined(channel)
+ else:
+ self.userJoined(nick, channel)
+
+ def irc_NICK(self, prefix, params):
+ """This is a modified implementation that checks the nick against
+ both our current nickname and our previous one. This ensures
+ that we get confirmation for our own nick changes.
+ """
+ nick = prefix.split('!')[0]
+ if nick in self.nicknames:
+ self.nickChanged(params[0])
+ else:
+ self.userRenamed(nick, params[0])
+
+ def irc_RPL_NAMREPLY(self, prefix, params):
+ """Collect usernames from this channel. Several of these
+ messages may be sent to cover the channel's full nicklist.
+ An RPL_ENDOFNAMES signals the end of the list.
+ """
+ # We just separate these into individual nicks and stuff them in
+ # the nickCollector, transferred to 'nicks' when we get the RPL_ENDOFNAMES.
+ channel = self.channels[params[2]]
+ for name in params[3].split():
+ # Remove operator and voice prefixes
+ if name[0] in '@+':
+ name = name[1:]
+ channel.nickCollector.append(name)
+
+ def irc_RPL_ENDOFNAMES(self, prefix, params):
+ """This is sent after zero or more RPL_NAMREPLY commands to
+ terminate the list of users in a channel.
+ """
+ channel = self.channels[params[1]]
+ channel.nicks = channel.nickCollector
+ channel.nickCollector = []
+
+ def userJoined(self, user, channel):
+ """Update the channel's nick list when we see someone join"""
+ self.channels[channel].nicks.append(user)
+
+ def irc_QUIT(self, prefix, params):
+ # Another user quit, remove them from any of our channel lists
+ nick = prefix.split('!')[0]
+ for channel in self.channels.itervalues():
+ try:
+ channel.nicks.remove(nick)
+ except ValueError:
+ pass
+
+ def userLeft(self, user, channel):
+ """Update the channel's nick list when a user voluntarily leaves"""
+ self.channels[channel].nicks.remove(user)
+
+ def userKicked(self, user, channel, kicker, message):
+ """Update the channel's nick list when a user is kicked"""
+ self.channels[channel].nicks.remove(user)
+
+ def userRenamed(self, oldname, newname):
+ # Blah, this doesn't give us a channel name. Search for this user
+ # in each of our channels, renaming them.
+ for channel in self.channels.itervalues():
+ try:
+ channel.nicks.remove(oldname)
+ channel.nicks.append(newname)
+ except ValueError:
+ pass
+
+ def queueMessage(self, target, text):
+ """A msg() workalike which queues messages and provides flood protection.
+ Text sent with queueMessage() isn't guaranteed to ever be sent to the
+ server.
+ """
+ self._messageQueue.put(target, text)
+ self._pollMessageQueue(time.time())
+
+ def _pollMessageQueue(self, now):
+ """Check whether it's safe to send any queued messages"""
+ while self._messageQueue.peek():
+ target, text = self._messageQueue.peek()
+
+ # Length estimate: Includes the "PRIVMSG %s :%s" boilerplate, plus the CRLF
+ length = len(text) + len(target) + 12
+
+ # Make a worst-case prediction of the server's buffer fill level after we
+ # would have sent this string.
+ predictedFill = self.txByteCount - self.txConfirmedBytes
+
+ # If we're up to half our buffer fill and we haven't sent a ping
+ # recently, send another in an attempt to lower the predicted
+ # fill estimate.
+ if predictedFill > self.network.bufferSize / 2 and self.lastPingTransmitTimestamp + 5 < now:
+ self.sendServerPing()
+
+ # If we'd go above the hard limit, we can't send the message.
+ if predictedFill > self.network.bufferSize:
+ break
+
+ self.msg(target, text)
+ self._messageQueue.flush()
+
+ def action(self, user, channel, message):
+ """Just for fun"""
+ text = message.lower().strip()
+ me = self.nickname.lower()
+ them = user.split('!')[0]
+
+ if text == 'hugs %s' % me:
+ self.me(channel, 'hugs %s' % them)
+
+ elif text == 'kicks %s' % me:
+ self.say(channel, 'ow')
+
+ elif text == 'kills %s' % me:
+ self.me(channel, 'dies')
+
+ elif text == 'eats %s' % me:
+ self.me(channel, 'tastes crunchy')
+
+ elif text == "rubs %s's tummy" % me:
+ self.say(channel, "*purr*")
+
+ def remote_getInfoDict(self):
+ return {
+ 'nickname': self.nickname,
+ 'current_channels': self.channels.keys(),
+ 'requested_channels': self.requestedChannels.keys(),
+ 'network': self.remote_getNetworkInfo(),
+ 'current_time': time.time(),
+ 'connect_time': self.connectTimestamp,
+ 'inactive_time': self.remote_getInactivity(),
+ 'is_full': self.isFull(),
+ 'lag': self.getLag(),
+ }
+
+ def remote_msg(self, target, text):
+ """A remote request directly to this bot, ignoring the usual queueing"""
+ self.msg(target, text)
+
+ def remote_getNickname(self):
+ return self.nickname
+
+ def remote_getChannels(self):
+ return self.channels.keys()
+
+ def remote_getRequestedChannels(self):
+ # Convert from an InsensitiveDict to a normal one
+ return self.requestedChannels.keys()
+
+ def remote_getNetworkInfo(self):
+ """Returns a (networkName, host, port) tuple"""
+ host, port = self.transport.addr
+ return (str(self.network), host, port)
+
+ def remote_repr(self):
+ return repr(self)
+
+ def remote_getConnectTimestamp(self):
+ return self.connectTimestamp
+
+ def remote_getInactivity(self):
+ """If this bot is inactive, returns the time at which it will be garbage
+ collected. If not, returns None.
+ """
+ if self in self.botNet.inactiveBots:
+ return self.botNet.inactiveBots[self].getTime()
+
+ def remote_isFull(self):
+ return self.isFull()
+
+ def remote_isEmpty(self):
+ return (not self.channels) and (not self.requestedChannels)
+
+ def remote_getStatusText(self):
+ """Get a textual description of this bot's status"""
+ indicators = []
+
+ if self.remote_isFull():
+ indicators.append('full')
+
+ if self.remote_isEmpty():
+ indicators.append('empty')
+
+ timer = self.remote_getInactivity()
+ if timer:
+ indicators.append('GC in %s' % TimeUtil.formatDuration(timer - time.time()))
+
+ return ', '.join(indicators)
+
+ def remote_getLag(self):
+ return self.getLag()
+
+
+class BotFactory(protocol.ClientFactory):
+ """Twisted ClientFactory for creating Bot instances"""
+ protocol = Bot
+
+ def __init__(self, botNet, network):
+ self.botNet = botNet
+ self.network = network
+
+ host, port = network.getNextServer()
+ log.msg("Using server %s:%s for %r" % (host, port, network))
+ reactor.connectTCP(host, port, self)
+
+ def clientConnectionLost(self, connector, reason):
+ log.msg("IRC Connection to %r lost: %r" % (self.network, reason))
+
+ def clientConnectionFailed(self, connector, reason):
+ log.msg("IRC Connection to %r failed: %r" % (self.network, reason))
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/IRC/Formatting.py
===================================================================
--- trunk/community/infrastructure/LibCIA/IRC/Formatting.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/IRC/Formatting.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,206 @@
+""" LibCIA.IRC.Formatting
+
+A thin abstraction for IRC formatting codes, and a converter
+that generates IRC-formatted text from a <colorText> XML document.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from LibCIA import XML, ColorText
+import types
+
+
+class FormattingCode(object):
+ """Represents a code used to format text in IRC"""
+ codes = {
+ # Colors
+ "black" : "\x0301",
+ "dark blue" : "\x0302",
+ "dark green" : "\x0303",
+ "green" : "\x0303",
+ "red" : "\x0304",
+ "light red" : "\x0304",
+ "dark red" : "\x0305",
+ "purple" : "\x0306",
+ "brown" : "\x0307", # On some clients this is orange, others it is brown
+ "orange" : "\x0307",
+ "yellow" : "\x0308",
+ "light green" : "\x0309",
+ "aqua" : "\x0310",
+ "light blue" : "\x0311",
+ "blue" : "\x0312",
+ "violet" : "\x0313",
+ "grey" : "\x0314",
+ "gray" : "\x0314",
+ "light grey" : "\x0315",
+ "light gray" : "\x0315",
+ "white" : "\x0316",
+
+ # Other formatting
+ "normal" : "\x0F",
+ "bold" : "\x02",
+ "reverse" : "\x16",
+ "underline" : "\x1F",
+ }
+
+ def __init__(self, name):
+ self.name = name
+ self.value = self.codes[name]
+
+ def __str__(self):
+ return self.value
+
+
+def format(text, *codeNames):
+ """Apply each formatting code from the given list of code names
+ to the given text, returnging a string ready for consumption
+ by an IRC client.
+ """
+ if codeNames:
+ codes = "".join([str(FormattingCode(codeName)) for codeName in codeNames])
+ return codes + text + str(FormattingCode('normal'))
+ else:
+ return text
+
+
+class ColorStack:
+ """This is An important building block for converting tree-structured documents
+ with color into IRC-formatted text. The document is represented as a list of
+ FormattingCodes and/or strings. This document can be wrapped in additional
+ formatting layers, using our stack to restore codes properly after a 'normal'
+ code is issued.
+
+ Normal usage is to push() a list of formatting codes, evaluate all children
+ of that formatting level, then wrap() the result of that evaluation.
+ """
+ def __init__(self):
+ self.empty()
+
+ def empty(self):
+ self.codes = []
+
+ def push(self, *codeNames):
+ self.codes.append([FormattingCode(name) for name in codeNames])
+
+ def wrap(self, children):
+ parent = self.codes.pop()
+
+ if not children:
+ # If we have nothing to wrap, return nothing. This is important
+ # for the correct operation of the <autoHide> modular formatter
+ # component, since a 'normal' code by itself still prevents it
+ # from hiding the group.
+ return []
+
+ parent.extend(children)
+
+ # An important optimization- since we're about to insert new codes
+ # for everything in codeStack, remove codes from our 'parent' list until
+ # we hit actual text. This prevents sequences from appearing in the output
+ # where several codes are applied then immediately erased by a 'normal' code.
+ # This also handles optimizing out formatting codes with no children.
+ while parent and isinstance(parent[-1], FormattingCode):
+ del parent[-1]
+
+ # Now stick on a 'normal' code and the contents of our stack,
+ # to revert the codes we were given.
+ parent.append(FormattingCode('normal'))
+ for level in self.codes:
+ parent.extend(level)
+ return parent
+
+
+def parseColorElement(xml):
+ """Given a <color> element, return the corresponding list of color code names"""
+ codes = []
+ bg = xml.getAttributeNS(None, 'bg')
+ fg = xml.getAttributeNS(None, 'fg')
+
+ if bg:
+ if bg in ColorText.allowedColors:
+ codes.append(bg)
+ codes.append('reverse')
+ else:
+ raise XML.XMLValidityError("%r is not a color" % bg)
+ if fg:
+ if fg in ColorText.allowedColors:
+ codes.append(fg)
+ else:
+ raise XML.XMLValidityError("%r is not a color" % fg)
+ return codes
+
+
+class ColortextFormatter(XML.XMLObjectParser):
+ r"""Given a DOM tree with <colorText>-formatted text
+ generate an equivalent message formatted for IRC.
+
+ >>> f = ColortextFormatter()
+ >>> f.parse(XML.parseString('<colorText><u><b>Hello</b> World</u></colorText>'))
+ '\x1f\x02Hello\x0f\x1f World\x0f'
+
+ >>> f.parse(XML.parseString(
+ ... "<colorText>" +
+ ... "<color bg='dark blue'><color fg='yellow'>" +
+ ... "<b>hello</b>" +
+ ... "</color></color>" +
+ ... "<u> world</u>" +
+ ... "</colorText>"))
+ '\x0302\x16\x0308\x02hello\x0f\x1f world\x0f'
+
+ """
+
+ def element_colorText(self, node):
+ # The root element converts our list of formatting codes to a flat string
+ codes = []
+ self.colorStack = ColorStack()
+ for childCodes in self.childParser(node):
+ codes.extend(childCodes)
+ return "".join(map(str, codes))
+
+ def codeWrap(self, node, *codeNames):
+ """Wrap the children of the given xml element with the given formatting codes.
+ This prepends the code list and appends a 'normal' tag, using colorStack
+ to restore any codes we don't want to disable with the 'normal' tag.
+ """
+ self.colorStack.push(*codeNames)
+ children = []
+ for child in self.childParser(node):
+ children.extend(child)
+ return self.colorStack.wrap(children)
+
+ def element_b(self, xml):
+ """Just wrap our contents in a bold tag"""
+ return self.codeWrap(xml, 'bold')
+
+ def element_u(self, xml):
+ """Just wrap our contents in an underline tag"""
+ return self.codeWrap(xml, 'underline')
+
+ def element_br(self, xml):
+ """Insert a literal newline"""
+ return ["\n"]
+
+ def element_color(self, xml):
+ """Generates formatting codes appropriate to represent a foreground and/or background color"""
+ return self.codeWrap(xml, *parseColorElement(xml))
+
+ def parseString(self, text):
+ return [text]
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/IRC/Handler.py
===================================================================
--- trunk/community/infrastructure/LibCIA/IRC/Handler.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/IRC/Handler.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,328 @@
+""" LibCIA.IRC.Handler
+
+The irc:// URI handler, acting as a frontend to our network of bots
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.spread import pb
+from twisted.internet import defer, reactor
+from twisted.python import log
+from LibCIA import Ruleset
+
+
+class IrcURIHandler(Ruleset.RegexURIHandler):
+ """Handles irc:// URIs in rulesets. This creates a message queue
+ for each URI, and delivers formatted messages to the proper
+ message queue instance.
+
+ This follows a subset of the internet draft at:
+
+ http://www.w3.org/Addressing/draft-mirashi-url-irc-01.txt
+
+ In short, the following sorts of URIs are valid:
+
+ irc://irc.foo.net/boing
+ Refers to the channel #boing at irc.foo.net on the default port
+
+ irc://irc.foo.net:1234/boing
+ Refers to the channel #boing at irc.foo.net, on port 1234
+
+ irc://irc.foo.net/muffin,isnick
+ Refers to a users with the nick 'muffin' on irc.foo.net's default port
+
+ The full hostname may also be replaced with a network name, as defined
+ in Network.py. This implies a random choice from multiple servers, and possibly
+ other network-specific behaviour.
+
+ This is starting to be updated to a newer irc:// draft that recommends
+ including the #, but still supports the above URIs.
+ """
+ scheme = 'irc'
+ regex = r"""
+ ^irc://(?P<host>( ([a-zA-Z]([a-zA-Z0-9.-]*[a-zA-Z0-9])?) |
+ ([0-9]+(\.[0-9]+){3})
+ ))
+ (:(?P<port>[0-9]+))?/(?P<target>[^\s,]+)(?P<isnick>,isnick)?$
+ """
+
+ def __init__(self, remoteBots):
+ # A map from URI to message queue
+ self.queueMap = {}
+ self.remoteBots = remoteBots
+ Ruleset.RegexURIHandler.__init__(self)
+
+ def createQueueFromURI(self, uri):
+ """Convert a URI to a new message queue instance"""
+ d = self.parseURI(uri)
+
+ if d['isnick']:
+ # This refers to a nickname- deliver private messages to that user
+ return PrivateMessageQueue(self.remoteBots, d['host'], d['port'], d['target'])
+ else:
+ # It's a channel. Add the # if necesary.
+ channel = d['target']
+ if channel[0] != '#':
+ channel = '#' + channel
+ return ChannelMessageQueue(self.remoteBots, d['host'], d['port'], channel)
+
+ def assigned(self, uri, newRuleset):
+ # If this URI is new, create a new message queue for it
+ if not uri in self.queueMap:
+ q = self.createQueueFromURI(uri)
+ self.queueMap[uri] = q
+
+ def unassigned(self, uri):
+ self.queueMap[uri].cancel()
+ del self.queueMap[uri]
+
+ def message(self, uri, message, content):
+ self.queueMap[uri].send(unicode(content).encode('utf-8'))
+
+ def rulesetsRefreshed(self):
+ """Synchronize our requests with the server's only once rulesets have
+ refreshed, so we don't inadvertently delete a request that hasn't
+ finished loading yet.
+ """
+ self.remoteBots.allowSync = True
+ self.remoteBots.syncRequests()
+
+
+class RequestIdentity:
+ """An object that can represent the identity of a Request remotely"""
+ __slots__ = ["host", "port", "channel"]
+ def __init__(self, host, port, channel):
+ self.__dict__.update(dict(
+ host = host,
+ port = port,
+ channel = channel,
+ ))
+
+ def __hash__(self):
+ return hash((self.host, self.port, self.channel))
+
+ def __cmp__(self, other):
+ return cmp((self.host, self.port, self.channel),
+ (other.host, other.port, other.channel))
+
+ def __setattr__(self, name, value):
+ raise TypeError("RequestIdentity instances are immuatble")
+
+
+class ReconnectingPBClient(pb.PBClientFactory):
+ """A PBClientFactory that automatically tries to reconnect
+ using a supplied method if the connection fails.
+ """
+ def __init__(self, reconnector, delay=4):
+ self.reconnector = reconnector
+ self.delay = delay
+ pb.PBClientFactory.__init__(self)
+
+ def clientConnectionFailed(self, connector, reason):
+ reactor.callLater(self.delay, self.reconnector)
+
+ def clientConnectionLost(self, connector, reason):
+ reactor.callLater(self.delay, self.reconnector)
+
+
+class RemoteBots:
+ """Represents a bot server running elsewhere, that we connect to
+ using Perspective Broker. This caches references to currently
+ registered requests, and ensures that the remote request list
+ matches the local one even when the bot server restarts.
+ """
+ def __init__(self, socketName):
+ self.socketName = socketName
+ self.botNet = None
+ self.factory = ReconnectingPBClient(self.connect)
+ self.allowSync = False
+
+ # Maps RequestIdentity instances to remote requests. Requests
+ # that should exist but don't yet are mapped to None.
+ self.requestMap = {}
+
+ self.connect()
+
+ def connect(self):
+ log.msg("Trying to connect to bots on socket %r..." % self.socketName)
+ self.botNet = None
+ reactor.connectUNIX(self.socketName, self.factory)
+ if not self.factory.rootObjectRequests:
+ self.factory.getRootObject().addCallback(
+ self._gotRootObject)
+
+ def _gotRootObject(self, root):
+ log.msg("Connected to bot server")
+ self.botNet = root
+
+ if self.allowSync:
+ # Sync on connect only if we can be sure we've loaded all requests
+ self.syncRequests()
+
+ def findRequest(self, requestId):
+ """Find a remote Request reference matching the given request ID,
+ creating it if necessary. Returns the remote reference via
+ a Deferred. Returns None if the request doesn't exist yet
+ and can't be created (for example, if the bot server is down).
+ Even if the bot server is unavailable, this request is added
+ to our local list.
+ """
+ result = defer.Deferred()
+
+ if self.requestMap.get(requestId) and self.botNet:
+ # This request already exists
+ result.callback(self.requestMap[requestId])
+ else:
+ # It doesn't exist. Mark that it should
+ self.requestMap[requestId] = None
+ if self.botNet:
+ # If we have a connection to the bot net, try to add the request
+ self.botNet.callRemote("findRequest", requestId.host, requestId.port,
+ requestId.channel).addCallback(
+ self._findRequest, requestId, result).addErrback(
+ result.errback)
+ else:
+ # Nope, give up for now
+ result.callback(None)
+
+ return result
+
+ def _findRequest(self, requestRef, requestId, result):
+ self.requestMap[requestId] = requestRef
+ result.callback(requestRef)
+
+ def removeRequest(self, requestId):
+ """Ask the bot server to remove a request, given a RequestIdentity.
+ Returns, via a Deferred, True if the request could be removed
+ or False if we can't currently contact the bot server. In either
+ case, the request is removed from our local list.
+ """
+ result = defer.Deferred()
+
+ if requestId in self.requestMap:
+ if self.botNet:
+ # We already have a reference to this request and our server is up.
+ # Send the request a message to cancel itself.
+ self.requestMap[requestId].callRemote("cancel").addCallback(
+ self._cancelledRequest, result).addErrback(
+ result.errback)
+ else:
+ result.callback(False)
+
+ # Whether the server is up or not, remove it from our local list
+ del self.requestMap[requestId]
+
+ else:
+ # We don't have a reference to it. See if it exists server-side
+ if self.botNet:
+ self.botNet.callRemote("findRequest", requestId.host, requestId.port,
+ requestId.channel, False).addCallback(
+ self._removeFoundRequest, result).addErrback(
+ result.errback)
+ else:
+ result.callback(False)
+
+ return result
+
+ def _cancelledRequest(self, completed, result):
+ # Our request was successfully cancelled
+ result.callback(True)
+
+ def _removeFoundRequest(self, ref, result):
+ if ref:
+ # There is indeed an instance of this request on the server. Cancel it.
+ ref.callRemote("cancel").addCallback(
+ self._cancelledRequest, result).addErrback(
+ result.errback)
+ else:
+ # Nothing to do
+ result.callback(True)
+
+ def _removeRequest(self, requestRef, requestId, result):
+ self.requestMap[requestId] = requestRef
+ result.callback(requestRef)
+
+ def syncRequests(self):
+ """Flush our local references to remote request objects. If a server
+ connection has been established, this retrieves new request references
+ and adds and removes requests as necessary.
+ """
+ log.msg("Synchronizing bot requests")
+ for id in self.requestMap.keys():
+ self.requestMap[id] = None
+
+ if self.botNet:
+ # Get the server's request list
+ self.botNet.callRemote("getRequests").addCallback(
+ self._syncRequests)
+
+ def _syncRequests(self, serverRequests):
+ for (host, port, channel), ref in serverRequests.iteritems():
+ id = RequestIdentity(host, port, channel)
+
+ if id in self.requestMap:
+ # Good, this is a request that we need and we have. Save the reference
+ self.requestMap[id] = ref
+
+ else:
+ # This is a request that the server has but we don't need. Cancel it
+ ref.callRemote("cancel")
+
+ # Start trying to reestablish all remaining requests with no reference
+ for id, ref in self.requestMap.iteritems():
+ if not ref:
+ self.findRequest(id)
+
+
+class MessageQueue:
+ """Abstract base class for a queue we can deliver IRC messages to"""
+ def __init__(self, remoteBots, host, port, channel, target):
+ self.target = target
+ self.remoteBots = remoteBots
+ self.requestId = RequestIdentity(host, port, channel)
+
+ self.remoteBots.findRequest(self.requestId)
+
+ def cancel(self):
+ """Cancel the request associated with this message queue"""
+ self.remoteBots.removeRequest(self.requestId)
+
+ def send(self, message):
+ """Split up a message into lines and queue it for transmission"""
+ self.remoteBots.findRequest(self.requestId).addCallback(self._send, message)
+
+ def _send(self, request, message):
+ """Once we have a reference to the remote request, send our message"""
+ # If the request doesn't exist yet on the server, ignore this.
+ if request:
+ request.callRemote("msgList", self.target, message.split('\n'))
+
+
+class PrivateMessageQueue(MessageQueue):
+ """Send private messages to a particular user, using one bot"""
+ def __init__(self, remoteBots, host, port, nick):
+ MessageQueue.__init__(self, remoteBots, host, port, None, nick)
+
+
+class ChannelMessageQueue(MessageQueue):
+ """Send messages to a channel, using multiple bots if necessary"""
+ def __init__(self, remoteBots, host, port, channel):
+ MessageQueue.__init__(self, remoteBots, host, port, channel, channel)
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/IRC/Network.py
===================================================================
--- trunk/community/infrastructure/LibCIA/IRC/Network.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/IRC/Network.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,193 @@
+""" LibCIA.IRC.Network
+
+IRC Networks define a collection of servers, and optionally
+define special behaviour relevant to that IRC network.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+
+class BaseNetwork:
+ """Base class for IRC networks. A network may consist of multiple servers-
+ it provides a 'connect' method that picks one to use. Networks may also
+ define other IRC-network-specific behaviour.
+ """
+ alias = None
+ defaultPort = 6667
+ currentServer = 0
+
+ # Maximum number of IRC channels a single bot is allowed in
+ maxChannels = 18
+
+ # Timeout, in seconds, for creating new bots. By default this is 16
+ # minutes, since most of the smaller networks we're on get annoyed
+ # if we retry too quickly. We override this in the larger networks below.
+ newBotTimeout = 60 * 16
+
+ # Maximum amount of data to buffer server-side, in bytes. Larger values improve
+ # speed, but if this is larger than a fixed-size buffer on the server we could be
+ # flood-kicked.
+ bufferSize = 1024
+
+ # How often to ping the server, in seconds
+ pingInterval = 60
+
+ def __str__(self):
+ return self.alias
+
+ def getIdentity(self):
+ """Return a (host, port) tuple that can be used to return this network
+ from a search.
+ """
+ return (self.alias, None)
+
+ def __repr__(self):
+ return "<IRC.Network.%s>" % self.__class__.__name__
+
+ def __cmp__(self, other):
+ if type(self) is type(other) and (
+ isinstance(self, BaseNetwork) and isinstance(other, BaseNetwork)):
+ return cmp(self.alias or self.servers,
+ other.alias or other.servers)
+ else:
+ return cmp(self, other)
+
+ def __hash__(self):
+ return hash(self.alias)
+
+ def getNextServer(self):
+ """Return the next server, as a (host, port) tuple, to use for this network."""
+ # By default, just try them all round-robin style
+ self.currentServer = self.currentServer % len(self.servers)
+ host, port = self.servers[self.currentServer]
+ self.currentServer += 1
+ if port is None:
+ port = self.defaultPort
+ return (host, port)
+
+
+class GenericNetwork(BaseNetwork):
+ """A generic IRC network has no alias, refers to only one server, and
+ uses defaults for all other parameters. This is used when a particular
+ network isn't specified, and it acts as the base class for all other
+ networks.
+ """
+ def __init__(self, host, port=None):
+ if port is not None:
+ port = int(port)
+ self.servers = ( (host, port), )
+
+ def getIdentity(self):
+ return self.servers[0]
+
+ def __str__(self):
+ host, port = self.servers[0]
+ if port is None:
+ return host
+ else:
+ return "%s:%d" % (host, port)
+
+ def __repr__(self):
+ return "<IRC.Network.%s %s>" % (self.__class__.__name__, self)
+
+ def __hash__(self):
+ return hash(self.servers)
+
+
+class Freenode(BaseNetwork):
+ alias = 'freenode'
+ # Really short timeout- there are a lot of freenode servers and they can
+ # take it. Reconnecting all hosts to Freenode just takes way too long with
+ # the default value.
+ newBotTimeout = 60
+ servers = (
+ ('irc.freenode.net', None),
+ ('saberhagen.freenode.net', None),
+ ('kornbluth.freenode.net', None),
+ ('orwell.freenode.net', None),
+ ('sterling.freenode.net', None),
+ ('calvino.freenode.net', None),
+ ('adams.freenode.net', None),
+ ('leguin.freenode.net', None),
+ )
+
+class Undernet(BaseNetwork):
+ alias = 'undernet'
+ servers = (
+ ('london.uk.eu.undernet.org', None),
+ ('washington.dc.us.undernet.org', None),
+ ('amsterdam.nl.eu.undernet.org', None),
+ )
+
+class Worldforge(BaseNetwork):
+ alias = 'worldforge'
+ servers = (
+ ('lester.mithis.com', None),
+ #('irc.worldforge.org', None),
+ #('purple.worldforge.org', None),
+ )
+
+class IRCNet(BaseNetwork):
+ alias = 'ircnet'
+ servers = (
+ #('irc.osanet.cz', None),
+ #('irc.felk.cvut.cz', 6666),
+
+ # Bert Hubert claims either of these should be fine
+ ('us.ircnet.org', None),
+ ('irc.choopa.net', None),
+ )
+
+class EFNet(BaseNetwork):
+ alias = 'efnet'
+ servers = (
+ ('irc.desync.com', None),
+ ('efnet.xs4all.nl', None),
+ ('irc.mzima.net', None),
+ )
+
+_aliasCache = None
+_instCache = {}
+
+def find(host, port=None):
+ """Find a network corresponding to the given host (or alias) and port"""
+ global _aliasCache
+ global _instCache
+
+ if _aliasCache is None:
+ # Cache a dict mapping aliases to classes
+ _aliasCache = {}
+ for name, obj in globals().iteritems():
+ if type(obj) is type(BaseNetwork) and issubclass(obj, BaseNetwork):
+ _aliasCache[obj.alias] = obj
+
+ if host in _aliasCache:
+ # It's a known alias, return a cached instance of that network class
+ try:
+ return _instCache[host]
+ except KeyError:
+ inst = _aliasCache[host]()
+ _instCache[host] = inst
+ return inst
+
+ else:
+ # Create a new GenericNetwork for this host and port
+ return GenericNetwork(host, port)
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/IRC/__init__.py
===================================================================
--- trunk/community/infrastructure/LibCIA/IRC/__init__.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/IRC/__init__.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,25 @@
+""" LibCIA.IRC
+
+A package containing all of CIA's IRC-related functionality. This includes
+a small library for managing collections of IRC bots, utilities for IRC
+color formatting, and the irc:// URI handler.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
Added: trunk/community/infrastructure/LibCIA/IncomingMail.py
===================================================================
--- trunk/community/infrastructure/LibCIA/IncomingMail.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/IncomingMail.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,170 @@
+""" LibCIA.IncomingMail
+
+Includes an IncomingMailParser that converts emails to Messages
+and delivers them to the Message.Hub.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from Message import Message
+from ColorText import ColorTextParser
+import XML, RpcServer
+import email
+
+
+# List of email headers worth logging.
+# This should cover all those that are really interesting
+# from a security or identity point of view. Most other headers
+# are redundant with other information included in the message.
+interestingHeaders = (
+ 'From',
+ 'Received',
+ 'Message-Id',
+ 'Date'
+ )
+
+
+class MailInterface(RpcServer.Interface):
+ """An XML-RPC interface for delivering messages via an IncomingMailParser
+ to the Message.Hub.
+ """
+ def __init__(self, hub):
+ self.hub = hub
+ RpcServer.Interface.__init__(self)
+
+ def xmlrpc_deliver(self, message):
+ """Given the raw text of an email message, log it and process it if applicable."""
+ parsed = IncomingMailParser().parseString(message)
+ if parsed:
+ self.hub.deliver(parsed)
+
+
+class IncomingMailParser:
+ """Parses commands from incoming email messages, generating an XML Message
+ object representing its contents. This returned object can be dispatched
+ by a Message.Hub.
+ """
+ def parseString(self, string):
+ """Convert the given string to an email.Message, then parse it"""
+
+ # The 'email' module does not support Unicode objects! It will
+ # generate a garbage message if we pass it in.
+ if type(string) is unicode:
+ string = string.encode('utf8')
+
+ return self.parse(email.message_from_string(string))
+
+ def parse(self, message):
+ """Given an email.Message instance, determines the command it represents
+ (if any) and passes control to it.
+ """
+ self.message = message
+ subject = message['Subject']
+ # No subject, ignore this mail
+ if not subject:
+ return None
+ subject = subject.strip()
+ if not subject:
+ return None
+
+ # The subject line is formatted like a simple command line
+ subjectFields = subject.split(" ")
+ command = subjectFields[0]
+ args = subjectFields[1:]
+
+ try:
+ f = getattr(self, "command_" + command)
+ except AttributeError:
+ # Unknown command, ignore this mail
+ return None
+
+ # Pass on control to the command_* function...
+ xml = f(*args)
+
+ # If the command generated a message, perform some common postprocessing
+ if xml:
+ return Message(self.postprocessMessage(xml))
+
+ def postprocessMessage(self, xml):
+ """Gets a chance to modify all XML messages before they're loaded
+ and dispatched to the Hub. This does the following:
+ - If there is no <generator> at all, adds a generic one
+ - Removes any <mailHeaders> tag that may already exist in <generator>
+ - Adds a correct <mailHeaders> tag to the <generator>
+ """
+ # Create the <generator> tag if it doesn't exist
+ if not XML.dig(xml, "message", "generator"):
+ xml.documentElement.appendChild(self.getLocalGenerator(xml))
+ generator = XML.dig(xml, "message", "generator")
+
+ # Delete an existing <mailHeaders>
+ for child in list(XML.getChildElements(generator)):
+ if child.nodeName == "mailHeaders":
+ generator.removeChild(child)
+
+ # Add a new <mailHeaders>
+ generator.appendChild(self.getXMLMailHeaders(xml))
+ return xml
+
+ def getLocalGenerator(self, document):
+ """Return a <generator> tag for messages produced locally"""
+ node = document.createElementNS(None, "generator")
+ XML.addElement(node, "name", content="CIA IncomingMailParser")
+ return node
+
+ def getXMLMailHeaders(self, document):
+ """Return a <mailHeaders> tag representing a subset of the headers
+ for this message. This is placed in the <generator> tag of any
+ message passing through this module, to document and log the
+ message's true source.
+ """
+ node = document.createElementNS(None, "mailHeaders")
+ for name, value in self.message.items():
+ if name in interestingHeaders:
+ XML.addElement(node, "header", content=str(value)).setAttributeNS(None, 'name', name)
+ return node
+
+ def command_Announce(self, project):
+ """Old-style announcements: Announce <project> in the subject line.
+ The body of the email contained the message's text, marked up
+ with {color} tags but with no metadata.
+ """
+ xml = XML.createRootNode()
+
+ # Convert the given project name to a <project> tag inside <source>,
+ # after filtering it a bit... in the old CIA project names and IRC channel
+ # names weren't particularly distinct, so preceeding "#" characters on
+ # projects were ignored. We preserve this behaviour.
+ if project[0] == "#":
+ project = project[1:]
+ XML.buryValue(xml, project, "message", "source", "project")
+
+ # Since old-style commits didn't have any metadata, the best we can do
+ # is to represent the log in a <colorText> element
+ colorText = ColorTextParser().parse(self.message.get_payload()).documentElement
+ XML.bury(xml, "message", "body").appendChild(xml.importNode(colorText, True))
+ return xml
+
+ def command_DeliverXML(self):
+ """Deliver a message already formatted in XML"""
+ # Note that parseString will convert UTF8 to Unicode for us.
+ return XML.parseString(self.message.get_payload())
+
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Message.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Message.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Message.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,718 @@
+""" LibCIA.Message
+
+Classes to represent, distribute, and filter messages represented
+by XML documents.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.python import log
+import time, types, re
+import XML, RpcServer
+
+
+class HubInterface(RpcServer.Interface):
+ """A simple interface for delivering XML messages to the hub over XML-RPC
+ """
+ def __init__(self, hub):
+ RpcServer.Interface.__init__(self)
+ self.hub = hub
+
+ def xmlrpc_deliver(self, xml):
+ """Deliver an XML message, returning its result on success or a Fault on failure"""
+ return self.hub.deliver(Message(xml))
+
+
+class Message(XML.XMLObject):
+ """Abstract container for a notification message. All messages
+ are represented by XML DOM trees. The message document type
+ is described with examples and a schema in the 'xml' directory
+ of this project.
+
+ For the most part, a message is very free-form and its structure
+ isn't validated. A few exceptions...
+
+ A message must have a <message> tag at its root:
+
+ >>> msg = Message('<message/>')
+ >>> msg = Message('<monkey/>')
+ Traceback (most recent call last):
+ ...
+ XMLValidityError: A Message's root node must be named 'message'
+
+ If a message doesn't include a timestamp, one will be added with
+ the current time:
+
+ >>> msg = Message('<message/>')
+ >>> t = XML.digValue(msg.xml, int, "message", "timestamp")
+ >>> time.time() - t < 2
+ True
+
+ """
+ immutable = True
+
+ def preprocess(self):
+ message = XML.dig(self.xml, "message")
+ if not message:
+ raise XML.XMLValidityError("A Message's root node must be named 'message'")
+
+ # Stamp it with the current time if it has no timestamp yet
+ if not XML.dig(message, "timestamp"):
+ XML.addElement(message, "timestamp", "%d" % time.time())
+
+
+class Hub(object):
+ """A central location where messages are delivered, filtered, and dispatched
+ to interested parties.
+ """
+ def __init__(self):
+ # Maps callables to filters
+ self.clients = {}
+ self.updateClients()
+
+ def addClient(self, callable, filter=None):
+ """Add a callable object to the list of those notified when new messages
+ arrive. If filter is not None, the callable is only called if the filter
+ evaluates to True.
+ If the callable returns non-None, the returned value will be returned
+ by deliver(). If multiple client callables return non-None, the last
+ seen value is used.
+ """
+ self.clients[callable] = filter
+ self.updateClients()
+
+ def delClient(self, callable):
+ """Delate a callable object from the list of those notified when new messages arrive"""
+ del self.clients[callable]
+ self.updateClients()
+
+ def updateClients(self):
+ """Update a cached items() list for our clients dict, to speed up deliver().
+ Must be called whenever clients changes.
+ """
+ self.clientItems = self.clients.items()
+
+ def deliver(self, message):
+ """Given a Message instance, determine who's interested
+ in its contents and delivers it to them.
+ """
+ result = None
+ for callable, filter in self.clientItems:
+ if filter and not filter(message):
+ continue
+ itemResult = callable(message)
+ if itemResult is not None:
+ result = itemResult
+ return result
+
+
+class Filter(XML.XMLFunction):
+ """A filter is a description of some subset of all valid Message objects,
+ described using a simple XML-based format and a subset of XPath. The
+ filter document type is described with examples and a schema in the
+ 'xml' directory of this project.
+
+ To test the filter against a message, call it with the message instance.
+ A boolean value will be returned.
+
+ Some examples, matching against the sample commit message...
+
+ >>> msg = Message(open('xml/samples/simple-message.xml'))
+
+ The <match> tag returns true if the entire text content of any tag
+ matched by the given XPath matches the text in the <match> tag:
+
+ >>> f = Filter('<match path="/message/source/project">navi-misc</match>')
+ >>> f(msg)
+ True
+ >>> Filter('<match path="/message/source/project">jetstream</match>')(msg)
+ False
+
+ It's important to note that the <match> tag can match any of the XPath
+ matches independently. Here, the XPath will return multiple <file> tags,
+ one of which satisfies the <match> tag:
+
+ >>> f = Filter('<match path="/message/body/commit/files/file">' +
+ ... ' trunk/cia/LibCIA/Message.py' +
+ ... '</match>')
+ >>> f(msg)
+ True
+
+ By the same rule, if the XPath never matches, the <match> tag never
+ gets a chance to either:
+
+ >>> Filter('<match path="/path/that/doesnt/exist"/>')(msg)
+ False
+
+ The match by default is case insensitive. The caseSensitive attribute can
+ be set to one to change this:
+
+ >>> f = Filter('<match path="/message/source/project">' +
+ ... ' NAVI-MISC' +
+ ... '</match>')
+ >>> f(msg)
+ True
+ >>> f = Filter('<match path="/message/source/project" caseSensitive="1">' +
+ ... ' NAVI-MISC' +
+ ... '</match>')
+ >>> f(msg)
+ False
+
+ The <find> tag is just like <match> but the given text only has to occur
+ in an XPath match rather than exactly matching it:
+
+ >>> Filter('<find path="/message/source/project">navi</find>')(msg)
+ True
+ >>> Filter('<find path="/message/source/project">' +
+ ... ' NAVI-MISC' +
+ ... '</find>')(msg)
+ True
+ >>> Filter('<find path="/message/source/project">navi-miscski</find>')(msg)
+ False
+ >>> Filter('<find path="/message/source">trunk</find>')(msg)
+ True
+
+ The <find> tag with an empty search string can be used to test for the
+ existence of an XPath match:
+
+ >>> Filter('<find path="/message/body/commit"/>')(msg)
+ True
+ >>> Filter('<find path="/message/body/snail"/>')(msg)
+ False
+
+ For completeness, there are tags that always evaluate to a constant value:
+
+ >>> Filter('<true/>')(msg)
+ True
+ >>> Filter('<false/>')(msg)
+ False
+
+ Tags can be combined using boolean algebra:
+
+ >>> Filter('<and><true/><false/><true/></and>')(msg)
+ False
+ >>> Filter('<and><true/><true/><true/></and>')(msg)
+ True
+ >>> Filter('<or><false/><false/><false/></or>')(msg)
+ False
+ >>> Filter('<or><false/><false/><true/></or>')(msg)
+ True
+
+ The <not> tag, in its simplest use, negates a single argument:
+
+ >>> Filter('<not><true/></not>')(msg)
+ False
+ >>> Filter('<not><false/></not>')(msg)
+ True
+
+ Of course, it would be silly for a tag to only work with one child.
+ It would be intuitive for <not> to also be useful for listing several items,
+ any of which can make the entire expression false. The <not> tag therefore
+ actually implements a logical NOR function:
+
+ >>> Filter('<not><false/><false/><false/></not>')(msg)
+ True
+ >>> Filter('<not><false/><true/><false/></not>')(msg)
+ False
+
+ As if we weren't already having too much fun, several of the Python bitwise
+ operators can be used like logical operators to combine Filter instances
+ after they're parsed but before their value has been determined:
+
+ >>> f = Filter('<false/>') | Filter('<false/>')
+ >>> f(msg)
+ False
+ >>> f = Filter('<true/>') | Filter('<false/>')
+ >>> f(msg)
+ True
+
+ >>> f = Filter('<false/>') & Filter('<true/>')
+ >>> f(msg)
+ False
+ >>> f = Filter('<true/>') & Filter('<true/>')
+ >>> f(msg)
+ True
+
+ >>> f = ~Filter('<true/>')
+ >>> f(msg)
+ False
+ >>> f = ~Filter('<false/>')
+ >>> f(msg)
+ True
+
+ """
+
+ def pathMatchTag(self, element, function, textExtractor=XML.shallowText):
+ """Implements the logic common to all tags that test the text matched by
+ an XPath against the text inside our element. The given function is used
+ to determine if the text matches. This implements the properties common to
+ several elements:
+
+ - The caseSensitive attribute defaults to 1, but can be set to zero
+ to force both strings to lowercase.
+
+ - Each XPath match is tested separately, with 'or' semantics:
+ if any of the XPath matches cause the provided function to match,
+ this returns True
+
+ - If there are no XPath matches, returns False
+ """
+ path = element.getAttributeNS(None, 'path')
+ xp = XML.XPath(XML.pathShortcuts.get(path, path))
+
+ # Are we doing a case sensitive match? Default is no.
+ caseSensitive = element.getAttributeNS(None, 'caseSensitive')
+ if caseSensitive:
+ caseSensitive = int(caseSensitive)
+ else:
+ caseSensitive = 0
+
+ text = XML.shallowText(element).strip()
+ if not caseSensitive:
+ text = text.lower()
+
+ def filterMatch(msg):
+ # Use queryobject then str() so that matched
+ # nodes without any text still give us at least
+ # the empty string. This is important so that <find>
+ # with an empty search string can be used to test
+ # for the existence of an XPath match.
+ nodes = xp.queryObject(msg)
+ if nodes:
+ matchStrings = map(textExtractor, nodes)
+
+ # Any of the XPath matches can make our match true
+ for matchString in matchStrings:
+ matchString = matchString.strip()
+ if not caseSensitive:
+ matchString = matchString.lower()
+ if function(matchString, text):
+ return True
+ return False
+ return filterMatch
+
+ def element_match(self, element):
+ """Evaluates to True if the text matched by our 'path' attribute matches
+ this element's content, not including leading and trailing whitespace.
+ """
+ return self.pathMatchTag(element, lambda matchString, text: matchString == text)
+
+ def element_find(self, element):
+ """Evaluates to True if the text in this tag is contained within any of the
+ XPath match strings.
+ """
+ return self.pathMatchTag(element, lambda matchString, text: matchString.find(text) >= 0,
+ textExtractor = XML.allText)
+
+ def element_and(self, element):
+ """Evaluates to True if and only if all child functions evaluate to True"""
+ childFunctions = list(self.childParser(element))
+ def filterAnd(msg):
+ for child in childFunctions:
+ if child and not child(msg):
+ return False
+ return True
+ return filterAnd
+
+ def element_or(self, element):
+ """Evaluates to True if and only if any child function evaluates to True"""
+ childFunctions = list(self.childParser(element))
+ def filterOr(msg):
+ for child in childFunctions:
+ if child and child(msg):
+ return True
+ return False
+ return filterOr
+
+ def element_not(self, element):
+ """The NOR function, returns false if and only if any child function evaluates to True.
+ For the reasoning behind calling this 'not', see the doc string for this class.
+ """
+ childFunctions = list(self.childParser(element))
+ def filterNot(msg):
+ for child in childFunctions:
+ if child and child(msg):
+ return False
+ return True
+ return filterNot
+
+ def element_true(self, element):
+ """Always evaluates to True"""
+ def alwaysTrue(msg):
+ return True
+ return alwaysTrue
+
+ def element_false(self, element):
+ """Always evaluates to False"""
+ def alwaysFalse(msg):
+ return False
+ return alwaysFalse
+
+ def __and__(self, other):
+ """Perform a logical 'and' on two Filters without evaluating them"""
+ newFilter = Filter()
+ newFilter.f = lambda msg: self(msg) and other(msg)
+ return newFilter
+
+ def __or__(self, other):
+ """Perform a logical 'or' on two Filters without evaluating them"""
+ newFilter = Filter()
+ newFilter.f = lambda msg: self(msg) or other(msg)
+ return newFilter
+
+ def __invert__(self):
+ """Perform a logical 'not' on this Filter without evaluating it"""
+ newFilter = Filter()
+ newFilter.f = lambda msg: not self(msg)
+ return newFilter
+
+
+class FormatterArgs:
+ """Contains all arguments passed between formatters. This includes
+ the input string, preferences dictionary, and the message being processed.
+ """
+ def __init__(self, message, input=None, preferences={}):
+ self.message = message
+ self.input = input
+ self.preferences = dict(preferences)
+
+ def copy(self, **kwargs):
+ """Make a copy of the arguments, optionally overriding attributes at the same time"""
+ new = FormatterArgs(self.message, self.input, self.preferences)
+ new.__dict__.update(kwargs)
+ return new
+
+ def getPreference(self, name, default):
+ """If a preference with the given name has been set, converts it to the same type
+ as the supplied default and returns it. Otherwise, returns the default.
+ """
+ if name in self.preferences:
+ return type(default)(self.preferences[name])
+ else:
+ return default
+
+
+class Formatter:
+ """An abstract object capable of creating and/or modifying alternate
+ representations of a Message. This could include converting it to HTML,
+ converting it to plaintext, or annotating the result of another Formatter
+ with additional information.
+ """
+ # If non-none, this is a Filter string that can be tested against
+ # a message to detect whether this formatter is applicable.
+ filter = None
+
+ # A string identifying this formatter's output medium. Could be 'html',
+ # 'irc', etc.
+ medium = None
+
+ def formatMessage(self, message):
+ """A convenience function to format a message without any prior input
+ or other arguments.
+ """
+ return self.format(FormatterArgs(message))
+
+ def format(self, args):
+ """Given a FormatterArgs instance, return a formatted representation of the message."""
+ pass
+
+ def loadParametersFrom(self, xml, unused=None):
+ """This is given a <formatter> element possibly containing
+ extra parameters for the formatter to process and store.
+ Any problems should be signalled with an XML.XMLValidityError.
+
+ By default, this tries to find a param_* handler for each
+ element it comes across.
+
+ Returns a set object, containing the names of all unused
+ parameters. This allows callers, during validation, to look
+ for misspelled or otherwise unused elements.
+ """
+ unused = set()
+ for tag in XML.getChildElements(xml):
+ f = getattr(self, 'param_'+tag.nodeName, None)
+ if f:
+ f(tag)
+ else:
+ unused.add(tag.nodeName)
+ return unused
+
+
+class MarkAsHidden(str):
+ """This object acts like an empty string, but has the side effect
+ of hiding a containing <autoHide> element. This should be returned
+ by components when their result is nonexistent or not applicable,
+ so that the colors, prefixes, or suffixes around that component
+ are also hidden.
+ """
+ def __new__(self):
+ return str.__new__(self, "")
+
+
+class ModularFormatter(Formatter):
+ """A Formatter consisting of multiple components that can be rearranged
+ by changing the 'format' parameter. The current component arrangement
+ is stored as a DOM tree in 'componentTree', generated from the XML
+ string in 'defaultComponentTree'. It is overridden in param_format().
+ """
+ defaultComponentTree = None
+ componentTree = None
+
+ # Cache a parsed defaultComponentTree
+ _cachedDefaultTree = None
+ _defaultTreeOwner = None
+
+ def param_format(self, tree):
+ """Handles the <format> parameter. Note that since the format generally
+ can't be shared between different types of formatters, there is an
+ optional (but recommended) 'appliesTo' attribute that lists one or more
+ space separated formatter names.
+ """
+ appliesTo = tree.getAttributeNS(None, 'appliesTo')
+ if appliesTo:
+ if self.__class__.__name__ not in appliesTo.split():
+ return
+
+ self.componentTree = tree
+
+ # Validate the component tree by parsing a null message once.
+ # Any errors in the components or their arguments should
+ # trigger XMLValidityErrors at this point.
+ self.formatMessage(Message("<message/>"))
+
+ def format(self, args):
+ """The formatter entry point. This just finds the current component
+ tree and invokes walkComponents and joinComponents on it.
+ """
+ # Parse the default component tree, caching it per-class
+ if self.__class__._defaultTreeOwner is not self.__class__.defaultComponentTree:
+ self.__class__._defaultTreeOwner = self.__class__.defaultComponentTree
+ self.__class__._cachedDefaultTree = XML.parseString(self.__class__.defaultComponentTree).documentElement
+
+ # This will use the default component tree if it hasn't been overridden in this instance
+ tree = self.componentTree or self.__class__._cachedDefaultTree
+ return self.joinComponents(self.walkComponents(tree.childNodes, args))
+
+ def evalComponent(self, node, args):
+ """Given a DOM node for a component, evaluate it and return the result
+ list. An empty list always indicates that the component ran but
+ had nothing to format- we return None if the node is ignored.
+
+ Text nodes return a list with only their data. Newline characters
+ in a text node are converted to spaces, so newlines and tabs can be used
+ to prettify the XML without it affecting the final output- this combined with
+ the redundant whitespace elimination in joinComponents() gives HTML-like
+ semantics. Literal linebreaks can be obtained with the <br/> component.
+
+ Elements invoke the corresponding component_* handler, which must
+ return a sequence.
+ """
+ if node.nodeType == node.TEXT_NODE:
+ return [node.data.replace("\n", " ")]
+
+ elif node.nodeType == node.ELEMENT_NODE:
+ f = getattr(self, "component_" + node.nodeName, None)
+ if f:
+ return f(node, args)
+ else:
+ raise XML.XMLValidityError("Unknown component name in %s: %r" %
+ (self.__class__.__name__, node.nodeName))
+
+ def walkComponents(self, nodes, args):
+ """Walk through all the given XML nodes, returning a list of formatted objects."""
+ results = []
+ for node in nodes:
+ results.extend(self.evalComponent(node, args))
+ return results
+
+ def component_autoHide(self, element, args):
+ """A built-in component that evaluates all children, hiding them all if
+ any return an empty list. This makes it easy to add prefixes, suffixes, or
+ other formatting to a component that disappears when it does.
+ """
+ results = self.walkComponents(element.childNodes, args)
+ for result in results:
+ if isinstance(result, MarkAsHidden):
+ # Hidden markers don't propagage, return a normal empty list
+ return []
+ return results
+
+ def component_br(self, element, args):
+ """A built-in component that just returns a newline"""
+ return ["\n"]
+
+ def textComponent(self, element, args, *path):
+ """A convenience function for defining components that just look for a node
+ in the message and return its shallowText.
+ """
+ element = XML.dig(args.message.xml, *path)
+ if element:
+ return [XML.shallowText(element)]
+ else:
+ return [MarkAsHidden()]
+
+ def component_text(self, element, args):
+ """This is a generic version of textComponent, in which 'path' can
+ be specified by users. Any textComponent can be rewritten as a
+ <text> component.
+ """
+ path = element.getAttributeNS(None, 'path')
+ if not path:
+ raise XML.XMLValidityError("The 'path' attribute on <text> is required.")
+ xp = XML.XPath(XML.pathShortcuts.get(path, path))
+
+ nodes = xp.queryObject(args.message)
+ if nodes:
+ return [XML.shallowText(nodes[0])]
+ else:
+ return [MarkAsHidden()]
+
+ def joinComponents(self, results):
+ """Given a list of component results, return the formatter's final result.
+ The default implementation converts to strings, joins, then removes excess
+ whitespace. Subclasses should override this if they're formatting Nouvelle
+ trees or other non-string data types.
+ """
+ return re.sub(r'[ \t]+', ' ', ''.join(map(unicode, results))).strip()
+
+
+filterCache = {}
+
+def getCachedFilter(xml):
+ """Get the Filter instance corresponding to the given XML
+ fragment, creating one if it doesn't exist yet. As the cache
+ doesn't get cleaned yet, this should only be used for
+ non-user-provided Filter strings.
+ """
+ global filterCache
+ try:
+ return filterCache[xml]
+ except KeyError:
+ f = Filter(xml)
+ filterCache[xml] = f
+ return f
+
+
+class NoFormatterError(Exception):
+ pass
+
+
+class FormatterFactory:
+ """An object that keeps track of a collection of Formatter objects
+ and can create formatters to match particular requirements.
+
+ This class should be constructed with a dictionary to pull Formatter
+ instances out of and catalog.
+ """
+ def __init__(self, *args):
+ self.nameMap = {}
+ self.mediumMap = {}
+ for arg in args:
+ self.install(arg)
+
+ def install(self, obj):
+ """Install a module, Formatter instance, or dict containing Formatters
+ such that they are searchable by this FormatterFactory.
+ """
+ if type(obj) == type(Formatter) and issubclass(obj, Formatter):
+ self.nameMap[obj.__name__] = obj
+ if obj.filter and obj.medium:
+ self.mediumMap.setdefault(obj.medium, []).append(obj)
+
+ elif type(obj) == dict:
+ for subobj in obj.itervalues():
+ # We only look at Formtter instances inside dicts, to
+ # avoid recursively drilling down into modules.
+ if type(subobj) == type(Formatter) and issubclass(subobj, Formatter):
+ self.install(subobj)
+
+ elif type(obj) == types.ModuleType:
+ self.install(obj.__dict__)
+
+ def findName(self, name):
+ """Find a particular formatter by name"""
+ try:
+ cls = self.nameMap[name]
+ except KeyError:
+ raise NoFormatterError('No such formatter "%s"' % name)
+ return cls()
+
+ def _getFormattersForMedium(self, medium):
+ """Return a non-empty list with all formatters for a particular medium."""
+ l = self.mediumMap.get(medium)
+ if l:
+ return l
+ else:
+ raise NoFormatterError('No formatters for the "%s" medium' % medium)
+
+ def findMedium(self, medium, message):
+ """Find a formatter for the given medium and matching the given message."""
+ for cls in self._getFormattersForMedium(medium):
+ if getCachedFilter(cls.filter)(message):
+ return cls()
+ raise NoFormatterError('No matching formatters for the "%s" medium' % medium)
+
+ def _reportUnusedParameters(self, unused):
+ if unused:
+ l = [ '"%s"' % el for el in unused ]
+ l.sort()
+ raise XML.XMLValidityError(
+ "Some formatter parameters were not recognized: " + ", ".join(l))
+
+ def fromXml(self, xml, message=None):
+ """Create a formatter to match the given <formatter> element.
+ The formatter element may have a 'name' attribute to specify a particular
+ formatter class or a 'medium' attribute to automatically find a matching
+ formatter to output to a given medium.
+
+ If 'message' is None and a medium is requested rather than a particular
+ formatter, this will return None after validating the medium.
+ """
+ attrNames = [attr.name for attr in xml.attributes.values()]
+
+ # Load a single formatter, by name
+ if attrNames == ['name']:
+ f = self.findName(xml.getAttributeNS(None, 'name'))
+ self._reportUnusedParameters(f.loadParametersFrom(xml))
+ return f
+
+ # Search for a matching formatter, by medium. If we have no
+ # message to match, this will validate the parameters against
+ # all formatters for the medium.
+ if attrNames == ['medium']:
+ medium = xml.getAttributeNS(None, 'medium')
+ if message:
+ f = self.findMedium(medium, message)
+ f.loadParametersFrom(xml)
+ return f
+ else:
+ # Take the intersection of all formatters' unused parameters
+ unused = None
+ for cls in self._getFormattersForMedium(medium):
+ next = cls().loadParametersFrom(xml)
+ if unused is None:
+ unused = next
+ else:
+ unused.intersection_update(next)
+ self._reportUnusedParameters(unused)
+ return
+
+ raise XML.XMLValidityError("<formatter> must have either a 'name' attribute or a 'medium' attribute")
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Message.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Message.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/RpcClient.py
===================================================================
--- trunk/community/infrastructure/LibCIA/RpcClient.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/RpcClient.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,129 @@
+""" LibCIA.RpcClient
+
+URI handlers that deliver messages over various RPC methods to other servers
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.web import xmlrpc
+import Ruleset
+import re
+
+
+class XmlrpcURIHandler(Ruleset.RegexURIHandler):
+ """Handles xmlrpc:// URIs, specifying an XML-RPC call to make,
+ with an argument list optionally including literals, the
+ message content, and the ruleset's resulting content.
+ If the specified server name includes no path, '/RPC2' will be assumed.
+ Some examples...
+
+ To deliver a message to another CIA server running on some.host.net:
+
+ xmlrpc://some.host.net/RPC2?hub.deliver(message)
+
+ To call the function 'recordData' with 'muffin' and 42 as the first
+ two arguments, and the formatter's result as the second argument, on
+ an XML-RPC server located at http://example.com:8080/services/rpc:
+
+ xmlrpc://example.com:8080/services/rpc?recordData('muffin',42,content)
+
+ Only string literals, numeric literals, and variables are currently supported.
+
+ Note that all connections are performed over http- this doesn't
+ support making XML-RPC connections over https, and there would
+ really be no point.
+ """
+ scheme = 'xmlrpc'
+
+ regex = r"""
+ ^xmlrpc:// # URI scheme
+ (?P<server>[a-zA-Z]([a-zA-Z0-9.-]*[a-zA-Z0-9])? # Hostname
+ (:[0-9]+)? # Port
+ (/[^ /\?]+)*) # Path
+ \?(?P<function>[^ \(\)]+) # Function name
+ \((?P<args>.*)\)$ # Args
+ """
+
+ argLexer = re.compile(r"""
+ # Whitespace before each parameter
+ \s*
+ (
+ # Literal types
+ (?P<float> \-?((\d+\.\d*|\.\d+)([eE][-+]?\d+)?|\d+[eE][-+]?\d+))|
+ (?P<int> \-?[1-9]\d*)|
+ (?P<string> ( \"([^\\\"]|\\.)*\" | \'([^\\\']|\\.)*\' ))|
+
+ # Variables
+ (?P<var> [a-zA-Z0-9_]+)
+ )
+ # Whitespace and an optional comma after each parameter
+ \s*,?\s*
+ """, re.VERBOSE)
+
+ def parseArgs(self, args, **vars):
+ """Parse an argument list, as a string, with the given variables.
+ Returns a list of parsed arguments, or raises an InvalidURIException.
+ """
+ argList = []
+ for token in self.argLexer.finditer(args):
+ for tokenType, tokenValue in token.groupdict().items():
+ if tokenValue is not None:
+ argList.append(getattr(self, 'argtoken_'+tokenType)(tokenValue, vars))
+ return argList
+
+ def argtoken_float(self, value, vars):
+ return float(value)
+
+ def argtoken_var(self, value, vars):
+ try:
+ return vars[value]
+ except KeyError:
+ raise Ruleset.InvalidURIException("Unknown variable name %r" % value)
+
+ def argtoken_int(self, value, vars):
+ return int(value)
+
+ def argtoken_string(self, value, vars):
+ # CHEESY HACK!
+ return value[1:-1]
+
+ def assigned(self, uri, newRuleset):
+ """Just validate the URI and argument list"""
+ self.parseArgs(self.parseURI(uri)['args'], content=None, message=None)
+
+ def message(self, uri, message, content):
+ groups = self.parseURI(uri)
+
+ # Add the implicit /RPC2 and http://, creating a server proxy object
+ server = groups['server']
+ if server.find('/') < 0:
+ server = server + '/RPC2'
+ server = 'http://' + server
+ proxy = xmlrpc.Proxy(server)
+
+ # Parse arguments, allowing use of our 'message' and 'content' variables.
+ args = self.parseArgs(groups['args'],
+ message = str(message),
+ content = content)
+
+ # Make the call, ignoring the resulting Deferred. If it succeeds,
+ # we don't really care. If it fails, the error will get logged.
+ proxy.callRemote(groups['function'], *args)
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/RpcServer.py
===================================================================
--- trunk/community/infrastructure/LibCIA/RpcServer.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/RpcServer.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,209 @@
+""" LibCIA.RpcServer
+
+Base classes implementing CIA's XML-RPC interface. This extend's twisted's
+usual XML-RPC support with our own security and exception handling code.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.web import xmlrpc, server
+from twisted.internet import defer
+from twisted.python import log
+import xmlrpclib, time, cPickle
+
+
+class Interface(xmlrpc.XMLRPC):
+ """A web resource representing a set of XML-RPC functions, optionally
+ with other Interfaces attached as subhandlers.
+
+ This is based on Twisted's XMLRPC server class, but overrides most
+ of its functions :)
+
+ Improvements in this, as compared to twisted's default XMLRPC class:
+
+ - This makes the request object available to functions if they
+ want it, making it possible for them to read HTTP headers.
+
+ - This implements multiple function prefixes, each defining a
+ different interface that function operates under.
+
+ - A 'protected' function prefix requires a valid capability key
+ to use the function, helping to enforce our security policies.
+
+ - Less buggy (compared to Twisted 1.1.1) handling of
+ nested subhandlers
+ """
+ def render(self, request):
+ """This is a modified version of render() from XMLRPC that will
+ pass keyword arguments with extra state information if it can.
+ Currently this just consists of the current request.
+ """
+ request.content.seek(0, 0)
+ args, functionPath = xmlrpclib.loads(request.content.read())
+ try:
+ function = self._getFunction(functionPath)
+ except xmlrpc.NoSuchFunction:
+ self._cbRender(
+ xmlrpc.Fault(self.NOT_FOUND, "no such function %s" % functionPath),
+ request
+ )
+ else:
+ request.setHeader("content-type", "text/xml")
+
+ # Pass as many keyword args to the function as we can.
+ # Note that this currently doesn't work on protected functions.
+ kwargs = {}
+ try:
+ if 'request' in function.func_code.co_varnames:
+ kwargs['request'] = request
+ except AttributeError:
+ pass
+
+ defer.maybeDeferred(function, *args, **kwargs).addErrback(
+ self._ebRender
+ ).addCallback(
+ self._cbRender, request
+ )
+ return server.NOT_DONE_YET
+
+ def call(self, fqname, *args):
+ """A convenience function for calling methods in the RPC tree internally.
+ Returns a Deferred.
+ """
+ return defer.maybeDeferred(self._getFunction(fqname), *args)
+
+ def _getFunction(self, fqname):
+ """Functions named xmlrpc_* are found and returned as-is, without any capability testing.
+
+ Functions named protected_* are called only if the first argument is verified to be
+ a valid capability key. By default, the list of capabilities accepted for a function is
+ generated as 'universe', the fully qualified XML-RPC name of this function, and the
+ fully qualified name of each interface containing this function.
+
+ This behaviour can be overriden per-function by defining caps_* for this function
+ as either a callable (called with the same arguments as the function) or a static
+ sequence of capabilities.
+ """
+ # The default _getFunction expands subhandlers recursively. We don't do that, since
+ # our capability generation code needs to know the fully qualified names involved.
+ if fqname.find(self.separator) == -1:
+ path = (fqname,)
+ else:
+ path = fqname.split(self.separator)
+
+ # Find the Interface instance that owns this function
+ interface = self
+ for subHandler in path[:-1]:
+ interface = interface.getSubHandler(subHandler)
+ if interface is None:
+ raise xmlrpc.NoSuchFunction
+
+ # Do we have a normal non-protected function?
+ f = getattr(interface, "xmlrpc_%s" % path[-1], None)
+ if f and callable(f):
+ return f
+
+ # Do we have a protected function?
+ f = getattr(interface, "protected_%s" % path[-1], None)
+ if f and callable(f):
+ return self.protect(interface, path, f)
+ raise xmlrpc.NoSuchFunction
+
+ def protect(self, interface, path, f):
+ """Given a protected RPC function, return a wrapper that
+ checks capabilities before executing the original function.
+ """
+ def rpcWrapper(*args):
+ import Security
+
+ result = defer.Deferred()
+
+ # First argument is the key
+ try:
+ key = args[0]
+ args = args[1:]
+ except IndexError:
+ raise TypeError("This is a protected function, the first argument must be a capability key")
+
+ # Look up the capabilities list
+ caps = getattr(interface, "caps_%s" % path[-1], None)
+ if not caps:
+ caps = self.makeDefaultCaps(path)
+ if callable(caps):
+ caps = caps(path, *args)
+
+ # A callback invoked when our key is successfully validated
+ def keyValidated(unimportant):
+ d = defer.maybeDeferred(f, *args)
+ d.addBoth(Security.logProtectedCall, path, args, user)
+ d.chainDeferred(result)
+
+ def keyError(failure):
+ Security.logProtectedCall(failure, path, args, user, allowed=False)
+ result.errback(failure)
+
+ # Check the capabilities asynchronously (requires a database query)
+ user = Security.User(key=str(key))
+ d = user.require(*caps)
+ d.addCallback(keyValidated)
+ d.addErrback(keyError)
+ return result
+ return rpcWrapper
+
+ def _cbRender(self, result, request):
+ """Wrap the default _cbRender, converting None (which can't be serialized) into True"""
+ if result is None:
+ result = True
+ xmlrpc.XMLRPC._cbRender(self, result, request)
+
+ def _ebRender(self, failure):
+ """Override the default errback for rendering such that non-Faults
+ are wrapped usefully rather than being converted into an unhelpful 'error'.
+ """
+ if isinstance(failure.value, xmlrpc.Fault):
+ return failure.value
+ log.err(failure)
+ return xmlrpc.Fault(str(failure.type), str(failure.value))
+
+ def makeDefaultCaps(self, path):
+ """Create a default list of acceptable capability IDs for the given path"""
+ caps = ['universe']
+ base = []
+ for segment in path:
+ base.append(str(segment))
+ caps.append(".".join(base))
+ return caps
+
+
+# rebuild()-friendly mojo.
+# Normally this will just create _rootInterface, but if we're rebuild()'ing this
+# module, overwriting it would be a Bad Thing(tm), causing a new root interface
+# with no children to replace it.
+if '_rootInterface' not in globals():
+ global _rootInterface
+ _rootInterface = None
+
+def getRootInterface():
+ # Interface should be a singleton, this retrieves the instance, creating it if necessary
+ global _rootInterface
+ if not _rootInterface:
+ _rootInterface = Interface()
+ return _rootInterface
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/RpcServer.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/RpcServer.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Ruleset.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Ruleset.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Ruleset.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,654 @@
+""" LibCIA.Ruleset
+
+Rulesets tie together all the major components of CIA. They hold
+filters and formatters that define which messages they apply
+to and how to display those messages, and specify a URI that
+gives the finished message a destination.
+
+A Ruleset by itself is just a function that, given a message,
+returns a formatted version of that message or None. To be of
+any use for actually processing messages, a URIHandler for the
+message's URI is looked up, and a RulesetDelivery object is used
+to join the Ruleset and URIHandler. The RulesetDelivery is registered
+as a client of the Message.Hub. Upon receiving a message, it runs
+it through the ruleset and if a result comes out, sends that result
+to the URIHandler.
+
+RulesetStorage holds a persistent list of rulesets, with URIHandlers
+assigned to each via URIRegistry.
+
+RulesetController attaches to a Message.Hub and listens for messages
+used to store and query rulesets in a RulesetStorage.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import XML, Message, Debug, Security, RpcServer, Formatters
+from twisted.python import log
+from twisted.internet import defer, reactor
+import sys, traceback, re, os
+
+
+class InvalidURIException(Exception):
+ """An exception that URI handlers can raise when a URI is invalid"""
+ pass
+
+
+class RulesetInterface(RpcServer.Interface):
+ """An XML-RPC interface used to set and query the rulesets in a RulesetStorage"""
+ def __init__(self, storage):
+ self.storage = storage
+ RpcServer.Interface.__init__(self)
+
+ def protected_store(self, xml):
+ """Stores a ruleset provided as XML text. Deleting a ruleset is equivalent
+ to storing an empty one with the same URI.
+ """
+ self.storage.store(xml)
+
+ def caps_store(self, path, xml):
+ """Generate a list of acceptable capabilities to grant access to 'store'.
+ In addition to the usual ones, allow ('ruleset.uri', x) where x is the
+ ruleset's URI.
+ """
+ uri = XML.parseString(xml).documentElement.getAttributeNS(None, 'uri')
+ return self.makeDefaultCaps(path) + [('ruleset.uri', uri)]
+
+ def protected_grantUri(self, uri, uid):
+ """Returns a key for the capability ('ruleset.uri', uri).
+ This can be used to delegate control of a particular URI's ruleset-
+ an administrator would call this function to retrieve a key for a particular
+ URI, then hand that to someone else who would only have the ability
+ to edit that URI.
+ """
+ return Security.User(int(uid)).grant(('ruleset.uri', str(uri)))
+
+ def xmlrpc_getUriList(self):
+ """Return a list of all URIs with non-empty rulesets"""
+ return self.storage.rulesetMap.keys()
+
+ def xmlrpc_getRuleset(self, uri):
+ """Return the ruleset associated with the given URI, or False if there is no ruleset"""
+ try:
+ return str(self.storage.rulesetMap[uri].ruleset)
+ except KeyError:
+ return False
+
+ def xmlrpc_getRulesetMap(self):
+ """Returns all rulesets in the form of a mapping from URI to ruleset text"""
+ results = {}
+ for delivery in self.storage.rulesetMap.itervalues():
+ results[delivery.ruleset.uri] = str(delivery.ruleset)
+ return results
+
+
+class RulesetReturnException(Exception):
+ """Used to implement <return> in a Ruleset.
+ This exception is caught in the <ruleset> root node,
+ causing it to return with the current result value right away.
+ """
+ pass
+
+
+class Ruleset(XML.XMLFunction):
+ r"""A ruleset is a tree that makes decisions about an incoming message
+ using filters and generates output using formatters.
+ A ruleset may contain the following elements, which are evaluated sequentially.
+ The output from the last formatter is returned upon calling this ruleset, and
+ each formatter is given the previous formatter's output as input, so they may be stacked.
+
+ <formatter *> : Chooses and applies a formatter. All attributes and
+ contents are passed on to the FormatterFactory, refer
+ to its documentation for more information.
+
+ any Filter tag : Evaluates the filter, terminating the current rule
+ if it returns false.
+
+ <rule> : Marks a section of the ruleset that can be exited
+ when a filter returns false. A <rule> with a filter
+ as the first child can be used to create conditionals.
+
+ <return [path='/foo/bar']> : Normally the result of the last formatter is returned.
+ This returns from the ruleset immediately. If a path
+ is supplied, the string value of that XPath match is
+ returned. If not, the content of the <return> tag is
+ returned. An empty tag or no XPath match will cause
+ the ruleset to return None.
+
+ <break> : Return immediately from the ruleset, but don't change
+ the current return value
+
+ <ruleset [uri="foo://bar"]> : Like <rule>, but this is always the root tag.
+ It may include a 'uri' attribute specifying the
+ destination of a message passed through this ruleset.
+ This ruleset object will store a URI if one is present,
+ but does not require one. The typical application of
+ rulesets however does require a URI to be specified.
+
+ >>> msg = Message.Message(
+ ... '<message>' +
+ ... '<source>' +
+ ... '<project>robo-hamster</project>' +
+ ... '</source>' +
+ ... '<body>' +
+ ... '<colorText>' +
+ ... '<b>Hello</b>World' +
+ ... '</colorText>' +
+ ... '</body>' +
+ ... '</message>')
+
+ >>> r = Ruleset('<ruleset><formatter medium="irc"/></ruleset>')
+ >>> r(msg)
+ '\x02Hello\x0fWorld'
+
+ >>> r = Ruleset('<ruleset><return>Boing</return><formatter medium="irc"/></ruleset>')
+ >>> r(msg)
+ u'Boing'
+
+ >>> r = Ruleset('<ruleset>' +
+ ... '<formatter medium="irc"/>' +
+ ... '<rule>' +
+ ... '<match path="/message/source/project">robo-hamster</match>' +
+ ... '<return>*censored*</return>' +
+ ... '</rule>' +
+ ... '</ruleset>')
+ >>> r(msg)
+ u'*censored*'
+
+ >>> r = Ruleset('<ruleset>' +
+ ... '<formatter medium="irc"/>' +
+ ... '<rule>' +
+ ... '<match path="/message/source/project">duck-invader</match>' +
+ ... '<return>Quack</return>' +
+ ... '</rule>' +
+ ... '<formatter name="IRCProjectName"/>' +
+ ... '</ruleset>')
+ >>> r(msg)
+ u'\x02robo-hamster:\x0f \x02Hello\x0fWorld'
+
+ >>> r = Ruleset('<ruleset><return/></ruleset>')
+ >>> r(msg) is None
+ True
+
+ >>> r = Ruleset('<ruleset><return path="/message/source/project"/></ruleset>')
+ >>> r(msg)
+ u'robo-hamster'
+
+ >>> Ruleset('<ruleset/>').uri is None
+ True
+ >>> Ruleset('<ruleset uri="sponge://"/>').uri
+ 'sponge://'
+ """
+ requiredRootElement = "ruleset"
+
+ def element_ruleset(self, element):
+ """<ruleset> for the most part works just like <rule>, but since
+ it's the root node it's responsible for initializing and returning
+ the ruleset's result.
+ """
+ if element is not self.xml.documentElement:
+ raise XML.XMLValidityError("The <ruleset> element must only occur at the document " +
+ "root. Use <rule> to create nested rules.")
+
+ # Go ahead and store the URI attribute if we have one.
+ # If not, this will be None.
+ self.uri = element.getAttributeNS(None, 'uri') or None
+
+ # URIs are always encoded if necessary, since just about everywhere we'd need to
+ # use a URI we can't support Unicode yet. Specific examples are IRC servers/channels
+ # and as dict keys in an XML-RPC response.
+ if type(self.uri) is unicode:
+ self.uri = self.uri.encode()
+
+ # Create a function to evaluate this element as a <rule> would be evaluated
+ ruleFunc = self.element_rule(element)
+
+ # Now wrap this function in one that performs our initialization and such
+ def rulesetRoot(msg):
+ self.result = None
+ try:
+ ruleFunc(msg)
+ except RulesetReturnException:
+ pass
+ result = self.result
+ del self.result
+ return result
+ return rulesetRoot
+
+ def element_rule(self, element):
+ """Evaluate each child element in sequence until one returns False"""
+ childFunctions = list(self.childParser(element))
+ def rulesetRule(msg):
+ for child in childFunctions:
+ if child:
+ if not child(msg):
+ break
+ return True
+ return rulesetRule
+
+ def element_return(self, element):
+ """Set the current result and exit the ruleset immediately"""
+ if element.hasAttributeNS(None, 'path'):
+ path = element.getAttributeNS(None, 'path')
+ xp = XML.XPath(XML.pathShortcuts.get(path, path))
+ # Define a rulesetReturn function that returns the value of the XPath
+ def rulesetReturn(msg):
+ nodes = xp.queryObject(msg)
+ if nodes:
+ self.result = XML.allText(nodes[0]).strip()
+ else:
+ self.result = None
+ raise RulesetReturnException()
+ return rulesetReturn
+
+ else:
+ # No path, define a rulesetReturn function that returns this element's string value
+ def rulesetReturn(msg):
+ self.result = XML.shallowText(element)
+ if not self.result:
+ self.result = None
+ raise RulesetReturnException()
+ return rulesetReturn
+
+ def element_break(self, element):
+ """Just exit the ruleset immediately"""
+ def rulesetBreak(msg):
+ raise RulesetReturnException()
+ return rulesetBreak
+
+ def element_formatter(self, element):
+ """Creates a Formatter instance matching the element's description,
+ returns a function that applies the formatter against the current
+ message and result.
+ """
+ # Evaluate this once at parse-time so any silly errors
+ # like unknown formatters or media can be detected.
+ Formatters.getFactory().fromXml(element)
+
+ def rulesetFormatter(msg):
+ args = Message.FormatterArgs(msg, self.result)
+ self.result = Formatters.getFactory().fromXml(element, msg).format(args)
+ return True
+ return rulesetFormatter
+
+ def unknownElement(self, element):
+ """Check whether this element is a filter before giving up"""
+ try:
+ f = Message.Filter(element)
+ except XML.UnknownElementError:
+ # Nope, not a filter.. let XMLFunction give an error
+ XML.XMLFunction.unknownElement(self, element)
+
+ # We can just return the filter, since it has the same calling
+ # signature as any of our other element implementation functions.
+ return f
+
+ def isEmpty(self):
+ """Returns True if the ruleset has no contents"""
+ return not XML.hasChildElements(self.xml.documentElement)
+
+
+class BaseURIHandler(object):
+ """A URIHandler instance defines a particular URI scheme, actions to
+ be taken when a URI matching that scheme is assigned or unassigned
+ a ruleset, and a way to deliver messages to a matching URI.
+ This is an abstract base class, it must be subclassed
+ to implement a particular URI scheme.
+ """
+ # Subclasses must either set this to the name of the URI scheme
+ # (such as 'irc' or 'http') or override the 'detect' function.
+ scheme = None
+
+ def detect(self, uri):
+ """Return True if this URI handler is applicable to the given URI.
+ The default implementation checks it against our URI scheme.
+ """
+ return uri.startswith(self.scheme + ':')
+
+ def assigned(self, uri, newRuleset):
+ """This optional function is called when a URI matching this handler is
+ assigned a new ruleset. This includes being assigned a ruleset after
+ previously not having one. Generally this is used whenever a connection
+ must be initialized to the object referred to by the URI.
+ """
+ pass
+
+ def unassigned(self, uri):
+ """This optional function is called when a URI matching this handler
+ has its ruleset removed. Generally this is used to disconnect any
+ connection formed by assigned().
+ """
+ pass
+
+ def message(self, uri, message, content):
+ """Deliver a message to the given URI. The 'message' is the original
+ unprocessed message resulting in this content, and the 'content'
+ is the result of passing the message through whatever ruleset
+ caused it to end up here.
+ """
+ pass
+
+ def rulesetsRefreshed(self):
+ """If the handler is interested, it can override this method to be
+ notified when rulesets are done refreshing.
+ """
+ pass
+
+
+class RegexURIHandler(BaseURIHandler):
+ """A URIHandler that validates and parses URIs using a regular expression.
+ This class provides a parseURI method that generates a dictionary of
+ groups matched in the regex, or raises an UnsupportedURI exception if
+ the regex does not match. It implements a default assigned() function
+ that runs parseURI just to cause an error in setting the URI if it's invalid.
+ """
+ regex = None
+ regexFlags = re.VERBOSE
+
+ def parseURI(self, uri):
+ """Given a valid URI, return a dictionary of named groups in the regex match"""
+ match = re.match(self.regex, uri, self.regexFlags)
+ if not match:
+ raise InvalidURIException("Invalid URI: %r" % uri)
+ return match.groupdict()
+
+ def assigned(self, uri, newRuleset):
+ self.parseURI(uri)
+
+
+class RulesetDelivery(object):
+ """Combines the given Ruleset and URIHandler handlers into a callable
+ object that can be used as a Message.Hub client. Incoming messages
+ are tested against the ruleset, and if a non-None result is generated
+ that is sent to the given URIHandler.
+ """
+ def __init__(self, ruleset, uriHandler):
+ self.ruleset = ruleset
+ self.uriHandler = uriHandler
+
+ def __call__(self, message):
+ # We catch exceptions here and log them, preventing one bad ruleset
+ # or URI handler from preventing message delivery to other clients.
+ # We can't do this in the Message.Hub because sometimes it's good for
+ # exceptions to be propagated to the original sender of the message.
+ # In ruleset delivery however, messages never have a return value
+ # and shouldn't raise exceptions.
+ try:
+ result = self.ruleset(message)
+ if result:
+ self.uriHandler.message(self.ruleset.uri, message, result)
+ except:
+ e = sys.exc_info()[1]
+ log.msg(("Exception occurred in RulesetDelivery for %r\n" +
+ "--- Original message\n%s\n--- Exception\n%s") %
+ (self.ruleset.uri,
+ unicode(message).encode('ascii', 'replace'),
+ "".join(traceback.format_exception(*sys.exc_info()))))
+
+
+class UnsupportedURI(Exception):
+ """Raised on failure in URIRegistry.query()"""
+ pass
+
+
+class URIRegistry(object):
+ """A central authority for storing URIHandler instances and looking up
+ one appropriate for a given URI.
+ """
+ def __init__(self, *handlers):
+ self.handlers = list(handlers)
+
+ def register(self, handler):
+ self.handlers.append(handler)
+
+ def query(self, uri):
+ """Return a URIHandler instance appropriate for the given URI.
+ Raises an UnsupportedURI exception on failure.
+ """
+ for handler in self.handlers:
+ if handler.detect(uri):
+ return handler
+ raise UnsupportedURI("No registered URI handler supports %r" % uri)
+
+ def rulesetsRefreshed(self):
+ """Notify all URI handlers that rulesets are done being refreshed"""
+ for handler in self.handlers:
+ handler.rulesetsRefreshed()
+
+
+class RulesetStorage:
+ """Abstract base class for a persistent list of Rulesets, stored in
+ RulesetDelivery objects with URIHandlers looked up from the provided
+ URIRegistry. The generated RulesetDelivery objects are automatically added
+ to and removed from the supplied Message.Hub.
+
+ At startup, rulesets are loaded from some persistent source and any
+ modifications are recorded. Subclasses must implement this persistence.
+ """
+ def __init__(self, hub, uriRegistry):
+ self.uriRegistry = uriRegistry
+ self.hub = hub
+ self._emptyStorage()
+ self.refreshUntilDone()
+
+ def refreshUntilDone(self, interval=5):
+ """Refresh rulesets, retrying if an error occurs"""
+ self.refresh().addErrback(self._reRefresh, interval)
+
+ def _reRefresh(self, failure, interval):
+ log.msg("Refresh failed, retrying in %r seconds (%s)" %
+ (interval, failure.getBriefTraceback()))
+ reactor.callLater(interval, self.refreshUntilDone)
+
+ def refresh(self):
+ """Begin the process of loading our rulesets in from the SQL database.
+ Returns a Deferred that signals the operation's completion.
+ """
+ log.msg("Starting to refresh rulesets...")
+ result = defer.Deferred()
+ defer.maybeDeferred(self.dbIter).addCallback(
+ self._refresh, result).addErrback(result.errback)
+ return result
+
+ def _refresh(self, seq, result):
+ self._emptyStorage()
+ count = 0
+ for ruleset in seq:
+ try:
+ self._store(Ruleset(ruleset))
+ count += 1
+ except:
+ log.msg("Failed to load ruleset %r:\n%s" % (
+ ruleset,
+ "".join(traceback.format_exception(*sys.exc_info())),
+ ))
+
+ log.msg("%d rulesets loaded" % count)
+ self.uriRegistry.rulesetsRefreshed()
+ result.callback(None)
+ return seq
+
+ def _emptyStorage(self):
+ # Remove any existing rulesets from the Message.Hub
+ if hasattr(self, 'rulesetMap'):
+ for value in self.rulesetMap.itervalues():
+ self.hub.delClient(value)
+
+ # self.rulesetMap maps URIs to RulesetDelivery instances
+ self.rulesetMap = {}
+
+ def store(self, rulesetXml):
+ """Find a URIHandler for the given ruleset and add it to
+ our mapping and to the hub. 'ruleset' is given as a DOM tree.
+
+ Storing an empty ruleset for a particular URI is equivalent
+ to removing that URI's ruleset.
+
+ It is important that this function doesn't actually
+ remove or change the ruleset in quesion unless any possible
+ input errors have already been detected.
+
+ This updates both our in-memory mapping of compiled rulesets,
+ and the table of XML rulesets in our persistent storage.
+
+ Returns a deferred, the callback of which is executed
+ when the SQL database for this ruleset has been updated.
+ The in-memory database will be updated immediately.
+ """
+ ruleset = Ruleset(rulesetXml)
+ self._store(ruleset)
+ self.dbStore(ruleset)
+
+ def _store(self, ruleset):
+ """Internal version of store() that doesn't update the database,
+ and requires the ruleset to already be parsed.
+ """
+ # We need to find an appropriate URI handler whether our ruleset
+ # is empty or not, since we have to be able to notify the handler.
+ handler = self.uriRegistry.query(ruleset.uri)
+
+ # Is this ruleset non-empty?
+ if not ruleset.isEmpty():
+ # It's important that we give the URI handler a chance
+ # to return errors before removing the old ruleset.
+ handler.assigned(ruleset.uri, ruleset)
+
+ # If there was an old ruleset, remove its hub client
+ if self.rulesetMap.has_key(ruleset.uri):
+ self.hub.delClient(self.rulesetMap[ruleset.uri])
+
+ # Stick on an appropriate URI handler and add the
+ # resulting RulesetDelivery instance to the message hub
+ delivery = RulesetDelivery(ruleset, handler)
+ self.rulesetMap[ruleset.uri] = delivery
+ self.hub.addClient(delivery)
+ log.msg("Set ruleset for %r" % ruleset.uri)
+ else:
+ # Remove the ruleset completely if there was one
+ if self.rulesetMap.has_key(ruleset.uri):
+ self.hub.delClient(self.rulesetMap[ruleset.uri])
+ del self.rulesetMap[ruleset.uri]
+
+ log.msg("Removed ruleset for %r" % ruleset.uri)
+ handler.unassigned(ruleset.uri)
+
+ def flatten(self):
+ """Return a flat list of all Ruleset objects so we can store 'em"""
+ for delivery in self.rulesetMap.itervalues():
+ yield delivery.ruleset
+
+ def dbIter(self):
+ """Returns (optionally via a Deferred) an iterable that
+ contains a Ruleset object (or a string or DOM representing
+ a ruleset) for every ruleset stored persistently.
+ Implemented by subclasses.
+ """
+ return []
+
+ def dbStore(self, ruleset):
+ """Stores the given Ruleset object persistently in our
+ database. If ruleset.isEmpty(), this should end up
+ deleting the stored ruleset for the given URI if it exists.
+ """
+ pass
+
+
+class DatabaseRulesetStorage(RulesetStorage):
+ """A RulesetStorage that keeps rulesets in an SQL database"""
+ def dbIter(self):
+ """Begin the process of loading our rulesets in from the SQL database.
+ Returns the list of rulesets via a deferred.
+ """
+ import Database
+ result = defer.Deferred()
+ d = Database.pool.runQuery("SELECT * FROM rulesets")
+ d.addCallback(self._storeDbRulesets, result)
+ d.addErrback(result.errback)
+ return result
+
+ def _storeDbRulesets(self, rulesets, result):
+ """The second step in the refresh() process, loading the retrieved list of rulesets"""
+ result.callback(self._iterRulesetValues(rulesets))
+ return rulesets
+
+ def _iterRulesetValues(self, rulesets):
+ for uri, ruleset in rulesets:
+ yield ruleset
+
+ def dbStore(self, ruleset):
+ """Store a ruleset persistently in our SQL database"""
+ import Database
+
+ # Delete the old ruleset, if there was one
+ result = defer.Deferred()
+ d = Database.pool.runOperation("DELETE FROM rulesets WHERE uri = %s" % Database.quote(ruleset.uri, 'text'))
+
+ # If we need to insert a new ruleset, do that after the delete finishes
+ if ruleset.isEmpty():
+ d.addCallback(result.callback)
+ else:
+ d.addCallback(self._insertRuleset, result, ruleset)
+ d.addErrback(result.errback)
+ return result
+
+ def _insertRuleset(self, none, result, ruleset):
+ """Callback used by store() to insert a new or modified ruleset into the SQL database"""
+ import Database
+ d = Database.pool.runOperation("INSERT INTO rulesets (uri, xml) values(%s, %s)" % (
+ Database.quote(ruleset.uri, 'text'), Database.quote(XML.toString(ruleset.xml), 'text')))
+ d.addCallback(result.callback)
+ d.addErrback(result.errback)
+
+
+class FileRulesetStorage(RulesetStorage):
+ """A simple RulesetStorage that uses a simple flat text file"""
+ def __init__(self, hub, uriRegistry, path):
+ self.path = path
+ RulesetStorage.__init__(self, hub, uriRegistry)
+
+ def dbIter(self):
+ if os.path.isfile(self.path):
+ dom = XML.parseStream(open(self.path))
+ for element in XML.getChildElements(dom.documentElement):
+ if element.nodeName == "ruleset":
+ yield element
+ else:
+ log.msg("The file %r does not exist, loading no rulesets" % self.path)
+
+ def dbStore(self, ruleset=None):
+ """Write all rulesets to disk in one big XML file"""
+ doc = XML.parseString("<rulesets>\n"
+ "<!--\n"
+ "This is a ruleset storage for CIA. It tells the CIA server\n"
+ "how to deliver messages. Don't edit it by hand while the server\n"
+ "is running, use tools/ruleset_editor.py\n"
+ "-->\n"
+ "\n"
+ "</rulesets>")
+ root = doc.documentElement
+ for ruleset in self.flatten():
+ root.appendChild(XML.Domlette.ConvertDocument(ruleset.xml).documentElement)
+ root.appendChild(doc.createTextNode("\n\n"))
+
+ f = open(self.path, "w")
+ XML.Domlette.Print(doc, f)
+ f.write("\n")
+
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Ruleset.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Ruleset.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Security.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Security.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Security.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,515 @@
+""" LibCIA.Security
+
+Implements CIA's simple security model. CIA uses a simple
+capabilities-like system, where a particular capability is represented
+on the wire as an unguessable random key, and internally by nearly any
+python object.
+
+Generally the 'universe' key will be saved somewhere only the server's
+owner can access it on startup. The 'universe' key can be used to grant
+other keys, which can then be distributed to other people or machines.
+
+Note that this system has many of the same qualities as traditional
+capabilities, but is not implemented in the same way. In traditional
+capabilities, the unguessable keys map directly to objects that provide
+whatever interface that key grants permissions to. This is simple and
+effective for some systems, however in CIA the meaning of a key must be
+preserved over a long period of time regardless of code changes- simply
+making keys map to pickled callable objects would be too fragile.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.web import xmlrpc
+from twisted.internet import defer
+from twisted.python import failure, log
+import string, os, md5, time, cPickle
+from cStringIO import StringIO
+import Database, RpcServer
+
+# PyCAPTCHA is optional. We use it for certain operations
+# that should be easy for humans to perform but should not
+# be scriptable, like creating new users and granting capabilities to them.
+try:
+ import Captcha
+ import Captcha.Visual.Tests
+ # This trick keeps our factory from being overwritten on rebuild()
+ if 'captchaFactory' not in globals():
+ captchaFactory = Captcha.Factory()
+except ImportError:
+ captchaFactory = None
+
+
+class SecurityException(Exception):
+ pass
+
+class NoSuchUser(SecurityException):
+ pass
+
+class InsufficientCapabilities(SecurityException):
+ pass
+
+
+class SecurityInterface(RpcServer.Interface):
+ """An XML-RPC interface to the global capabilities database"""
+ def protected_createUser(self, fullName, email, loginName=None):
+ """Create a new user with the given identity, returning
+ a (uid, key) tuple for the new user.
+ """
+ u = User(full_name=fullName, email=email,
+ login_name=loginName, create=True)
+ result = defer.Deferred()
+ u.getInfo().addCallback(self._createUser, result).addErrback(result.errback)
+ return result
+
+ def _createUser(self, info, result):
+ """This gets the user info dict from User.getInfo()
+ and reformats it into a (uid, key) tuple for createUser()
+ """
+ result.callback((info['uid'], info['secret_key']))
+
+ def protected_grant(self, capability, uid):
+ """Grant the given capability to the given user. Note that this means
+ that the capability to use this function is effectively equivalent
+ to the 'universe' key
+ """
+ # XML-RPC will munge tuples into lists, when we really want a tuple
+ if type(capability) == list:
+ capability = tuple(capability)
+
+ return User(int(uid)).grant(capability)
+
+ def xmlrpc_test(self, key, *capabilities):
+ """Test the given key against one or more capabilities, returning True if it
+ matches any of them, False otherwise.
+ """
+ return caps.test(key, *capabilities)
+
+ def getCaptchaFactory(self):
+ """Return our current CAPTCHA factory"""
+ global captchaFactory
+ if not captchaFactory:
+ raise SecurityException("PyCAPTCHA is not installed")
+ return captchaFactory
+
+ def getCaptchaFromID(self, id):
+ """Get a CAPTCHA test given its ID"""
+ test = self.getCaptchaFactory().get(id)
+ if not test:
+ raise SecurityException("No test with the given ID was found, it might have expired")
+ return test
+
+ def xmlrpc_newCaptcha(self):
+ """Creates a new CAPTCHA test that the caller can solve to perform
+ certain operations that humans should be able to do easily but
+ shouldn't be scriptable. Returns an ID that uniquely identifies
+ this CAPTCHA test.
+ """
+ return self.getCaptchaFactory().new(Captcha.Visual.Tests.PseudoGimpy).id
+
+ def xmlrpc_renderCaptcha(self, id):
+ """Renders a (value, type) tuple containing a rendered version of
+ the CAPTCHA test with the given ID. Currently the type will
+ always be image/jpeg.
+ """
+ io = StringIO()
+ self.getCaptchaFromID(id).render().save(io, "JPEG")
+ return (xmlrpc.Binary(io.getvalue()), "image/jpeg")
+
+ def requireCaptcha(self, id, solutions):
+ """A utility for XML-RPC interfaces that require that they were
+ originally invoked by a human. The caller must have previously
+ requested and rendered a test using newCaptcha() and renderCaptcha(),
+ then presented it to the user. They must then present the test ID and
+ solution(s) to gain access to this function.
+ A particular CAPTCHA test can only be used once.
+ """
+ if not self.getCaptchaFromID(id).testSolutions(solutions):
+ raise SecurityException("Incorrect or expired CAPTCHA solution(s)")
+
+ def xmlrpc_selfCreateUser(self, captchaId, captchaSolutions, fullName, email, loginName=None):
+ """Whereas createUser() above requires special permissions,
+ anyone can call this function if they can prove they're human
+ by solving a CAPTCHA test. This lets people create new accounts
+ for themselves on their own. Unlike createUser(), this can not be
+ used to retrieve information about an existing user.
+ """
+ self.requireCaptcha(captchaId, captchaSolutions)
+ u = User(full_name=fullName, email=email,
+ login_name=loginName, create=True)
+ result = defer.Deferred()
+ u.getInfo().addCallback(self._selfCreateUser, u, result).addErrback(result.errback)
+ return result
+
+ def _selfCreateUser(self, info, user, result):
+ if not user.newUser:
+ raise SecurityException("That user already exists")
+ result.callback((info['uid'], info['secret_key']))
+
+ def xmlrpc_selfGrant(self, captchaId, captchaSolutions, userKey, capability):
+ """Any user can use this function, and a CAPTCHA solution (to prevent
+ scripted aquisition of many capabilities) to grant themselves certain
+ capabilities. Only a limited number of capabilities can be granted
+ with this function.
+ """
+ self.requireCaptcha(captchaId, captchaSolutions)
+ # XML-RPC will munge tuples into lists, when we really want a tuple
+ if type(capability) == list:
+ capability = tuple(capability)
+
+ # Make sure the capability is one we allow
+ if not self.isSelfGrantable(capability):
+ raise SecurityException("This capability can not be self-granted")
+
+ return User(key=userKey).grant(capability)
+
+ def isSelfGrantable(self, capability):
+ """Defines policy on which capabilities can be self-granted by a user"""
+ if type(capability) == tuple and len(capability) == 2:
+ # Two-element tuples...
+
+ if capability[0] == "stats.path":
+ # Granting a capability to a stats path is allowed
+ return True
+
+ return False
+
+
+def createRandomKey(bytes = 24,
+ allowedChars = string.ascii_letters + string.digits):
+ """Create a somewhat-secure random string of the given length.
+ This implementation probably only works on Linux and similar systems.
+ Also note that since we're using /dev/urandom instead of /dev/random,
+ the system might be out of entropy. Using /dev/random however could
+ block, and would make everything else here more complex.
+ The result will be base64-encoded.
+ """
+ s = ''
+ f = open("/dev/urandom")
+ for i in xrange(bytes):
+ s += allowedChars[ ord(f.read(1)) % len(allowedChars) ]
+ f.close()
+ return s
+
+
+def logProtectedCall(result, path, args, user, allowed=True):
+ """This should be called when a protected call was attempted,
+ successful or not. It logs the attempt and its results in the
+ audit_trail database. This audit trail can be used for several things-
+ listing recently updated metadata (perhaps for a 'whats new?' page)
+ or detecting and recovering from malicious use of keys.
+ """
+ # Store the first argument separately so we can relatively efficiently search for it
+ if args:
+ main_param = str(args[0])
+ else:
+ main_param = None
+
+ # Get the user's UID. If it hasn't even been looked up successfully,
+ # this is just a failed operation on a nonexistent user and it's not worth logging.
+ uid = user.getCachedUid()
+ if uid is None:
+ return
+
+ Database.pool.runOperation(
+ "INSERT INTO audit_trail (timestamp, uid, action_domain, action_name,"
+ " main_param, params, allowed, results)"
+ " VALUES(%d, %d, 'protected_call', %s, %s, '%s', %d, '%s')" % (
+ time.time(),
+ uid,
+ Database.quote(".".join(path), 'text'),
+ Database.quote(main_param, 'text'),
+ Database.quoteBlob(cPickle.dumps(args)),
+ allowed,
+ Database.quoteBlob(cPickle.dumps(result))))
+ return result
+
+
+class User:
+ """Representation of one user. A user always has a secret key
+ and a list of capabilities. Most users have a name, though
+ this isn't required. Users may also have login information.
+
+ A User instance may be created in several different ways:
+ - From a user ID
+ - From a full name
+ - From a login name
+ - From a secret key
+ - If nothing else is specified, this returns the default (system)
+ user, which has no name.
+
+ Normally if a matching user can't be found, this throws a NoSuchUser
+ exception. If create=True and we're searching by full name, this
+ can create a new user if necessary.
+
+ Note that any exception won't be generated until the user is
+ first queried (such as via getUid) since a database lookup
+ must be made.
+ """
+ def __init__(self, uid=None, full_name=None, login_name=None, key=None,
+ email=None, create=False):
+ self._uid = uid
+ self._full_name = full_name
+ self._login_name = login_name
+ self._key = key
+ self._create = create
+ self._email = email
+ self.newUser = False
+
+ self._validatedUid = None
+
+ def getUid(self):
+ """Return, via a Deferred, this user's ID"""
+ return Database.pool.runInteraction(self._getUid)
+
+ def _getUid(self, cursor):
+ """Return the user ID, given a database cursor. Caches the validated UID"""
+ if self._validatedUid is None:
+ self._validatedUid = self._uncachedGetUid(cursor)
+ return self._validatedUid
+
+ def getCachedUid(self):
+ """If we've already looked up this UID before, this will immediately
+ return the UID. It returns None instead of a deferred if we don't
+ know what our UID is yet.
+ """
+ return self._validatedUid
+
+ def _uncachedGetUid(self, cursor):
+ """Internal function to find a valid UID, without caching"""
+ if self._uid is not None:
+ return self._validateUid(cursor, self._uid)
+
+ if self._key is not None:
+ # Look for a user with the given key
+ return self._getUidFromKey(cursor, self._key)
+
+ if self._login_name is not None:
+ # Get/create a user with the given login name
+ try:
+ return self._getUidFromLoginName(cursor, self._login_name)
+ except NoSuchUser:
+ if self._create:
+ return self._createUser(cursor)
+ else:
+ raise
+
+ if self._full_name is not None:
+ # Get/create a user with the given full name
+ try:
+ return self._getUidFromFullName(cursor, self._full_name)
+ except NoSuchUser:
+ if self._create:
+ return self._createUser(cursor)
+ else:
+ raise
+
+ if self._email is not None:
+ # Look for a user by email address (can't create one unless a name is given too)
+ return self._getUidFromEmail(cursor, self._email)
+
+ # Nothing specified, get/create the default user.
+ # Since we should always have a default user, ignore the create flag
+ # and always create if necessary.
+ try:
+ return self._getUidFromFullName(cursor, None)
+ except NoSuchUser:
+ return self._createUser(cursor)
+
+ def getInfo(self):
+ """Return, via a Deferred, a dictionary of information about this user.
+ Dict keys match database columns. This includes uid, secret_key,
+ active, full_name, email, creation_time, key_atime, login_name,
+ login_passwd_md5, login_atime, and login_mtime.
+ """
+ return Database.pool.runInteraction(self._getInfo)
+
+ def _getInfo(self, cursor):
+ uid = self._getUid(cursor)
+ cursor.execute("SELECT * FROM users WHERE uid = %d" % uid)
+ row = cursor.fetchone()
+ d = {}
+ for i in xrange(len(row)):
+ d[cursor.description[i][0]] = row[i]
+ return d
+
+ def saveKey(self, file, *grantCapabilities):
+ """Save a key for this user to disk, after optionally
+ granting them capabilities if necessary. This is here to
+ bootstrap the capability system, since normally creating
+ users and granting capabilities requires one to already have
+ a powerful user's key.
+ """
+ return Database.pool.runInteraction(self._saveKey, file, *grantCapabilities)
+
+ def _saveKey(self, cursor, file, *grantCapabilities):
+ file = os.path.expanduser(file)
+ self._grant(cursor, *grantCapabilities)
+ key = self._getInfo(cursor)['secret_key']
+ f = open(file, "w")
+ os.chmod(file, 0600)
+ f.write(key)
+ f.close()
+
+ def test(self, *capabilities):
+ """Test this user for one or more of the given capabilities.
+ Returns true if the user has any of the given capabilities,
+ false if they have none, or raises a NoSuchUser exception
+ if this isn't a valid user. Returns its result in a Deferred.
+
+ If the capabilities list is empty, this will always return True
+ for a valid user.
+ """
+ return Database.pool.runInteraction(self._test, *capabilities)
+
+ def _test(self, cursor, *capabilities):
+ # If the user has been disabled, they have no capabilities
+ cursor.execute("SELECT active FROM users WHERE uid = %d" % self._getUid(cursor))
+ if not int(cursor.fetchone()[0]):
+ return False
+
+ if cursor.execute(self._createTestQuery(self._getUid(cursor), capabilities)):
+ # We do have permission, update the key's access time and return
+ cursor.execute("UPDATE users SET key_atime = %d WHERE uid = %d" %
+ (time.time(), self._getUid(cursor)))
+ return True
+ else:
+ return False
+
+ def grant(self, *capabilities):
+ """Grant all capabilities in the given list, ignoring any the user already has"""
+ return Database.pool.runInteraction(self._grant, *capabilities)
+
+ def _grant(self, cursor, *capabilities):
+ uid = self._getUid(cursor)
+ for capability in capabilities:
+ rep = repr(capability)
+ cursor.execute("INSERT IGNORE INTO capabilities (uid, cap_md5, cap_repr)"
+ " VALUES(%d, %s, %s)" % (
+ uid,
+ Database.quote(md5.new(rep).hexdigest(), 'char'),
+ Database.quote(rep, 'text')))
+
+ def require(self, *capabilities):
+ """Like test(), but in case none of the listed capabilities have been
+ granted to this user, raises a SecurityException.
+ This is guaranteed to either return None or a Failure, where the
+ Failure should be a SecurityException.
+ """
+ return Database.pool.runInteraction(self._require, *capabilities)
+
+ def _require(self, cursor, *capabilities):
+ try:
+ if not self._test(cursor, *capabilities):
+ raise InsufficientCapabilities("One of the following capabilities are required: " +
+ repr(capabilities)[1:-1])
+ except:
+ return failure.Failure()
+
+ def _createTestQuery(self, uid, capabilities):
+ """Create an SQL query that returns something nonzero if a uid matches any of
+ a list of capabilities. If the capabilities list is empty, this creates a
+ query that always has a nonzero result.
+ """
+ if capabilities:
+ return "SELECT 1 FROM capabilities WHERE uid = %d AND (%s) LIMIT 1" % (
+ uid,
+ " OR ".join(["cap_md5 = " + Database.quote(md5.new(repr(c)).hexdigest(),
+ 'char') for c in capabilities]),
+ )
+ else:
+ return "SELECT 1"
+
+ def _getUidFromKey(self, cursor, key):
+ """Find a user by their key"""
+ cursor.execute("SELECT uid FROM users WHERE secret_key = %s" %
+ Database.quote(key.strip(), 'varchar'))
+ row = cursor.fetchone()
+ if row:
+ return int(row[0])
+ else:
+ raise NoSuchUser("No user found matching the given key")
+
+ def _getUidFromLoginName(self, cursor, name):
+ """Find a user by login name"""
+ cursor.execute("SELECT uid FROM users WHERE login_name = %s" %
+ Database.quote(name, 'text'))
+ row = cursor.fetchone()
+ if row:
+ return int(row[0])
+ else:
+ raise NoSuchUser("No such login name: %r" % name)
+
+ def _getUidFromFullName(self, cursor, name):
+ """Find a user by full name"""
+ if name is None:
+ cursor.execute("SELECT uid FROM users WHERE full_name is NULL")
+ else:
+ cursor.execute("SELECT uid FROM users WHERE full_name = %s" %
+ Database.quote(name, 'text'))
+ row = cursor.fetchone()
+ if row:
+ return int(row[0])
+ else:
+ raise NoSuchUser("No such name: %r" % name)
+
+ def _getUidFromEmail(self, cursor, name):
+ """Find a user by email address"""
+ cursor.execute("SELECT uid FROM users WHERE email = %s" %
+ Database.quote(name, 'text'))
+ row = cursor.fetchone()
+ if row:
+ return int(row[0])
+ else:
+ raise NoSuchUser("No such email address: %r" % name)
+
+ def _createUser(self, cursor):
+ """Create a new user, optionally setting the given parameters.
+ Returns the new user ID.
+ """
+ log.msg("Creating new user %r" % self._full_name)
+ self.newUser = True
+ cursor.execute("INSERT INTO users (secret_key, creation_time, full_name, email, login_name) "
+ "VALUES (%s, %d, %s, %s, %s)" % (
+ Database.quote(createRandomKey(), 'varchar'),
+ time.time(),
+ Database.quote(self._full_name, 'text'),
+ Database.quote(self._email, 'text'),
+ Database.quote(self._login_name, 'varchar')))
+ cursor.execute("SELECT LAST_INSERT_ID()")
+ return int(cursor.fetchone()[0])
+
+ def _validateUid(self, cursor, uid):
+ """We have a UID. Validate it, returning a valid UID or raising NoSuchUser"""
+ cursor.execute("SELECT 1 FROM users WHERE uid = %d" % uid)
+ if cursor.fetchone():
+ return uid
+ else:
+ raise NoSuchUser("No such user ID: %d" % uid)
+
+ def getCapabilities(self):
+ """Return a list of capability names (strings) that this user has, via a Deferred"""
+ return Database.pool.runInteraction(self._getCapabilities)
+
+ def _getCapabilities(self, cursor):
+ uid = self._getUid(cursor)
+ cursor.execute("SELECT cap_repr FROM capabilities WHERE uid = %d" % uid)
+ return [row[0] for row in cursor.fetchall()]
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Security.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Security.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Stats/Graph.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Stats/Graph.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Stats/Graph.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,418 @@
+""" LibCIA.Stats.Graph
+
+Maintains an undirected graph of associations between stats targets.
+These associations are reinforced when one message is delivered to
+multiple targets.
+
+Using this graph stored in the database, various methods of visualization
+have been implemented here.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from LibCIA import Database, Cache
+from twisted.python import log
+from twisted.internet import reactor, defer, protocol, error
+from cStringIO import StringIO
+import time, math
+
+
+class Relation:
+ """Represents a relationship between two stats targets- an edge on the stats graph.
+ These relationships can be queried or reinforced. You can ask a stats target for
+ a list of relations containing it.
+ """
+ def __init__(self, a, b):
+ # Our targets must be sorted by path, to make this edge unique
+ if a.path > b.path:
+ a, b = b, a
+ self.a = a
+ self.b = b
+
+ def reinforce(self):
+ """Increment this relation's strength and set its freshness to the current time"""
+ return Database.pool.runInteraction(self._reinforce)
+
+ def _reinforce(self, cursor):
+ """Database interaction implementing reinforce()"""
+ # First touch the edge to make sure it exists. We have to do this
+ # inside two autoCreateTargetFor levels, to create both stats targets
+ # if they don't yet exist.
+ self.a._autoCreateTargetFor(cursor, self.b._autoCreateTargetFor,
+ cursor, cursor.execute,
+ "INSERT IGNORE INTO stats_relations "
+ "(target_a_path, target_b_path) VALUES(%s, %s)" %
+ (Database.quote(self.a.path, 'varchar'),
+ Database.quote(self.b.path, 'varchar')))
+
+ cursor.execute("UPDATE stats_relations "
+ "SET strength = strength + 1, freshness = %s "
+ "WHERE target_a_path = %s AND target_b_path = %s" %
+ (Database.quote(int(time.time()), 'bigint'),
+ Database.quote(self.a.path, 'varchar'),
+ Database.quote(self.b.path, 'varchar')))
+
+
+class Selector:
+ """A selector provides SQL and Python syntax for picking some
+ subset of stats targets, and describes how those stats targets
+ should be rendered.
+ This is an abstract base class.
+ """
+ def getSQL(self, varName):
+ """Returns an SQL expression that is true if the stats target
+ path in 'varName' is part of this selector's subset.
+ """
+ pass
+
+ def __contains__(self, path):
+ """Returns True if the given path is part of this selector's subset."""
+ pass
+
+ def getAttributes(self, path):
+ """Given a path contained by this selector's subset, return a
+ dictionary of node attributes to pass to Graphviz.
+ """
+ pass
+
+
+class PrefixSelector(Selector):
+ """A selector that returns all nodes beginning with a particular prefix,
+ drawing them without the prefix but with a distinctive style.
+ """
+ def __init__(self, prefix, **style):
+ self.prefix = prefix
+ self.style = style
+
+ def __repr__(self):
+ return "<PrefixSelector %r %r>" % (self.prefix, self.style)
+
+ def getSQL(self, varName):
+ return "%s LIKE '%s%%'" % (varName, self.prefix)
+
+ def __contains__(self, path):
+ return path.startswith(self.prefix)
+
+ def getAttributes(self, path):
+ d = dict(self.style)
+ d['label'] = path[len(self.prefix):]
+ return d
+
+
+class RelationGrapher:
+ """Creates graphs showing relationships between stats targets.
+ One or more Selectors are used to find interesting nodes and
+ describe how they should be drawn. The output is given as a
+ 'dot' file that can be passed to the 'neato' utility in the
+ AT&T Graphviz package.
+ """
+ def __init__(self, *selectors):
+ self.selectors = selectors
+
+ def __repr__(self):
+ return "<RelationGrapher %r>" % (self.selectors,)
+
+ def getAllSelectorsSQL(self, varName):
+ """Return an SQL expression that is true if any selector is
+ interested in the stats target in the given variable name.
+ """
+ return "(%s)" % " OR ".join(["(%s)" % sel.getSQL(varName) for sel in self.selectors])
+
+ def getQuery(self):
+ """Returns an SQL query that returns all interesting edges in the
+ relation graph, according to our list of selectors.
+ """
+ return ("SELECT target_a_path, target_b_path, strength, freshness "
+ "FROM stats_relations WHERE %s AND %s" % (
+ self.getAllSelectorsSQL("target_a_path"),
+ self.getAllSelectorsSQL("target_b_path")))
+
+ def render(self, f):
+ """Generate the 'dot' code for our graph, writing it to the given
+ file-like object. Returns a Deferred signaling completion.
+ """
+ result = defer.Deferred()
+ Database.pool.runQuery(self.getQuery()).addCallback(
+ self._generateDot, f, result).addErrback(result.errback)
+ return result
+
+ def quote(self, t):
+ """Quote the given text for inclusion in a dot file"""
+ if type(t) in (int, float, bool):
+ return str(t)
+ else:
+ t = str(t)
+ for badChar in '\\":<>|':
+ t = t.replace(badChar, '\\' + badChar)
+ return '"' + t + '"'
+
+ def dictToAttrs(self, d):
+ """Convert a dictionary to a dot attribute string"""
+ if d:
+ return "[%s]" % ",".join(['%s=%s' % (key, self.quote(value))
+ for key, value in d.iteritems()])
+ else:
+ return ""
+
+ def _generateDot(self, rows, f, result):
+ """Finish generateDot after receiving the SQL query results"""
+ graphAttrs = {
+ 'packmode': 'graph',
+ 'center': True,
+ 'Damping': 0.9,
+ }
+
+ f.write("graph G {\n")
+ f.write(''.join(['\t%s=%s;\n' % (key, self.quote(value))
+ for key, value in graphAttrs.iteritems()]))
+
+ # Make a unique list of all nodes in this graph
+ nodes = {}
+ for row in rows:
+ nodes[row[0]] = None
+ nodes[row[1]] = None
+
+ # Keep track of node parents, so later we can use them for special cases
+ # and for determining how many children a particular node has.
+ parents = {}
+ for node in nodes:
+ segments = node.split('/')
+ # Make sure all ancestors of this node exist in the parents map
+ for i in xrange(len(segments)):
+ parent = '/'.join(segments[:i])
+ if parent:
+ parents.setdefault(parent, [])
+ # Add the node to its immediate parent
+ if len(segments) > 1:
+ parents[parent].append(node)
+
+ # Find a selector for each node, and write out their attributes
+ for node in nodes.keys():
+ attributes = {}
+ for selector in self.selectors:
+ if node in selector:
+ nodes[node] = selector
+ attributes = selector.getAttributes(node)
+ break
+ f.write('\t%s %s;\n' % (self.quote(node), self.dictToAttrs(attributes)))
+
+ # Find the maximum strength and minimum freshness in our dataset
+ now = time.time()
+ maxStrength = 0
+ minFreshness = now
+ for row in rows:
+ a, b, strength, freshness = row
+ if strength > maxStrength:
+ maxStrength = strength
+ if freshness < minFreshness:
+ minFreshness = freshness
+
+ # Write edges
+ for row in rows:
+ a, b, strength, freshness = row
+
+ # Exclude edges between nodes of dissimilar selectors when a parent node
+ # is involved. The rationale for this is that this edge doesn't give any
+ # real information, since the non-parent node will also have an edge
+ # connecting to a child of the parent node. For example, this would
+ # prevent author/bob from being linked to both project/kde/libwidgets
+ # and project/kde. The link between author/bob and project/kde would
+ # just clutter up the graph.
+ if (parents.has_key(a) or parents.has_key(b)) and nodes[a] != nodes[b]:
+ continue
+
+ # Scale the strength and freshness logarithmically to be values between 0 and 1
+ unitStrength = math.log(strength) / math.log(maxStrength)
+ unitFreshness = 1 - (math.log(now - freshness) / math.log(now - minFreshness))
+
+ # Determine the number of children this edge involves, for length scaling purposes
+ children = len(parents.get(a,())) + len(parents.get(b,()))
+
+ attributes = {
+ # The length is semantically unimportant, but try to give the graph a good
+ # layout. Increase the length proportionately with the number of children
+ # our nodes have.
+ 'len': 5.0 + math.log(children + 1),
+ 'weight': 5.0,
+
+ # Scale the line width according to the edge's strength
+ 'style': 'setlinewidth(%d)' % int(unitStrength * 10 + 1),
+ }
+
+ f.write('\t%s -- %s %s;\n' % (self.quote(a),
+ self.quote(b),
+ self.dictToAttrs(attributes)))
+
+ f.write("}\n")
+ result.callback(None)
+
+
+class StreamProcessProtocol(protocol.ProcessProtocol):
+ """Upon connection to the child process, stdinCallback is called with
+ a file-like object representing the process's stdin. When the
+ stdinCallback returns (possibly via a Deferred) the subprocess'
+ stdin is closed.
+
+ The process' stdout is directed to the provided file-like object
+ stdoutStream. When the process finishes, a None is sent to
+ resultDeferred. Errors are errback()'ed to resultDeferred.
+ """
+ def __init__(self, stdinCallback, stdoutStream, resultDeferred):
+ self.stdinCallback = stdinCallback
+ self.stdoutStream = stdoutStream
+ self.resultDeferred = resultDeferred
+
+ def connectionMade(self):
+ defer.maybeDeferred(self.stdinCallback, self.transport).addCallback(
+ self._finishedWriting).addErrback(self.resultDeferred.errback)
+
+ def _finishedWriting(self, result):
+ self.transport.closeStdin()
+
+ def outReceived(self, data):
+ self.stdoutStream.write(data)
+
+ def errReceived(self, data):
+ log.err(data)
+
+ def processEnded(self, reason):
+ if isinstance(reason.value, error.ProcessDone):
+ self.resultDeferred.callback(None)
+ else:
+ self.resultDeferred.errback(reason)
+
+
+class GraphLayout:
+ """Given an object with a render(f) function that writes 'dot' source to
+ a file-like object, this runs Graphviz's 'neato' utility to lay out the
+ graph and produce a vector image.
+ """
+ def __init__(self, dataSource, format="svg", bin="neato"):
+ self.dataSource = dataSource
+ self.bin = bin
+ self.args = [bin, "-T%s" % format]
+
+ def __repr__(self):
+ return "<GraphLayout for %r using %r>" % (self.dataSource, self.args)
+
+ def render(self, f):
+ """Lay out a graph using input from our data source, writing
+ the completed graph to the given file-like object.
+ """
+ result = defer.Deferred()
+ p = StreamProcessProtocol(self.dataSource.render, f, result)
+ reactor.spawnProcess(p, self.bin, self.args, env=None)
+ return result
+
+
+class HeaderChoppingProtocol(StreamProcessProtocol):
+ """This is a protocol that works around sodipodi's tencency to spew some
+ debuggative cruft to stdout rather than to stderr. Ignores all writes
+ we get before the first one >= 1 kB.
+ """
+ inhibit = True
+
+ def outReceived(self, data):
+ if self.inhibit:
+ if len(data) < 1024:
+ return
+ self.inhibit = False
+ self.stdoutStream.write(data)
+
+
+class SvgRasterizer:
+ """Uses sodipodi to rasterize SVG vector images into PNG bitmaps.
+ dataSource is any object with a render(f) function that can
+ write an SVG to a file-like object and might return a deferred.
+ This provides a render() function that writes the resulting PNG
+ image to a file-like object.
+ """
+ def __init__(self, dataSource,
+ width = None,
+ height = None,
+ dpi = None,
+ size = None,
+ background = None,
+ bin = "sodipodi"):
+ self.dataSource = dataSource
+ self.bin = bin
+ self.args = [bin, "-f", "-", "-e", "/dev/stdout"]
+ if dpi:
+ self.args.extend(["-d", str(dpi)])
+ if background:
+ self.args.extend(["-b", str(background)])
+ if size:
+ self.args.extend(["-w", str(size[0]), "-h", str(size[1])])
+ if width:
+ self.args.extend(["-w", str(width)])
+ if height:
+ self.args.extend(["-h", str(height)])
+
+ def __repr__(self):
+ return "<SvgRasterizer for %r using %r>" % (self.dataSource, self.args)
+
+ def render(self, f):
+ result = defer.Deferred()
+ p = HeaderChoppingProtocol(self.dataSource.render, f, result)
+ reactor.spawnProcess(p, self.bin, self.args, env=None)
+ return result
+
+
+class RenderCache(Cache.AbstractStringCache):
+ """A cache that transparently wraps any object with a render(f) function
+ that writes its results to a file-like object and has a repr() that
+ uniquely represents its state.
+
+ Cached results expire after 4 hours by default.
+ """
+ def __init__(self, obj, lifespan=60*60*4):
+ self.obj = obj
+ self.lifespan = lifespan
+
+ def __repr__(self):
+ return "<RenderCache of %r>" % self.obj
+
+ def render(self, f):
+ """A render() function that should appear to work just like the
+ original object's render() function.
+ """
+ result = defer.Deferred()
+ self.get(self.obj).addCallback(
+ self._render, f, result).addErrback(result.errback)
+ return result
+
+ def _render(self, value, f, result):
+ f.write(value)
+ result.callback(None)
+
+ def miss(self, obj):
+ """The cache miss function, as required by AbstractStringCache.
+ The cache is keyed on our renderable object, and this eventually
+ returns a string.
+ """
+ f = StringIO()
+ result = defer.Deferred()
+ defer.maybeDeferred(obj.render, f).addCallback(
+ self._finishedRendering, f, result).addErrback(result.errback)
+ return result
+
+ def _finishedRendering(self, d, f, result):
+ result.callback(f.getvalue())
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Stats/Graph.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Stats/Graph.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Stats/Handler.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Stats/Handler.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Stats/Handler.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,64 @@
+""" LibCIA.Stats.Handler
+
+The stats:// URI handler
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from LibCIA import Ruleset
+import time, posixpath
+from Target import StatsTarget
+from Graph import Relation
+
+
+class StatsURIHandler(Ruleset.RegexURIHandler):
+ """Handles stats:// URIs. A stats target is chosen,
+ and the message is delivered to it.
+ """
+ scheme = 'stats'
+ regex = r"^stats://(?P<path>([^/]+(/[^/]+)*)?)$"
+
+ def __init__(self):
+ self.lastMessage = None
+ self.messageTargets = []
+
+ def message(self, uri, message, content):
+ """Appends 'content' to the path represented by the given URI
+ and delivers a message to its associated stats target.
+
+ This includes a bit of a hack for tracking associations
+ between stats targets. We assume that messages are delivered
+ one at a time- if we get a duplicate message (presumably to a
+ different stats target) we add that stats target to the list of
+ targets this message has been delivered to, and reinforce the
+ new relations this forms.
+ """
+ path = posixpath.join(self.parseURI(uri)['path'], content)
+ target = StatsTarget(path)
+ target.deliver(message)
+
+ if message == self.lastMessage:
+ for prevTarget in self.messageTargets:
+ Relation(prevTarget, target).reinforce()
+ else:
+ self.messageTargets = []
+ self.messageTargets.append(target)
+ self.lastMessage = message
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Stats/Handler.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Stats/Handler.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Stats/Interface.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Stats/Interface.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Stats/Interface.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,248 @@
+""" LibCIA.Stats.Handler
+
+The stats:// URI handler, the root of the stats XML-RPC interface,
+and the maintenance system. All the usual entry points for the stats system.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.python import log
+from twisted.internet import defer
+from twisted.web import xmlrpc
+from LibCIA import RpcServer, Database, XML
+from Target import StatsTarget
+import cPickle, urlparse, time
+
+
+class StatsInterface(RpcServer.Interface):
+ """An XML-RPC interface used to query stats"""
+ def __init__(self):
+ RpcServer.Interface.__init__(self)
+ self.putSubHandler('metadata', MetadataInterface())
+ self.putSubHandler('subscribe', SubscriptionInterface())
+
+ def xmlrpc_catalog(self, path=''):
+ """Return a list of subdirectories within this stats path"""
+ result = defer.Deferred()
+ d = StatsTarget(path).catalog()
+ d.addErrback(result.errback)
+ d.addCallback(self._catalog, result)
+ return result
+
+ def _catalog(self, items, result):
+ """Convert the returned catalog into target names and return them via
+ the provided Deferred instance.
+ """
+ result.callback([target.name for target in items])
+
+ def xmlrpc_getLatestMessages(self, path, limit=None):
+ """Return 'limit' latest messages delivered to this stats target,
+ or all available recent messages if 'limit' isn't specified.
+ """
+ return [(id, XML.toString(doc)) for id, doc in
+ StatsTarget(path).messages.getLatest(limit)]
+
+ def xmlrpc_getCounterValues(self, path, name):
+ """Returns a dictionary with current values for the given counter.
+ Note that times are returned as UNIX-style seconds since
+ the epoch in UTC.
+ """
+ return StatsTarget(path).counters.getCounter(name)
+
+ def protected_clearTarget(self, path):
+ """Deletes any data stored at a given stats target or in any of its subtargets"""
+ log.msg("Clearing stats path %r" % path)
+ return StatsTarget(path).clear()
+
+ def caps_clearTarget(self, rpcPath, statsPath):
+ """In addition to the usual capabilities, allow ('stats.path', path)"""
+ return self.makeDefaultCaps(rpcPath) + [('stats.path', statsPath)]
+
+
+class MetadataInterface(RpcServer.Interface):
+ """An XML-RPC interface for querying and modifying stats metadata"""
+ def wrapTuple(self, t):
+ """Wrap the value in a (value, type) tuple in an xmlrpc.Binary
+ if the type doesn't start with text/.
+ """
+ if t and not t[1].startswith("text/"):
+ return (xmlrpc.Binary(t[0]), t[1])
+ else:
+ return t
+
+ def xmlrpc_get(self, path, name, default=False):
+ """Get a (value, type) tuple for the metadata key with the given
+ name, returning 'default' if it isn't found
+ """
+ result = defer.Deferred()
+ StatsTarget(path).metadata.get(name, default).addCallback(
+ self._get, result).addErrback(result.errback)
+ return result
+
+ def _get(self, t, result):
+ """Backend for get() that ensures the results are serializable"""
+ result.callback(self.wrapTuple(t))
+
+ def xmlrpc_dict(self, path):
+ """Return a mapping of names to (value, type) tuples for the given path"""
+ result = defer.Deferred()
+ StatsTarget(path).metadata.dict().addCallback(
+ self._dict, result).addErrback(result.errback)
+ return result
+
+ def _dict(self, original, result):
+ """Backend for dict() that ensures the results are serializable"""
+ d = {}
+ for name, t in original.iteritems():
+ d[name] = self.wrapTuple(t)
+ result.callback(d)
+
+ def caps_set(self, rpcPath, statsPath, name, value, mimeType=None):
+ """In addition to the usual capabilities, allow ('stats.path', path),
+ ('stats.metadata.path', path), and ('stats.metadata.key', path, name)
+ for setting metadata.
+ """
+ return self.makeDefaultCaps(rpcPath) + [
+ ('stats.path', statsPath),
+ ('stats.metadata.path', statsPath),
+ ('stats.metadata.key', statsPath, name),
+ ]
+
+ def caps_clear(self, rpcPath, statsPath):
+ return self.makeDefaultCaps(rpcPath) + [
+ ('stats.path', statsPath),
+ ('stats.metadata.path', statsPath),
+ ]
+
+ def caps_remove(self, rpcPath, statsPath, name):
+ return self.makeDefaultCaps(rpcPath) + [
+ ('stats.path', statsPath),
+ ('stats.metadata.path', statsPath),
+ ('stats.metadata.key', statsPath, name),
+ ]
+
+ def protected_set(self, path, name, value, mimeType='text/plain'):
+ """Set a metadata key's value and MIME type"""
+ return StatsTarget(path).metadata.set(name, str(value), mimeType)
+
+ def protected_clear(self, path):
+ """Remove all metadata for one target"""
+ return StatsTarget(path).metadata.clear()
+
+ def protected_remove(self, path, name):
+ """Remove one metadata key for this target, if it exists"""
+ return StatsTarget(path).metadata.remove(name)
+
+
+class SubscriptionInterface(RpcServer.Interface):
+ """An XML-RPC interface for subscribing to be notified when changes
+ occur to a stats target. This provides multiple ways of subscribing,
+ for compatibility with multiple existing standards.
+ """
+ def _unsubscribe(self, cursor, client):
+ """Remove all subscriptions for the given client.
+ This is intended to be run inside a database interaction.
+ """
+ cursor.execute("DELETE FROM stats_subscriptions WHERE client = %s" %
+ Database.quote(client, 'varchar'))
+
+ def makeTrigger(self, triggerFunc, *triggerArgs, **triggerKwargs):
+ """Given a trigger function and the arguments to pass it,
+ returns a pickled representation of the trigger.
+ """
+ return cPickle.dumps((triggerFunc, triggerArgs, triggerKwargs))
+
+ def _subscribe(self, cursor, target, client, trigger, scope=None, ttl=25*60*60):
+ """A database interaction for adding subscriptions.
+ 'target' must be the StatsTarget object this subscription is for.
+ 'client' is the IP address of the client requesting this subscription.
+ 'trigger' is a trigger pickle, as returned by makeTrigger
+ 'scope' refers to a part of the stats target this refers to. By default, all of it.
+ 'ttl' is the time to live for this subscription, 25 hours by default.
+ """
+ cursor.execute("INSERT INTO stats_subscriptions "
+ "(target_path, expiration, scope, client, `trigger`) "
+ "VALUES (%s, %s, %s, %s, '%s')" %
+ (Database.quote(target.path, 'varchar'),
+ Database.quote(int(time.time() + ttl), 'bigint'),
+ Database.quote(scope, 'varchar'),
+ Database.quote(client, 'varchar'),
+ Database.quoteBlob(trigger)))
+
+ def xmlrpc_rss2(self, procedureName, clientPort, responderPath, protocol, urls, request=None):
+ """This is the flavor of subscription required for the RSS 2.0 <cloud>
+ tag. The client IP should be determined from this request. 'urls'
+ is a list of URLs the client is interested in monitoring- we have
+ to convert those into stats targets.
+ """
+ log.msg("Received an RSS 2.0 subscription request for %r reporting via %r at %r on port %r to %r" %
+ (urls, protocol, responderPath, clientPort, procedureName))
+
+ # Reject protocols we can't handle
+ if protocol != 'xml-rpc':
+ log.msg("Rejecting request: unsupported protocol")
+ return False
+
+ # Poing off into a database interaction for most of this...
+ return Database.pool.runInteraction(self._rss2, procedureName, clientPort,
+ responderPath, urls, request.getClientIP())
+
+ def _rss2(self, cursor, procedureName, clientPort, responderPath, urls, clientIP):
+ """The database interaction implementing RSS 2.0 subscriptions"""
+ # Unsubscribe all of this client's previous requests
+ self._unsubscribe(cursor, clientIP)
+
+ # Generate a new subscription for each of its new URLs
+ for url in urls:
+ # Figure out the associated stats target for this URL
+ target = self.getTargetFromURL(url)
+ if not target:
+ log.msg("Ignoring URL %r which doesn't appear to be a stats target" % url)
+ continue
+
+ # Make a trigger that notifies the client as requested.
+ xmlrpcUrl = "http://%s:%s%s" % (clientIP, clientPort, responderPath)
+ trigger = self.makeTrigger(xmlrpc.Proxy(xmlrpcUrl).callRemote,
+ procedureName, url)
+
+ # Finally jam the trigger in our database
+ self._subscribe(cursor, target, clientIP, trigger)
+
+ # According to the RSS 2.0 spec, return True to indicate success
+ return True
+
+ def getTargetFromURL(self, url):
+ """Given a URL that presumably refers to our HTTP server, extract a stats
+ target from it. The URL should be to an RSS feed, but we won't be too
+ picky. If we can't find a related stats target, return None.
+ """
+ # We ignore everything except the path, assuming the URL actually
+ # refers to this host. There's no reason for someone to ask us to
+ # notify them if another server's page changes, and it would take
+ # more effort than it's worth to accurately determine if the given
+ # host refers to this CIA server, with all the proxying and DNS
+ # multiplicity that could be going on.
+ path = urlparse.urlparse(url)[2]
+
+ # FIXME: really cheesy hack!
+ if not path.startswith("/stats/"):
+ return None
+ return StatsTarget(path[7:].split("/.",1)[0])
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Stats/Interface.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Stats/Interface.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Stats/Messages.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Stats/Messages.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Stats/Messages.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,748 @@
+""" LibCIA.Stats.Messages
+
+This package provides a file format that stores a bounded amount
+of recent message data, as an indexed ring buffer of compressed
+SAX events.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import struct, os, datetime
+import xml.sax, xml.dom.minidom
+
+class CircularFile:
+ """This is a file-like object wrapper that turns a fixed-size
+ normal file into a file that wraps around at the end, creating
+ a ring buffer. This can optionally work within a subset of
+ the entire file by providing a smaller size and a nonzero origin.
+ """
+ def __init__(self, original, size, origin=0):
+ self.original = original
+ self.origin = origin
+ self.size = size
+ self.origin = origin
+
+ def seek(self, offset):
+ self.original.seek(self.origin + (offset % self.size))
+
+ def tell(self):
+ return self.original.tell() - self.origin
+
+ def write(self, data, offset=None):
+ """Write data to the circular file, wrapping around if we hit the end.
+ """
+ if offset is not None:
+ self.seek(offset)
+ initial = self.tell()
+ dataLen = len(data)
+
+ while dataLen > 0:
+ final = initial + dataLen
+
+ if final < self.size:
+ # This write will fit without wrapping
+ self.original.write(data)
+ return
+
+ elif final == self.size:
+ # This write just barely fits, but the file pointer needs
+ # to come back to the beginning afterwards.
+ self.original.write(data)
+ self.seek(0)
+ return
+
+ else:
+ # This write will straddle the end of the file. Write
+ # what we can now, and catch the rest later. This iterative
+ # method handles writes that are larger than the entire
+ # buffer without any special cases.
+ chunkLen = self.size - initial
+ chunk = data[:chunkLen]
+ data = data[chunkLen:]
+ dataLen -= chunkLen
+
+ self.original.write(chunk)
+ del chunk
+ self.seek(0)
+ initial = 0
+
+ def read(self, dataLen, offset=None):
+ """Read from the circular file, wrapping around if we hit the end"""
+ blocks = []
+ self.readBlocks(dataLen, blocks, offset)
+ return ''.join(blocks)
+
+ def readBlocks(self, dataLen, blocks, offset=None):
+ """Like read(), but instead of returning a single string this
+ adds each individual piece to the 'blocks' list. This is intended
+ to improve efficiency when doing several reads into the
+ same buffer.
+
+ If an initial offset is given, we seek to that rather than
+ reading the current offset in the file.
+ """
+ if offset is not None:
+ self.seek(offset)
+ offset = self.tell()
+
+ while dataLen > 0:
+ final = offset + dataLen
+
+ if final < self.size:
+ # This read will fit without wrapping
+ blocks.append(self.original.read(dataLen))
+ break
+
+ elif final == self.size:
+ # This read just barely fits, but the file pointer needs
+ # to come back to the beginning afterwards.
+ blocks.append(self.original.read(dataLen))
+ self.seek(0)
+ break
+
+ else:
+ # This read will straddle the end of the file
+ chunkLen = self.size - offset
+ dataLen -= chunkLen
+
+ blocks.append(self.original.read(chunkLen))
+ self.seek(0)
+ offset = 0
+
+ def readSlice(self, firstOffset, stoppingOffset):
+ """Read all data starting at the firstOffset and stopping just before
+ we reach the stoppingOffset. This is the same style of half-open range
+ that python list slicing uses. stoppingOffset may be less than
+ firstOffset, in which case we wrap around. If stoppingOffset equals
+ firstOffset, this returns nothing.
+ """
+ return self.read((stoppingOffset - firstOffset) % self.size, offset=firstOffset)
+
+
+class SAXEncoder(xml.sax.handler.ContentHandler):
+ """Encodes a SAX event stream as a unicode text
+ stream, with string memoization.
+
+ Every event begins with a single unicode
+ character that has an opcode packed into the low
+ 2 bits, and a paramter occupying all upper bits.
+
+ Opcodes:
+
+ 0. The parameter specifies the number of endElement events to emit
+
+ 1. The parameter is a string length L. The following L bytes define
+ a new string, which is also added to the memo.
+
+ 2. Emit a memoized string. The parameter is the memo ID of that string.
+ Memo IDs start at zero, but the memo is generally pre-populated with
+ a database-wide dictionary.
+
+ 3. Emit whitespace. The parameter is a number of blank spaces to insert.
+
+ Strings, either included directly or copied from the memo, may appear in
+ multiple forms:
+
+ - In the default context, a string can be interpreted as either character
+ data or as an element. Character data strings are prefixed with 0x01,
+ while element data strings are prefixed with 0x02 plus their attribute
+ count. After an element with attributes, the stream alternates between
+ attribute name and attribute value contexts.
+
+ - In attribute name/value contexts, no string prefix is used.
+
+ This method of encoding actually makes an element's arity part of its name,
+ as far as our compression is concerned.
+ """
+ def __init__(self, dictionary=()):
+ self.dictionary = dictionary
+
+ def startDocument(self):
+ self.output = []
+ self.memo = {}
+ self.state = (None,)
+ for item in self.dictionary:
+ self.memoize(item)
+
+ def memoize(self, string):
+ self.memo[string] = len(self.memo)
+
+ def startElement(self, name, attrs):
+ self.flush()
+
+ attrNames = attrs.getNames()
+ self.encode_element(name, len(attrNames))
+
+ for attr in attrNames:
+ self.encode_string(attr)
+ self.encode_string(attrs[attr])
+
+ def endElement(self, name):
+ if self.state[0] == 'END':
+ self.state = ('END', self.state[1] + 1)
+ else:
+ self.flush()
+ self.state = ('END', 1)
+
+ def characters(self, content):
+ if self.state[0] == 'CHAR':
+ self.state = ('CHAR', self.state[1] + content)
+ else:
+ self.flush()
+ self.state = ('CHAR', content)
+
+ def flush(self):
+ if self.state[0] == 'CHAR':
+ self.encode_chardata(self.state[1])
+ elif self.state[0] == 'END':
+ self.encode_end(self.state[1])
+ self.state = (None,)
+
+ def encode_chardata(self, data):
+ # Split the character data into leading whitespace,
+ # a normal string, and trailing whitespace
+
+ dlen = len(data)
+ s = data.lstrip()
+ slen = len(s)
+
+ if slen != dlen:
+ self.encode_whitespace(dlen - slen)
+ data = s
+ dlen = slen
+
+ dlen = len(data)
+ s = data.rstrip()
+ slen = len(s)
+
+ if s:
+ self.encode_string(unichr(1) + s)
+
+ if slen != dlen:
+ self.encode_whitespace(dlen - slen)
+
+ def encode_element(self, name, params):
+ self.encode_string(unichr(params + 2) + name)
+
+ def encode_string(self, data):
+ if data in self.memo:
+ self.encode_memoized(self.memo[data])
+ return
+ self.memoize(data)
+
+ self.output.append(unichr((len(data) << 2) | 1))
+ self.output.append(data)
+
+ def encode_end(self, count):
+ self.output.append(unichr(count << 2))
+
+ def encode_memoized(self, index):
+ self.output.append(unichr((index << 2) | 2))
+
+ def encode_whitespace(self, count):
+ self.output.append(unichr((count << 2) | 3))
+
+ def getvalue(self):
+ return u''.join(self.output).encode('utf-8')
+
+def SAXDecoder(encoded, dictionary):
+ """Decode the stream produced by SAXEncoder, returning a minidom tree.
+ This function is a bit hard to follow and it relies on minidom
+ internals- unfortunately it's a big speed bottleneck, so
+ any extra performance is worth a little obfuscation here.
+ """
+ dom = xml.dom.minidom
+ memo = list(dictionary)
+ stack = [dom.Document()]
+ attrs = None
+ top = stack[-1]
+ m = unicode(encoded, 'utf8')
+ mlen = len(m)
+ i = 0
+
+ while i < mlen:
+ op = ord(m[i])
+ type = op & 3
+ param = op >> 2
+
+ # Decode a memoized string
+ if type == 2:
+ item = memo[param]
+ i += 1
+
+ # Decode and memoize a literal string
+ elif type == 1:
+ i += 1
+ item = m[i:i+param]
+ i += param
+ memo.append(item)
+
+ # End elements
+ elif type == 0:
+ del stack[-param:]
+ top = stack[-1]
+ i += 1
+ continue
+
+ # Whitespace
+ elif type == 3:
+ node = dom.Text()
+ node.data = " " * param
+
+ # _append_child(top, node)
+ childNodes = top.childNodes
+ if childNodes:
+ last = childNodes[-1]
+ node.__dict__["previousSibling"] = last
+ last.__dict__["nextSibling"] = node
+ childNodes.append(node)
+ node.__dict__["parentNode"] = top
+ node.__dict__["ownerDocument"] = stack[0]
+
+ i += 1
+ continue
+
+ # Processing this string in the default context
+ if attrs is None:
+ op = ord(item[0])
+
+ # Character data
+ if op == 1:
+ node = dom.Text()
+ node.data = item[1:]
+
+ # _append_child(top, node)
+ childNodes = top.childNodes
+ if childNodes:
+ last = childNodes[-1]
+ node.__dict__["previousSibling"] = last
+ last.__dict__["nextSibling"] = node
+ childNodes.append(node)
+ node.__dict__["parentNode"] = top
+ node.__dict__["ownerDocument"] = stack[0]
+
+ # Element with no attributes (fast path)
+ elif op == 2:
+ node = dom.Element(item[1:])
+
+ # _append_child(top, node)
+ childNodes = top.childNodes
+ if childNodes:
+ last = childNodes[-1]
+ node.__dict__["previousSibling"] = last
+ last.__dict__["nextSibling"] = node
+ childNodes.append(node)
+ node.__dict__["parentNode"] = top
+ node.__dict__["ownerDocument"] = stack[0]
+
+ stack.append(node)
+ top = node
+
+ # Element with attributes, general
+ else:
+ attrRemaining = op - 2
+ attrKey = None
+ attrs = dom.Element(item[1:])
+
+ # Attribute context
+ else:
+
+ # Attribute key
+ if attrKey is None:
+ attrKey = dom.Attr(item)
+
+ # Attribute value
+ else:
+ d = attrKey.__dict__
+ d["value"] = d["nodeValue"] = item
+ d["ownerDocument"] = stack[0]
+
+ # _set_attribute_node(attrs, attrKey)
+ attrs._attrs[attrKey.name] = attrKey
+ attrKey.__dict__['ownerElement'] = attrs
+
+ attrRemaining -= 1
+ attrKey = None
+
+ # Finished an element with attributes?
+ if not attrRemaining:
+ # _append_child(top, attrs)
+ childNodes = top.childNodes
+ if childNodes:
+ last = childNodes[-1]
+ node.__dict__["previousSibling"] = last
+ last.__dict__["nextSibling"] = node
+ childNodes.append(attrs)
+ attrs.__dict__["parentNode"] = top
+ attrs.__dict__["ownerDocument"] = stack[0]
+
+ stack.append(attrs)
+ top = attrs
+ attrs = None
+
+ return stack[0]
+
+
+class MessageBuffer:
+ """This object represents on-disk storage for a stats target's
+ recent messages. It is based on two ring buffers: one with
+ message IDs, and one with message content.
+
+ Message IDs are implemented as seek offsets into the content
+ buffer. The IDs increase monotonically, are unique within
+ this particular MessageBuffer, and are validated on access.
+
+ Message content is encoded using a simple method of representing
+ SAX event streams as Unicode strings. This gives about the same
+ performance as gzip on small messages, but worse performance on
+ large messages. The huge advantage of this SAX format, though,
+ is that message buffers as a whole compress much better. This
+ is important for backups.
+
+ Format:
+ - Fixed-size header
+ - Magic number
+ - Dictionary size
+ - Index size
+ - Content size
+ - Tail pointer
+ - Dictionary, NUL-separated UTF-8
+ - Index ringbuffer
+ - Content ringbuffer
+ """
+ magic = "CIA Message Buffer v1\r\n\x00"
+ headerFmt = ">24sIIII"
+ headerSize = struct.calcsize(headerFmt)
+
+ #
+ # Defaults only.
+ # When opening existing databases, these are read from the header.
+ #
+ indexSize = 4096
+ contentSize = 256 * 1024
+
+ def __init__(self, path, filename="_msg", file=None):
+ self.dirPath = path
+ self.file = file
+ self.filePath = os.path.join(path, filename)
+ self._initialized = False
+
+ def close(self):
+ if self._initialized:
+ self.file.close()
+ self._initialized = False
+
+ def _initRingBuffers(self):
+ # Place our two ring buffers in the file. This should be called
+ # as soon as the sizes in the header are all known.
+ self.indexRing = CircularFile(self.file, self.indexSize,
+ self.headerSize + self.dictionarySize)
+ self.contentRing = CircularFile(self.file, self.contentSize,
+ self.headerSize + self.dictionarySize +
+ self.indexSize)
+
+ def _init(self, msg=None):
+ """Either read header data from an existing database, or set
+ up the header and dictionary on a new database.
+
+ If the database doesn't exist and 'msg' is not None, this
+ message will be used to help build the dictionary.
+
+ If the 'msg' is None, this will never set up a new database.
+ If the database is not already initialized, it will return
+ False and change nothing.
+ """
+ if not self.file:
+ self.file = os.fdopen(os.open(self.filePath, os.O_RDWR | os.O_CREAT, 0666), 'w+')
+ header = self.file.read(self.headerSize)
+
+ if header:
+ # Unpack the header
+ try:
+ (magic, self.dictionarySize, self.indexSize,
+ self.contentSize, self.tail) = struct.unpack(self.headerFmt, header)
+ except struct.error:
+ magic = None
+ if magic != self.magic:
+ raise ValueError("File header is incorrect")
+
+ # Unpack the original dictionary
+ self.dictionary = unicode(self.file.read(self.dictionarySize), 'utf8').split('\x00')
+ self._initRingBuffers()
+
+ elif msg is None:
+ return False
+
+ else:
+ self.dictionary = self._createDictionary(msg)
+
+ # Measure the encoded dictionary and set up critical sizes
+ encodedDict = u'\x00'.join(self.dictionary).encode('utf8')
+ self.dictionarySize = len(encodedDict)
+ self._initRingBuffers()
+
+ # Zero out the whole index, point to the beginning of it
+ self.tail = 0
+ self.indexRing.write('\x00' * self.indexSize, 0)
+
+ # Initialize the dictionary
+ self.file.seek(self.headerSize)
+ self.file.write(encodedDict)
+
+ self._commitHeader()
+
+ self._initialized = True
+ return True
+
+ def _commitHeader(self):
+ self.file.seek(0)
+ self.file.write(struct.pack(
+ self.headerFmt, self.magic, self.dictionarySize,
+ self.indexSize, self.contentSize, self.tail))
+ self.file.flush()
+
+ def _createDictionary(self, msg):
+ """Decide on a dictionary, using the first message as a hint"""
+ #
+ # Start out with a default dictionary that contains some
+ # general-purpose strings, so that even if msg is some
+ # pathological case we still have a somewhat usable
+ # dictionary.
+ #
+ encoder = SAXEncoder((
+ u'\x01name', u'\x03header', u'\x02file', u'\x01Received',
+ u'\x02source', u'\x02project', u'\x02name', u'\x02generator',
+ u'\x02body', u'\x02timestamp', u'\x02message', u'\x02commit',
+ u'\x02author', u'\x02log', u'\x02files', u'\x02version',
+ u'\x03file', u'action', u'From', u'Date', u'\x02mailHeaders',
+ u'Message-Id', u'\x02module', u'modify', u'\x02revision',
+ u'\x02url', u'\x02diffLines', u'\x01uri', u'\x02branch',
+ u'\x04file', u'\x01add',
+ ))
+
+ # Feed in the first message, to populate the memo with strings
+ # that may be common in this particular database.
+ xml.sax.parseString(msg, encoder)
+
+ #
+ # Put shorter symbols first, since they're
+ # more likely to be the common ones. Completely
+ # discard anything really long, since it's likely
+ # to be just a log message or something equally
+ # useless.
+ #
+ dict = [ key for key in encoder.memo if len(key) < 64 ]
+ dict.sort(lambda a,b: cmp(len(a), len(b)))
+
+ # Put an upper limit on the number of dictionary symbols,
+ # to avoid really pathological messages using up lots of
+ # disk space in the file header.
+ return dict[:128]
+
+ def push(self, msg):
+ """Append a new message to the buffer, returning its ID"""
+ if not self._initialized:
+ # Create the parent directory if necessary
+ if not os.path.isdir(self.dirPath):
+ os.makedirs(self.dirPath)
+ self._init(msg)
+
+ # Parse the message into a compressed SAX event stream
+ encoder = SAXEncoder(self.dictionary)
+ xml.sax.parseString(msg, encoder)
+ zmsg = encoder.getvalue()
+
+ # Read the tail offset. This is our new message ID
+ msgId = struct.unpack(">I", self.indexRing.read(4, self.tail))[0]
+
+ # Every message is preceeded with a marker that identifies its
+ # full message ID (to identify a location uniquely across wrap-arounds).
+ # The zeroes on either side are for framing- messages never include
+ # zeroes, so this ensures that we're actually reading a marker.
+ content = struct.pack(">BIIB", 0, msgId, len(zmsg), 0) + zmsg
+ self.contentRing.write(content, msgId)
+
+ # Update the index with the next message ID
+ self.tail += 4
+ nextId = msgId + len(content)
+ self.indexRing.write(struct.pack(">I", nextId), self.tail)
+ self._commitHeader()
+ return msgId
+
+ def getMessageById(self, msgId):
+ """Retrieve a particular message, by ID. Returns None if the
+ message doesn't exist or has been overwritten.
+ """
+ if not self._initialized:
+ # This shouldn't ever create the file
+ if not os.path.isfile(self.filePath):
+ return
+ if not self._init():
+ return
+
+ try:
+ zero1, checkId, size, zero2 = struct.unpack(
+ ">BIIB", self.contentRing.read(10, msgId))
+ except struct.error:
+ return
+ if checkId == msgId and zero1 == 0 and zero2 == 0:
+ return SAXDecoder(self.contentRing.read(size), self.dictionary)
+
+ def getLatest(self, limit=None):
+ """Yield up to 'limit' of the most recent messages, in
+ the same order they were added. Yields (id, msg) tuples.
+ """
+ if not self._initialized:
+ # This shouldn't ever create the file
+ if not os.path.isfile(self.filePath):
+ return
+ if not self._init():
+ return
+ if limit is None:
+ limit = self.indexSize // 4
+
+ tail = self.tail
+ head = max(0, tail - min(limit * 4, self.indexSize))
+ index = self.indexRing.read(tail - head, head)
+ for i in xrange(0, len(index), 4):
+ msgId = struct.unpack(">I", index[i:i+4])[0]
+ msg = self.getMessageById(msgId)
+ if msg is not None:
+ yield (msgId, msg)
+
+
+class MessageArchive:
+ """Whereas a MessageBuffer stores a limited number of recent messages,
+ and uses a built-in index, a MessageArchive is designed to store an
+ unlimited number of messages using an external index.
+
+ MessageArchives are not pre-sorted by stats target. All incoming
+ messages are sent to a single MessageArchive, which is separated
+ into files by day. (Note that these dates are based on the *current*
+ date, not the message's timestamp! We don't want to allow backdated
+ commit messages to modify data which may have already been backed
+ up.)
+
+ MessageArchives use a static dictionary. To avoid the hassle of
+ atomically generating a file header, dictionaries are specified
+ per-message. In the future this may be used to smoothly upgrade
+ the dictionary when new elements are added to the XML schema.
+
+ The MessageArchive file has no header. Each message is represented by:
+
+ 1. One NUL byte, indicating the beginning of the message. This
+ is used as a flag character, since messages may not contain
+ an internal NUL.
+
+ 2. The length of this message, including all headers, as a
+ 32-bit big endian integer.
+
+ 3. The message version, as a single byte. This identifies
+ the dictionary used to encode the message, and it can indicate
+ the presence of extra fields in the message header.
+
+ There is no unique message identifier, since it isn't worth the
+ effort to generate one atomically. The file name and file offset
+ will become a unique identifier the moment the message is appended
+ to disk.
+ """
+ LATEST_VERSION = 1
+
+ # All versions use the same header currently
+ header_format = ">BIB"
+ header_size = struct.calcsize(header_format)
+
+ # You may define new versions, but never change the existing dicts!
+ dictionaries = {
+ 1: (u'\x01name', u'\x03header', u'\x02file', u'\x01Received',
+ u'\x02source', u'\x02project', u'\x02name', u'\x02generator',
+ u'\x02body', u'\x02timestamp', u'\x02message', u'\x02commit',
+ u'\x02author', u'\x02log', u'\x02files', u'\x02version',
+ u'\x03file', u'action', u'From', u'Date', u'\x02mailHeaders',
+ u'Message-Id', u'\x02module', u'modify', u'\x02revision',
+ u'\x02url', u'\x02diffLines', u'\x01uri', u'\x02branch',
+ u'\x04file', u'\x01add'),
+ }
+
+ def __init__(self, path):
+ self._path = path
+ self._file = None
+ self._date = None
+
+ def _setDate(self, date):
+ """Change days, opening a new log file."""
+ if date == self._date:
+ return
+ self._date = date
+
+ # makedirs() is racy- better for us to just call mkdir()
+ # multiple times and ignore failures.
+ p = self._path
+ for segment in ("%04d" % date.year,
+ "%04d-%02d" % (date.year, date.month),
+ "%04d-%02d-%02d.bsax" % (date.year, date.month, date.day)):
+ try:
+ os.mkdir(p)
+ except OSError:
+ pass
+ p = os.path.join(p, segment)
+
+ self._file = open(p, 'ab')
+
+ def push(self, msg):
+ """Append a new message to the buffer, given a UTf-8 byte string.
+ Returns None, as we can't efficiently return the message ID just yet.
+ """
+ self._setDate(datetime.datetime.now().date())
+
+ # Parse the message into a compressed SAX event stream
+ version = self.LATEST_VERSION
+ encoder = SAXEncoder(self.dictionaries[version])
+ xml.sax.parseString(msg, encoder)
+ zmsg = encoder.getvalue()
+
+ # Pack the message header
+ header = struct.pack(self.header_format, 0,
+ self.header_size + len(zmsg),
+ version)
+
+ # Atomically write to the file
+ os.write(self._file.fileno(), header + zmsg)
+
+ def deliver(self, msg):
+ """Deliver a Message instance to the archive. This is silly, because
+ we have to convert the Message (a DOM) right back to UTF-8 text
+ just so we can re-serialize it with SAX. This is a stopgap measure
+ until the message filtering architecture is redesigned.
+ """
+ self.push(unicode(msg).encode('utf-8'))
+
+ def readNextFromFile(self, f):
+ """Read a single message from a file-like object, advancing the file
+ pointer to the next message. Returns None on EOF.
+ """
+ header = f.read(self.header_size)
+ if not header:
+ return None
+
+ zero, size, version = struct.unpack(self.header_format, header)
+ assert zero == 0
+ return SAXDecoder(f.read(size - self.header_size), self.dictionaries[version])
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Stats/Messages.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Stats/Messages.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Stats/Metadata.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Stats/Metadata.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Stats/Metadata.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,194 @@
+""" LibCIA.Stats.Metadata
+
+A system for storing and retrieving metadata associated with a
+stats target. This includes an AbstractStringCache subclass
+used to cache thumbnails generated with PIL.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.internet import defer
+from LibCIA import Database, Cache
+import time
+from cStringIO import StringIO
+import Image
+
+
+class Metadata:
+ """An abstraction for the metadata that may be stored for any stats target.
+ Metadata objects consist of a name, a MIME type, and a value. The value
+ can be any binary or text object stored as a string.
+ """
+ def __init__(self, target):
+ self.target = target
+ self._cache = None
+
+ def get(self, name, default=None):
+ """Return a Deferred that results in the (value, type) tuple for the the
+ given metadata key, or 'default' if a result can't be found.
+ """
+ return Database.pool.runInteraction(self._get, name, default)
+
+ def getValue(self, name, default=None, typePrefix='text/'):
+ """Like get(), but only returns the value. Ensures the MIME type
+ begins with typePrefix. If it doesn't, a TypeError is raised.
+ """
+ return Database.pool.runInteraction(self._getValue, name, default, typePrefix)
+
+ def getMTime(self, name, default=None):
+ """Returns a Deferred that eventually results in the modification
+ time for the specified metadata object, in seconds since the epoch.
+ This returns the specified default if the item doesn't exist. Note that
+ None might be returned if the object was created before mtime support
+ was included in the metadata table.
+ """
+ # This is not supported by the new stats database, and it isn't
+ # needed now that images are stored separately.
+ raise NotImplementedError
+
+ def set(self, name, value, mimeType='text/plain'):
+ """Set a metadata key, creating it if it doesn't exist"""
+ # Writing to metadata from the LibCIA codebase is deprecated
+ raise NotImplementedError
+
+ def keys(self):
+ """Return (via a Deferred) a list of all valid metadata key names"""
+ return Database.pool.runInteraction(self._keys)
+
+ def dict(self):
+ """Return (via a Deferred) a mapping from names to (value, type) tuples"""
+ return Database.pool.runInteraction(self._dict)
+
+ def clear(self):
+ """Delete all metadata for this target. Returns a Deferred"""
+ # Writing to metadata from the LibCIA codebase is deprecated
+ raise NotImplementedError
+
+ def remove(self, name):
+ """Remove one metadata key, with the given name"""
+ # Writing to metadata from the LibCIA codebase is deprecated
+ raise NotImplementedError
+
+ def has_key(self, name):
+ """Returs True (via a Deferred) if the given key name exists for this target"""
+ return Database.pool.runInteraction(self._has_key, name)
+
+ def _update_cache(self, cursor):
+ """Retrieve all metadata keys for this target, populating self._cache."""
+
+ # XXX: This only works for text keys. Images can't be fully represented
+ # in the old system, so we're ignoring them until the transition is complete.
+ keys = ('title', 'subtitle', 'url', 'description', 'links-filter', 'related-filter')
+
+ # Our new column names use underscores instead of dashes
+ columns = [key.replace('-', '_') for key in keys]
+
+ cursor.execute("SELECT %s FROM stats_statstarget WHERE path = %s" % (
+ ', '.join(columns),
+ Database.quote(self.target.path, 'varchar')))
+
+ row = cursor.fetchone()
+ self._cache = {}
+ if row:
+ for i, key in enumerate(keys):
+ self._cache[key] = row[i]
+
+ def _has_key(self, cursor, name):
+ """Database interaction implemented has_key()"""
+ if self._cache is None:
+ self._update_cache(cursor)
+ return self._cache.get(name) is not None
+
+ def _get(self, cursor, name, default):
+ """Database interaction to return the value and type for a particular key"""
+ if self._cache is None:
+ self._update_cache(cursor)
+ v = self._cache.get(name)
+ if v is None:
+ return default
+ else:
+ return v, 'text/plain'
+
+ def _getValue(self, cursor, name, default, typePrefix):
+ # XXX: typePrefrix is being ignored. This is an artifact of the
+ # transition from old metadata system to new...
+ if self._cache is None:
+ self._update_cache(cursor)
+ v = self._cache.get(name)
+ if v is None:
+ return default
+ else:
+ return v
+
+ def _dict(self, cursor):
+ """Database interaction to return to implement dict()"""
+ if self._cache is None:
+ self._update_cache(cursor)
+ d = {}
+ for k, v in self._cache.items():
+ d[k] = (v, 'text/plain')
+ return d
+
+ def _keys(self, cursor):
+ """Database interaction implementing keys()"""
+ if self._cache is None:
+ self._update_cache(cursor)
+ return self._cache.keys()
+
+
+class MetadataThumbnailCache(Cache.AbstractFileCache):
+ """A cache for thumbnails of image metadata, generated using PIL.
+ The cache is keyed by a target, metadata key, maximum
+ thumbnail size, and metadata key modification time.
+ """
+ # The mime type our generated thumbnails should be in
+ mimeType = 'image/png'
+
+ def miss(self, target, metadataKey, size, mtime=None):
+ # Get the metadata value first
+ result = defer.Deferred()
+ target.metadata.getValue(metadataKey, typePrefix='image/').addCallback(
+ self.makeThumbnail, size, result).addErrback(result.errback)
+ return result
+
+ def makeThumbnail(self, imageData, size, result):
+ i = Image.open(StringIO(imageData))
+
+ # Our thumbnails look much better if we paste the image into
+ # a larger transparent one first with a margin about equal to one
+ # pixel in our final thumbnail size. This smoothly blends
+ # the edge of the image to transparent rather than chopping
+ # off a fraction of a pixel. It looks, from experimentation,
+ # like this margin is only necessary on the bottom and right
+ # sides of the image.
+ margins = (i.size[0] // size[0] + 1,
+ i.size[1] // size[1] + 1)
+ bg = Image.new("RGBA",
+ (i.size[0] + margins[0],
+ i.size[1] + margins[1]),
+ (255, 255, 255, 0))
+ bg.paste(i, (0,0))
+
+ bg.thumbnail(size, Image.ANTIALIAS)
+
+ tempFilename = self.getTempFilename()
+ bg.save(tempFilename, 'PNG')
+ result.callback(tempFilename)
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Stats/Metadata.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Stats/Metadata.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Stats/Target.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Stats/Target.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Stats/Target.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,472 @@
+""" LibCIA.Stats.Target
+
+Implements the core objects for representing and
+interacting with stats targets.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import os, string
+from twisted.internet import defer
+from twisted.python import log
+from LibCIA import Ruleset, Database, TimeUtil, Files
+import time, posixpath, sys, cPickle, random
+from LibCIA.Stats.Metadata import Metadata
+from LibCIA.Stats.Messages import MessageBuffer
+
+
+class StatsTarget(object):
+ """Encapsulates all the stats-logging features used for one particular
+ target. This can be one project, one class of messages, etc.
+ Every StatsTarget is identified by a UNIX-style pathname.
+ The root stats target's path is the empty string.
+
+ This object doesn't store any of the actual data, it's just a way to
+ access the persistent data stored in our global SQL database.
+ """
+ def __init__(self, path=''):
+ self.setPath(path)
+ self._messages = None
+ self._counters = None
+ self._metadata = None
+
+ def _getMessages(self):
+ if self._messages is None:
+ self._messages = MessageBuffer(self.getDiskPath())
+ return self._messages
+ messages = property(_getMessages)
+
+ def _getCounters(self):
+ if self._counters is None:
+ self._counters = Counters(self)
+ return self._counters
+ counters = property(_getCounters)
+
+ def _getMetadata(self):
+ if self._metadata is None:
+ self._metadata = Metadata(self)
+ return self._metadata
+ metadata = property(_getMetadata)
+
+ def setPath(self, path):
+ # Remove leading and trailing slashes, remove duplicate
+ # slashes, process '.' and '..' directories. We don't
+ # allow paths beginning with '.' or '_'.
+ self.pathSegments = []
+ for segment in path.split('/'):
+ if segment == '..':
+ if self.pathSegments:
+ del self.pathSegments[-1]
+ elif segment in ('.', ''):
+ pass
+ elif segment[0] in ('.', '_'):
+ raise ValueError("Stats path segment %r begins with a reserved character"
+ % segment)
+ else:
+ self.pathSegments.append(segment)
+ self.path = '/'.join(self.pathSegments)
+
+ # Our database uses VARCHAR(128), make sure this fits
+ if len(self.path) > 128:
+ raise Ruleset.InvalidURIException("Stats paths are currently limited to 128 characters")
+
+ # Our name is the last path segment, or None if we're the root
+ if self.pathSegments:
+ self.name = self.pathSegments[-1]
+ else:
+ self.name = None
+
+ def getDiskPath(self):
+ """Every target gets a directory on disk. This returns it, without any
+ guarantee that it exists yet.
+ """
+ return Files.tryGetDir(Files.dbDir, 'stats', *map(string.lower, self.pathSegments))
+
+ def deliver(self, message=None):
+ """An event has occurred which should be logged by this stats target"""
+ if message:
+ self.messages.push(unicode(message).encode('utf-8'))
+
+ # XXX:
+ # We want to close the file now, even if this StatsTarget instance lingers
+ # for a while due to references from Deferreds. I've seen StatsTargets stick
+ # around for a seemingly infinite amount of time- probably a side effect of
+ # all the circular references between here and Messages/Counters :(
+ self.messages.close()
+
+ self.counters.increment()
+
+ # XXX: Disable subscriptions for speed. Nobody uses them anyway.
+ # SubscriptionDelivery(self).notify('messages')
+
+ def child(self, name):
+ """Return the StatsTarget for the given sub-target name under this one"""
+ return StatsTarget(posixpath.join(self.path, name))
+
+ def parent(self):
+ """Return the parent StatsTarget of this one, or None if we're the root"""
+ if self.path:
+ return self.child('..')
+
+ def catalog(self):
+ """Return a list of StatsTargets instances representing all children of this target"""
+ return Database.pool.runInteraction(self._catalog)
+
+ def getTitle(self):
+ """Return the human-readable title of this stats target. In
+ decreasing order of preference, this is:
+ - our 'title' metadata key
+ - self.name, the last segment of our path
+ - 'Stats'
+ The result will always be a Deferred.
+ """
+ result = defer.Deferred()
+ self.metadata.getValue('title').addCallback(self._getTitle, result).addErrback(result.errback)
+ return result
+
+ def _getTitle(self, metadataTitle, result):
+ if metadataTitle is not None:
+ result.callback(metadataTitle)
+ elif self.name:
+ result.callback(self.name)
+ else:
+ result.callback('Stats')
+
+ def getMTime(self):
+ """Get the modification time of this stats target, in seconds since the epoch.
+ Currently this ignores metadata changes and reports the lastEventTime of the
+ 'forever' event counter. The result will be delivered via a Deferred.
+ """
+ result = defer.Deferred()
+ self.counters.getCounter('forever').addCallback(
+ self._getMTime, result
+ ).addErrback(result.errback)
+ return result
+
+ def _getMTime(self, counter, result):
+ if counter:
+ result.callback(counter['lastEventTime'])
+ else:
+ result.callback(None)
+
+ def clear(self):
+ """Delete everything associated with this stats target. Returns a Deferred
+ indicating the completion of this operation.
+ """
+ # Delete the item in stats_target- the other tables will be
+ # deleted due to cascading foreign keys
+ return Database.pool.runOperation("DELETE FROM stats_catalog WHERE target_path = %s" %
+ Database.quote(self.path, 'varchar'))
+
+ def __repr__(self):
+ return "<StatsTarget at %r>" % self.path
+
+ def _create(self, cursor):
+ """Internal function to create a new stats target, meant to be run from
+ inside a database interaction. This is actually a recursive operation
+ that tries to create parent stats targets if necessary.
+
+ NOTE: this -must- ignore duplicate keys to avoid a race condition in which
+ one thread, in _autoCreateTargetFor, decides to create a new target
+ but before that target is fully created another thread also decides
+ it needs a new target.
+ """
+ parent = self.parent()
+ if parent:
+ # If we have a parent, we have to worry about creating it
+ # if it doesn't exist and generating the proper parent path.
+ parent._autoCreateTargetFor(cursor, cursor.execute,
+ "INSERT IGNORE INTO stats_catalog (parent_path, target_path) VALUES(%s, %s)" %
+ (Database.quote(parent.path, 'varchar'),
+ Database.quote(self.path, 'varchar')))
+ else:
+ # This is the root node. We still need to insert a parent to keep the
+ # table consistent, but our parent in this case is NULL.
+ cursor.execute("INSERT IGNORE INTO stats_catalog (target_path) VALUES(%s)" %
+ Database.quote(self.path, 'varchar'))
+
+ def _autoCreateTargetFor(self, cursor, func, *args, **kwargs):
+ """Run the given function. If an exception occurs that looks like a violated
+ foreign key constraint, add our path to the database and try
+ again (without attempting to catch any exceptions).
+ This is fast way to create stats targets that don't exist without
+ a noticeable performance penalty when executing operations on
+ existing stats targets.
+
+ NOTE: This is meant to be run inside a database interaction, hence
+ a cursor is required. This cursor will be used to
+ create the new stats target if one is required.
+ """
+ try:
+ func(*args, **kwargs)
+ except:
+ # Cheesy way to detect foreign key errors without being too DBMS-specific
+ if str(sys.exc_info()[1]).find("foreign key") >= 0:
+ self._create(cursor)
+ func(*args, **kwargs)
+ else:
+ raise
+
+ def _catalog(self, cursor):
+ """Database interaction representing the internals of catalog()"""
+ cursor.execute("SELECT target_path FROM stats_catalog WHERE parent_path = %s" %
+ Database.quote(self.path, 'varchar'))
+ results = []
+ while True:
+ row = cursor.fetchone()
+ if row is None:
+ break
+ results.append(StatsTarget(row[0]))
+ return results
+
+
+class SubscriptionDelivery:
+ """The object responsible for actually notifiying entities that
+ have subscribed to a stats target.
+ """
+ def __init__(self, target):
+ self.target = target
+
+ def notify(self, scope):
+ """Notify all subscribers to this stats target of a change in 'scope'"""
+ # Get a list of applicable triggers from the database
+ Database.pool.runQuery("SELECT id, `trigger` FROM stats_subscriptions "
+ "WHERE target_path = %s "
+ "AND (scope is NULL or scope = %s)" %
+ (Database.quote(self.target.path, 'varchar'),
+ Database.quote(scope, 'varchar'))).addCallback(self.runTriggers)
+
+ def runTriggers(self, rows):
+ """After retrieving a list of applicable triggers, this calls them.
+ 'rows' should be a sequence of (id, trigger) tuples.
+ """
+ if rows:
+ log.msg("Notifying %d subscribers for %r" % (len(rows), self.target))
+ for id, trigger in rows:
+ f, args, kwargs = cPickle.loads(trigger)
+ defer.maybeDeferred(f, *args, **kwargs).addCallback(
+ self.triggerSuccess, id).addErrback(
+ self.triggerFailure, id)
+
+ def triggerSuccess(self, result, id):
+ """Record a successful trigger run for the given subscription id"""
+ # Zero the consecutive failure count
+ Database.pool.runOperation("UPDATE stats_subscriptions SET failures = 0 WHERE id = %s" %
+ Database.quote(id, 'bigint'))
+
+ def triggerFailure(self, failure, id):
+ """Record an unsuccessful trigger run for the given subscription id"""
+ Database.pool.runInteraction(self._triggerFailure, failure, id)
+
+ def _triggerFailure(self, cursor, failure, id, maxFailures=3):
+ # Increment the consecutive failure count
+ log.msg("Failed to notify subscriber %d for %r: %r" % (id, self.target, failure))
+ cursor.execute("UPDATE stats_subscriptions SET failures = failures + 1 WHERE id = %s" %
+ Database.quote(id, 'bigint'))
+
+ # Cancel the subscription if we've had too many failures
+ cursor.execute("DELETE FROM stats_subscriptions WHERE id = %s AND failures > %s" %
+ (Database.quote(id, 'bigint'),
+ Database.quote(maxFailures, 'int')))
+ if cursor.rowcount:
+ log.msg("Unsubscribing subscriber %d for %r, more than %d consecutive failures" %
+ (id, self.target, maxFailures))
+
+
+class Counters:
+ """A set of counters which are used together to track how many
+ events occur and how frequently in each of several time intervals.
+ """
+ def __init__(self, target):
+ self.target = target
+ self.cache = None
+
+ def increment(self):
+ """Increment all applicable counters, signaling the arrival of a new event"""
+ # Automatically create the stats target if it doesn't exist
+ return Database.pool.runInteraction(self._incrementWrapper)
+
+ def _incrementWrapper(self, cursor):
+ """Database interaction implementing increment(). Ensures
+ the stats target exists while calling _increment().
+ """
+ self.target._autoCreateTargetFor(cursor, self._increment, cursor)
+
+ def _createCounter(self, cursor, name):
+ """Internal function to create one blank counter if it doesn't exist."""
+ try:
+ cursor.execute("INSERT INTO stats_counters (target_path, name) VALUES(%s, %s)" %
+ (Database.quote(self.target.path, 'varchar'),
+ Database.quote(name, 'varchar')))
+ except:
+ # Ignore duplicate key errors
+ if str(sys.exc_info()[1]).find("duplicate key") < 0:
+ raise
+
+ def _increment(self, cursor):
+ """Internal function, run within a database interaction, that ensures
+ all required counters exist then updates them all.
+ """
+ self._incrementCounter(cursor, 'forever')
+ self._incrementCounter(cursor, 'today')
+ self._incrementCounter(cursor, 'thisWeek')
+ self._incrementCounter(cursor, 'thisMonth')
+
+ def _incrementCounter(self, cursor, name):
+ """Increment one counter, creating it if necessary"""
+ now = int(time.time())
+ # Invalidate the cache for this counter if we have one
+ self.cache = None
+
+ # Insert a default value, which will be ignored if the counter already exists
+ cursor.execute("INSERT IGNORE INTO stats_counters (target_path, name, first_time) VALUES(%s, %s, %s)" %
+ (Database.quote(self.target.path, 'varchar'),
+ Database.quote(name, 'varchar'),
+ Database.quote(now, 'bigint')))
+
+ # Increment the counter and update its timestamp
+ cursor.execute("UPDATE stats_counters SET "
+ "event_count = event_count + 1,"
+ "last_time = %s "
+ "WHERE target_path = %s AND name = %s" %
+ (Database.quote(now, 'bigint'),
+ Database.quote(self.target.path, 'varchar'),
+ Database.quote(name, 'varchar')))
+
+ def getCounter(self, name):
+ """Return a Deferred that eventually results in a dictionary,
+ including the following keys:
+
+ firstEventTime : The time, in UTC seconds since the epoch, when the first event occurred
+ lastEventTime : The time when the most recent event occurred
+ eventCount : The number of events that have occurred
+ """
+ if self.cache is not None:
+ result = defer.Deferred()
+ result.callback(self.cache.get(name))
+ return result
+ else:
+ return Database.pool.runInteraction(self._getCounter, name)
+
+ def dict(self):
+ """Return a Deferred that eventually results in a dictionary mapping
+ counter name to the dictionary that would be returned by getCounter.
+ """
+ if self.cache is not None:
+ # This is the cache itself
+ result = defer.Deferred()
+ result.callback(self.cache)
+ return result
+ else:
+ return Database.pool.runInteraction(self._dict)
+
+ return Database.pool.runInteraction(self._dict)
+
+ def _updateCache(self, cursor):
+ """Database interaction to update our counter cache"""
+ cursor.execute("SELECT name, first_time, last_time, event_count FROM stats_counters WHERE"
+ " target_path = %s" %
+ Database.quote(self.target.path, 'varchar'))
+ results = {}
+ while True:
+ row = cursor.fetchone()
+ if row is None:
+ break
+ results[row[0]] = {
+ 'firstEventTime': row[1],
+ 'lastEventTime': row[2],
+ 'eventCount': row[3],
+ }
+ self.cache = results
+
+ def _getCounter(self, cursor, name):
+ """Database interaction implementing getCounter"""
+ self._updateCache(cursor)
+ return self.cache.get(name)
+
+ def _dict(self, cursor):
+ """Database interaction implementing _getCounterDict"""
+ self._updateCache(cursor)
+ return self.cache
+
+ def clear(self):
+ """Delete all counters for this target. Returns a Deferred"""
+ self.cache = {}
+ return Database.pool.runOperation("DELETE FROM stats_counters WHERE target_path = %s" %
+ Database.quote(self.target.path, 'varchar'))
+
+
+class Maintenance:
+ """This class performs periodic maintenance of the stats database, including
+ counter rollover and removing old messages.
+ """
+ def __init__(self):
+ self.targetQueue = []
+
+ def run(self):
+ """Performs one stats maintenance cycle, returning a Deferred that
+ yields None when the maintenance is complete.
+ """
+ return Database.pool.runInteraction(self._run)
+
+ def _run(self, cursor):
+ """Database interaction implementing the maintenance cycle"""
+ self.checkRollovers(cursor)
+ self.pruneSubscriptions(cursor)
+
+ def checkOneRollover(self, cursor, previous, current):
+ """Check for rollovers in one pair of consecutive time intervals,
+ like yesterday/today or lastMonth/thisMonth. This is meant to
+ be run inside a database interaction.
+ """
+ # Delete counters that are too old to bother keeping at all
+ cursor.execute("DELETE FROM stats_counters "
+ "WHERE (name = %s OR name = %s) "
+ "AND first_time < %s" %
+ (Database.quote(previous, 'varchar'),
+ Database.quote(current, 'varchar'),
+ Database.quote(long(TimeUtil.Interval(previous).getFirstTimestamp()), 'bigint')))
+
+ # Roll over remaining counters that are too old for current
+ # but still within the range of previous. Note that there is a
+ # race condition in which this update will fail because a timer
+ # has been incremented between it and the above DELETE. It could
+ # be prevented by locking the table, but it's probably not worth
+ # it. If the rollover fails this time, it will get another chance.
+ cursor.execute("UPDATE stats_counters SET name = %s "
+ "WHERE name = %s "
+ "AND first_time < %s" %
+ (Database.quote(previous, 'varchar'),
+ Database.quote(current, 'varchar'),
+ Database.quote(long(TimeUtil.Interval(current).getFirstTimestamp()), 'bigint')))
+
+ def checkRollovers(self, cursor):
+ """Check all applicable counters for rollovers. This should
+ be executed inside a database interaction.
+ """
+ self.checkOneRollover(cursor, 'yesterday', 'today')
+ self.checkOneRollover(cursor, 'lastWeek', 'thisWeek')
+ self.checkOneRollover(cursor, 'lastMonth', 'thisMonth')
+
+ def pruneSubscriptions(self, cursor, maxFailures=3):
+ """Delete subscriptions that have expired"""
+ cursor.execute("DELETE FROM stats_subscriptions WHERE expiration < %s" %
+ Database.quote(int(time.time()), 'bigint'))
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Stats/Target.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Stats/Target.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Stats/__init__.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Stats/__init__.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Stats/__init__.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,47 @@
+""" LibCIA.Stats
+
+The stats subsystem of CIA provides a way to associate messages
+with 'stats targets', storing and retrieving the resulting data.
+
+A target is a place messages can be delivered, identified by
+a unix-style path name. Note that this path has no connection
+to the real filesystem, it's just a convenient notation.
+
+When a message is delivered to a stats target, the message itself
+is stored in a FIFO, counters keeping track of that target's
+activity are incremented, and associations between stats targets
+are strengthened.
+
+Additionally, metadata can be assigned to these stats targets
+to help present them in a less abstract way.
+
+This package is the backend for the stats system. It takes care
+of actually storing and retrieving the data, and it provides
+python and XML-RPC interfaces to this functionality. Messages
+are directed into the stats subsystem using a stats:// URI handler.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+# Convenience imports
+import Handler
+import Interface
+import Target
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Stats/__init__.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Stats/__init__.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/TimeUtil.py
===================================================================
--- trunk/community/infrastructure/LibCIA/TimeUtil.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/TimeUtil.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,192 @@
+""" LibCIA.TimeUtil
+
+Date/time utilities. This includes a class for representing common
+time intervals like 'today' and 'last week', and methods for formatting
+dates and durations.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import Units
+import time, datetime, calendar
+
+
+class Interval(object):
+ """Represents some interval of time, like 'yesterday' or 'this week'.
+ Provides functions for returning the bounds of the
+ interval and testing whether a particular time is within it.
+ Represents time using datetime objects. By default, the interval
+ is created relative to the current date and time in UTC. To override
+ this, a datetime object can be passed to the constructor.
+
+ Interval is constructed with the name of the interval
+ it should represent.
+
+ >>> now = datetime.datetime(2003, 12, 19, 2, 19, 39, 50279)
+
+ >>> Interval('today', now)
+ <Interval from 2003-12-19 00:00:00 to 2003-12-20 00:00:00>
+
+ >>> Interval('yesterday', now)
+ <Interval from 2003-12-18 00:00:00 to 2003-12-19 00:00:00>
+
+ >>> Interval('thisWeek', now)
+ <Interval from 2003-12-15 00:00:00 to 2003-12-22 00:00:00>
+
+ >>> Interval('lastWeek', now)
+ <Interval from 2003-12-08 00:00:00 to 2003-12-15 00:00:00>
+
+ >>> Interval('thisMonth', now)
+ <Interval from 2003-12-01 00:00:00 to 2004-01-01 00:00:00>
+
+ >>> Interval('lastMonth', now)
+ <Interval from 2003-11-01 00:00:00 to 2003-12-01 00:00:00>
+ """
+ def __init__(self, name, now=None):
+ if not now:
+ now = datetime.datetime.utcnow()
+ self.range = getattr(self, name)(now)
+
+ def __repr__(self):
+ return "<Interval from %s to %s>" % self.range
+
+ def __contains__(self, dt):
+ if not isinstance(dt, datetime.datetime):
+ dt = datetime.datetime.utcfromtimestamp(dt)
+
+ return dt >= self.range[0] and dt < self.range[1]
+
+ def getFirstTimestamp(self):
+ """Return the first timestamp value within this range"""
+ return datetimeToTimestamp(self.range[0])
+
+ def today(self, now):
+ midnightToday = now.replace(hour=0, minute=0, second=0, microsecond=0)
+ midnightTomorrow = midnightToday + datetime.timedelta(days=1)
+ return (midnightToday, midnightTomorrow)
+
+ def yesterday(self, now):
+ midnightToday = now.replace(hour=0, minute=0, second=0, microsecond=0)
+ midnightYesterday = midnightToday - datetime.timedelta(days=1)
+ return (midnightYesterday, midnightToday)
+
+ def thisWeek(self, now):
+ midnightToday = now.replace(hour=0, minute=0, second=0, microsecond=0)
+ beginning = midnightToday - datetime.timedelta(days=calendar.weekday(now.year, now.month, now.day))
+ end = beginning + datetime.timedelta(weeks=1)
+ return (beginning, end)
+
+ def lastWeek(self, now):
+ thisWeek = self.thisWeek(now)
+ return (thisWeek[0] - datetime.timedelta(weeks=1), thisWeek[0])
+
+ def thisMonth(self, now):
+ beginning = now.replace(hour=0, minute=0, second=0, microsecond=0, day=1)
+ try:
+ end = beginning.replace(month=now.month+1)
+ except ValueError:
+ # Next year
+ end = beginning.replace(month=1, year=now.year+1)
+ return (beginning, end)
+
+ def lastMonth(self, now):
+ end = now.replace(hour=0, minute=0, second=0, microsecond=0, day=1)
+ try:
+ beginning = end.replace(month=now.month-1)
+ except ValueError:
+ # Last year
+ beginning = end.replace(month=12, year=now.year-1)
+ return (beginning, end)
+
+
+formatDuration = Units.TimeUnits().format
+
+
+def formatDate(t):
+ """Format a date, in UTC seconds since the epoch.
+ This should use a format that doesn't necessarily adhere to
+ any standard, but is easy to read and fairly universal.
+ """
+ return time.strftime("%H:%M on %b %d, %Y", time.gmtime(t))
+
+def formatRelativeDate(t):
+ """Like formatDate, this is an arbitrary format chosen to be
+ easily human-readable, but this function is allowed to
+ render it relative to the current time.
+ """
+ now = time.time()
+ delta = int(now - t)
+
+ if delta < 0:
+ return "%ss in the future" % -delta
+ if delta < 1:
+ return "< 1 sec ago"
+ if delta < 60:
+ return "%d sec ago" % delta
+
+ delta /= 60
+ if delta < 60:
+ return "%d min ago" % delta
+
+ tmNow = time.gmtime(now)
+ tm = time.gmtime(t)
+ if tm.tm_year == tmNow.tm_year:
+
+ if tm.tm_yday == tmNow.tm_yday:
+ return time.strftime("%H:%M today", tm)
+
+ if tm.tm_yday == tmNow.tm_yday - 1:
+ return time.strftime("%H:%M yesterday", tm)
+
+ if tmNow.tm_yday - tm.tm_yday < 7:
+ return time.strftime("%H:%M %A", tm)
+
+ return time.strftime("%H:%M on %b %d", tm)
+
+ return time.strftime("%H:%M on %b %d, %Y", tm)
+
+
+def formatLogDate(t):
+ """Format a date in the format they should be in our log file.
+ This should look just like the date Twisted puts in automatically,
+ in the local time zone.
+ """
+ y,mon,d,h,min, iigg,nnoo,rree,daylight = time.localtime(t)
+ return "%0.4d/%0.2d/%0.2d %0.2d:%0.2d %s" % (
+ y, mon, d, h, min, time.tzname[daylight])
+
+def formatDateRFC822(t):
+ """Format a date, in UTC seconds since the epoch, using RFC822 formatting"""
+ return time.strftime("%a, %d %b %Y %H:%M:%S +0000", time.gmtime(t))
+
+
+def formatDateISO8601(t):
+ """Format a date, in UTC seconds since the epoch, using ISO 8601 format"""
+ return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(t))
+
+
+def mktime_utc(data):
+ t = time.mktime(data[:8] + (0,))
+ return t - time.timezone
+
+def datetimeToTimestamp(datetime):
+ """Convert a UTC datetime object to a UTC timestamp"""
+ return mktime_utc(datetime.utctimetuple())
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/TimeUtil.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/TimeUtil.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Units.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Units.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Units.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,89 @@
+""" LibCIA.Units
+
+A small framework for defining collections of units for a particular
+quantity and formatting values with a proper unit automatically.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from __future__ import division
+
+class UnitCollection:
+ """An abstract group of units for some quntity"""
+ # A list of (singular name, plural name, multiplier) tuples.
+ # Must be sorted by multiplier in descending order.
+ units = []
+
+ # If the converted value would be less than this
+ # threshold, a smaller unit will be chosen.
+ threshold = 0.8
+
+ # Smallest decimal place to represent
+ precision = 0.01
+
+ def format(self, value):
+ """Pick an appropriate unit and format the given value"""
+ for singular, plural, multiplier in self.units:
+ converted = value / multiplier
+ if converted > self.threshold:
+ break
+
+ # Round the converted value to our set precision
+ converted += self.precision / 2
+ converted -= converted % self.precision
+
+ # Is it close enough to 1 to use our singular form?
+ if abs(converted - 1) < self.precision:
+ unit = singular
+ else:
+ unit = plural
+
+ # Chop off the trailing .0 if there is one
+ s = str(converted)
+ if s.endswith('.0'):
+ s = s[:-2]
+ return s + ' ' + unit
+
+
+class TimeUnits(UnitCollection):
+ """Time units, standard unit is 1 second"""
+ units = [
+ ('year', 'years', 365 * 24 * 60 * 60),
+ ('month', 'months', 30 * 24 * 60 * 60),
+ ('week', 'weeks', 7 * 24 * 60 * 60),
+ ('day', 'days', 24 * 60 * 60),
+ ('hour', 'hours', 60 * 60),
+ ('minute', 'minutes', 60),
+ ('second', 'seconds', 1),
+ ('millisecond', 'milliseconds', 0.001),
+ ('microsecond', 'microseconds', 0.000001),
+ ]
+
+
+class StorageUnits(UnitCollection):
+ """Storage unit, standard is 1 bytes"""
+ units = [
+ ('TB', 'TB', 1024 * 1024 * 1024 * 1024),
+ ('GB', 'GB', 1024 * 1024 * 1024),
+ ('MB', 'MB', 1024 * 1024),
+ ('kB', 'kB', 1024),
+ ('byte', 'bytes', 1),
+ ]
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Units.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Units.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Info.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Info.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Info.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,246 @@
+""" LibCIA.Web.Info
+
+Just a cute little page with informational doodads on it.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from __future__ import division
+import Template, Server
+from LibCIA import TimeUtil, XML, Database, Units, Cache
+from Nouvelle import place, tag
+import LibCIA, Nouvelle
+from twisted.internet import defer
+import time, sys, os
+
+
+class Component(Server.Component):
+ """A server component showing the info page"""
+ name = 'Server Info'
+
+ def __init__(self):
+ self.resource = Page()
+
+ def __contains__(self, page):
+ return isinstance(page, Page)
+
+
+class Clock(Template.Section):
+ title = 'UTC clock'
+
+ def render_rows(self, context):
+ return [TimeUtil.formatDate(time.time())]
+
+
+class Version(Template.Section):
+ title = 'version'
+
+ def render_rows(self, context):
+ rows = [Template.value[LibCIA.__version__]]
+ svnRev = self.getSvnRevision()
+ if svnRev:
+ rows.append(["SVN revision ", Template.value[svnRev]])
+ rows.append(["Database version ", self.getDbVersion()])
+ return rows
+
+ def getSvnRevision(self):
+ """Return the current Subversion repository revision, or None
+ if we're not in an svn working copy or it can't be parsed.
+ """
+ try:
+ entries = XML.parseString(open(".svn/entries").read()).documentElement
+ highestRev = 0
+ for tag in XML.getChildElements(entries):
+ if tag.nodeName == 'entry':
+ rev = tag.getAttributeNS(None, 'committed-rev')
+ if rev and rev > highestRev:
+ highestRev = rev
+ return highestRev
+ except:
+ return None
+
+ def getDbVersion(self):
+ """Return our database's schema version via a Deferred"""
+ return Database.pool.runInteraction(self._getDbVersion)
+
+ def _getDbVersion(self, cursor):
+ cursor.execute("SELECT value FROM meta WHERE name = 'version'")
+ row = cursor.fetchone()
+ if row:
+ return Template.value[ row[0] ]
+ else:
+ return Template.error["unknown"]
+
+
+class WebServer(Template.Section):
+ title = 'web server'
+
+ def render_requestCount(self, context):
+ return context['request'].site.requestCount
+
+ def render_uptime(self, context):
+ return TimeUtil.formatDuration(time.time() - context['request'].site.serverStartTime)
+
+ def render_mtbr(self, context):
+ site = context['request'].site
+ if site.requestCount > 0:
+ return TimeUtil.formatDuration((time.time() - site.serverStartTime) / site.requestCount)
+ else:
+ return Template.error["unknown"]
+
+ rows = [
+ [
+ Template.value[ place('requestCount') ],
+ ' requests have been processed since the server was loaded',
+ ],
+ [
+ 'The server has been up for ',
+ Template.value[ place('uptime') ],
+ ],
+ [
+ 'On average, there is a request every ',
+ Template.value[ place('mtbr') ],
+ ],
+ ]
+
+
+class IndexedStorageColumn(Nouvelle.IndexedColumn):
+ """An IndexedColumn that renders its content as a
+ size, in bytes- rescaled into an appropriate unit
+ """
+ def render_data(self, context, row):
+ return Units.StorageUnits().format(self.getValue(row))
+
+
+class HitRatioColumn(Nouvelle.Column):
+ """Show the ratio of hits to total cache accesses, as a percentage"""
+ heading = 'hit ratio'
+
+ def getValue(self, perf):
+ total = perf.hits + perf.misses
+ if total > 0:
+ return perf.hits / total * 100
+
+ def render_data(self, context, perf):
+ value = self.getValue(perf)
+ if value is None:
+ return Template.error["unknown"]
+ else:
+ return "%.02f %%" % value
+
+
+class CachePerformance(Template.Section):
+ title = 'cache performance'
+
+ columns = [
+ Nouvelle.AttributeColumn('name', 'name'),
+ Nouvelle.AttributeColumn('hits', 'hits'),
+ Nouvelle.AttributeColumn('misses', 'misses'),
+ HitRatioColumn(),
+ ]
+
+ def render_rows(self, context):
+ perfList = list(Cache.getCachePerformanceList())
+ if perfList:
+ return [Template.Table(perfList, self.columns, id='cachePerf')]
+ else:
+ return []
+
+
+class DbTables(Template.Section):
+ title = 'database tables'
+
+ columns = [
+ Nouvelle.IndexedColumn('name', 0),
+ IndexedStorageColumn('data size', 5),
+ IndexedStorageColumn('index size', 7),
+ Nouvelle.IndexedColumn('type', 1),
+ Nouvelle.IndexedColumn('approx. rows', 3),
+ ]
+
+ def render_rows(self, context):
+ # Fetch the results of a 'show table status' before we can do anything
+ result = defer.Deferred()
+ Database.pool.runQuery('SHOW TABLE STATUS').addCallback(
+ self._render_rows, result).addErrback(result.errback)
+ return result
+
+ def _render_rows(self, tableInfo, result):
+ result.callback([
+ Template.Table(list(tableInfo), self.columns, id='db'),
+ ])
+
+
+class System(Template.Section):
+ title = 'system'
+
+ def render_pyInfo(self, context):
+ result = []
+ for line in ('Python %s on %s' % (sys.version, sys.platform)).split("\n"):
+ if result:
+ result.append(tag('br'))
+ result.append(line)
+ return result
+
+ def render_sysUptime(self, context):
+ # This only works on linux systems for now
+ try:
+ seconds = float(open("/proc/uptime").read().split()[0])
+ except:
+ return "System uptime unknown"
+ else:
+ return ["This system has been up for ",
+ Template.value[ TimeUtil.formatDuration(seconds) ]]
+
+ def render_load(self, context):
+ # Also only works on linux now
+ try:
+ load = os.getloadavg()
+ except OSError:
+ return "Load average unknown"
+ else:
+ return ["Load average: ",
+ Template.value[ load[0] ],
+ " ", load[1], " ", load[2]]
+
+ rows = [
+ place('pyInfo'),
+ place('sysUptime'),
+ place('load'),
+ ]
+
+
+class Page(Template.Page):
+ """A web page showing information about the server"""
+ mainTitle = 'Server Info'
+ subTitle = 'so that everyone needs to hang on tighter just to keep from being thrown to the wolves'
+
+ leftColumn = [
+ Version(),
+ Clock(),
+ ]
+
+ mainColumn = [
+ WebServer(),
+ System(),
+ CachePerformance(),
+ DbTables(),
+ ]
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Info.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Info.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Overview.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Overview.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Overview.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,193 @@
+""" LibCIA.Web.Overview
+
+A front page for CIA showing a concise overview of the day's open
+source development activity.
+
+Note that this is currently the *only* module in CIA that makes
+assumptions about the way the stats hierarchy is organized. If you
+have CIA set up differently, you will need to modify this page
+or choose not to use it in your .tac file.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import random
+import Template
+from LibCIA import Database
+from LibCIA.Stats import Target
+from LibCIA.Web.Stats import Columns
+from Nouvelle import tag, place
+import Nouvelle
+from twisted.internet import defer
+
+
+class ActivitySection(Template.Section):
+ """A Section displaying links to the most or least active items within
+ a given stats target.
+ """
+ query = """
+ SELECT
+ STAT.target_path,
+ ST.title,
+ ICO.path,
+ STAT.%(counter_attrib)s,
+ ICO.width,
+ ICO.height
+ FROM stats_counters STAT FORCE INDEX (%(counter_attrib)s)
+ LEFT OUTER JOIN stats_catalog T ON (STAT.target_path = T.target_path)
+ LEFT OUTER JOIN stats_statstarget ST ON (T.target_path = ST.path)
+ LEFT OUTER JOIN images_imageinstance ICO ON (ICO.source_id = IF(ST.icon_id IS NOT NULL, ST.icon_id, ST.photo_id) AND ICO.thumbnail_size = 32)
+ WHERE STAT.name = %(counter)s AND T.parent_path = %(path)s
+ ORDER BY STAT.%(counter_attrib)s %(sort)s
+ LIMIT %(limit)d
+ """
+
+ def __init__(self, targetPath, title,
+ numItems = 15,
+ counter = 'today',
+ counterAttrib = 'event_count',
+ sort = 'DESC',
+ columnTitle = 'events today',
+ ):
+ self.targetPath = targetPath
+ self.title = title
+ self.numItems = numItems
+ self.counter = counter
+ self.counterAttrib = counterAttrib
+ self.sort = sort
+ self.columnTitle = columnTitle
+ self.title = title
+ self.initQuery()
+ self.initColumns()
+
+ def initQuery(self):
+ self.query = self.query % dict(
+ path = Database.quote(self.targetPath, 'varchar'),
+ limit = self.numItems,
+ counter = Database.quote(self.counter, 'varchar'),
+ counter_attrib = self.counterAttrib,
+ sort = self.sort,
+ )
+
+ def initColumns(self):
+ self.columns = [
+ Columns.IndexedIconColumn(iconIndex=2, widthIndex=4, heightIndex=5),
+ Columns.TargetTitleColumn(pathIndex=0, titleIndex=1),
+ Columns.IndexedBargraphColumn(self.columnTitle, 3),
+ ]
+
+ def render_rows(self, context):
+ # First we run a big SQL query to gather all the data for this catalog.
+ # Control is passed to _render_rows once we have the query results.
+ result = defer.Deferred()
+ Database.pool.runQuery(self.query).addCallback(
+ self._render_rows, context, result
+ ).addErrback(result.errback)
+ return result
+
+ def _render_rows(self, queryResults, context, result):
+ # Use a non-resortable table for this, since we've already dictated
+ # how our data is sorted in the SQL query and the user can't do much about it yet.
+ result.callback([Nouvelle.Table.BaseTable(list(queryResults), self.columns)])
+
+
+class TimestampSection(ActivitySection):
+ """A section showing the newest or oldest items within a stats target"""
+
+ def __init__(self, targetPath, title,
+ numItems = 15,
+ counter = 'forever',
+ counterAttrib = 'first_time',
+ sort = 'DESC',
+ columnTitle = 'first event',
+ ):
+ ActivitySection.__init__(self, targetPath, title, numItems,
+ counter, counterAttrib, sort, columnTitle)
+
+ def initColumns(self):
+ self.columns = [
+ Columns.IndexedIconColumn(iconIndex=2, widthIndex=4, heightIndex=5),
+ Columns.TargetTitleColumn(pathIndex=0, titleIndex=1),
+ Columns.TargetLastEventColumn(self.columnTitle, 3),
+ ]
+
+
+class OverviewPage(Template.Page):
+ """A web page showing an overview of open source activity, meant to
+ be used as a front page for the CIA web site. We want to act as
+ a jumping-off point for the rest of the site, so this page doesn't
+ include its own sidebar- it will copy the sidebar from a given page.
+ """
+
+ titleElements = [
+ tag('img', _class='banner', src='/media/img/banner-70-nb.png', width=329, height=52,
+ alt='CIA.vc: The open source version control informant.'),
+ ]
+
+ logoElements = []
+
+ heading = Template.pageBody[
+ "This is a brief overview of the information collected recently. ",
+ tag("a", href="/doc")[ "Learn more about CIA" ],
+ ]
+
+ def __init__(self, sidebarPath='doc/.default.sidebar'):
+ self.leftColumn = self._loadSidebar(sidebarPath)
+
+ def _loadSidebar(self, path):
+ """Load sidebar links from a simple text file format.
+ Lines beginning with a dash specify a new section heading,
+ links are made by lines of the form 'title :: URL'.
+ Other lines are ignored.
+ """
+ sections = []
+ for line in open(path).xreadlines():
+ line = line.strip()
+ if not line:
+ continue
+
+ if line[0] == '-':
+ sections.append(Template.StaticSection(line[1:].strip()))
+ continue
+
+ pieces = line.split("::", 1)
+ if len(pieces) > 1:
+ title, url = pieces
+ sections[-1].rows.append( tag('a', href=url.strip())[title.strip()] )
+
+ return sections
+
+ def render_mainColumn(self, context):
+ return [
+ self.heading,
+ [
+ Template.SectionGrid(
+ [
+ ActivitySection("project", "Most active projects today"),
+ ActivitySection("author", "Most active authors today"),
+ ],
+ [
+ TimestampSection("project", "Newest projects"),
+ TimestampSection("author", "Newest authors"),
+ ],
+ ),
+ ]
+ ]
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Overview.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Overview.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/RegexTransform.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/RegexTransform.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/RegexTransform.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,165 @@
+""" LibCIA.Web.RegexTransform
+
+Applies formatting to text in Nouvelle trees using regular expressions.
+Each formatting class includes a dictionary of regular expressions mapped
+to handler functions. When a regular expression is encountered, the handler
+is called with the match object and the returned Nouvelle tree is inserted
+in the original text's place.
+
+This system can be used to auto-hyperlink URLs, format source code, and
+probably much more.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import re, types, Nouvelle
+import Template
+
+
+class RegexTransformBase(object):
+ """Abstract base class for regex transformation engines. Subclasses
+ must provide their own 'regexes' dictionary, mapping regex
+ strings or compiled regexes to member functions. The constructor
+ for this class walks through the class' ancestors and compiles
+ regular expressions as necessary.
+ """
+ regexes = {}
+
+ def __init__(self):
+ # If this class doesn't have a compiled regex map, generate it
+ if not hasattr(self.__class__, '_compiled'):
+ self.__class__._compiled = self.compile()
+
+ def compile(self):
+ """Create a map of compiled regexes to handlers
+ for this class and all its ancestors.
+ """
+ c = {}
+ for cls in self.__class__.__mro__:
+ # Stop when we get to this base class
+ if cls is RegexTransformBase:
+ break
+
+ for uncompiled, handler in cls.regexes.iteritems():
+ if type(uncompiled) in types.StringTypes:
+ compiled = re.compile(uncompiled)
+ else:
+ compiled = uncompiled
+ c[compiled] = handler
+ return c
+
+ def apply(self, tree, memo={}):
+ """Recursively apply our compiled regexes to a Nouvelle tree,
+ returning the transformed Nouvelle tree.
+ """
+ if type(tree) is tuple or type(tree) is list:
+ return [self.apply(item, memo) for item in tree]
+
+ elif type(tree) in types.StringTypes:
+ # Yay, we found a string. Can we match any regexes?
+
+ # Remember in the memo each regex that we process
+ # so that we can avoid processing the same tree fragment
+ # with this regex again later.
+ memo = dict(memo)
+
+ for regex, handler in self.__class__._compiled.iteritems():
+ if regex in memo:
+ continue
+ memo[regex] = 1
+
+ results = []
+ allStrings = True
+
+ # For each regex, scan our string for matches.
+ # Non-matched sections of the string and transformed
+ # matches are accumulated into 'results'. If all
+ # results are strings, the results are joined and
+ # we can move on to the next regex immediately.
+ # If not, we have to give this up now and recursively
+ # transform each result.
+ lastMatch = None
+ for match in regex.finditer(tree):
+ # Store the fragment between the last match (if there was one)
+ # and the beginning of this match.
+ if lastMatch:
+ start = lastMatch.end()
+ else:
+ start = 0
+ results.append(tree[start:match.start()])
+
+ # Store the transformed result of this match.
+ # Note that we're passing self to handler!
+ # This is a bit of a hack since the handlers in
+ # self.regexes won't be bound methods normally.
+ transformed = handler(self, match)
+ results.append(transformed)
+ if type(transformed) not in types.StringTypes:
+ allStrings = False
+ lastMatch = match
+
+ # Store the fragment between the last match and the end
+ if lastMatch:
+ results.append(tree[lastMatch.end():])
+
+ # If we can join the results back into a string, do so
+ # and go ahead on to the next regex. If not, abort this
+ # and recursively process the results.
+ if allStrings:
+ tree = "".join(results)
+ else:
+ return [self.apply(item, memo) for item in results]
+ return tree
+
+ elif isinstance(tree, Nouvelle.tag):
+ # Recursively process the tag's content
+ return tree[self.apply(tree.content, memo)]
+
+ # Anything else (hopefully a Nouvelle-serializable class) passes through unmodified
+ return tree
+
+
+class AutoHyperlink(RegexTransformBase):
+ """A transform class that automatically hyperlinks all URLs.
+ For Example:
+
+ >>> linker = AutoHyperlink()
+ >>> serial = Nouvelle.Serializer()
+
+ >>> tree = linker.apply('Visit http://foo.bar today for a free toothbrush')
+ >>> serial.render(tree)
+ 'Visit <a href="http://foo.bar">http://foo.bar</a> today for a free toothbrush'
+
+ (This works for email addresses as well, but they aren't demonstrated here because
+ the automatic email address obfuscation is not deterministic)
+ """
+ def link_url(self, match):
+ url = match.group()
+ return Nouvelle.tag('a', href=url)[ url ]
+
+ def link_email(self, match):
+ address = match.group()
+ return Template.EmailLink("mailto:"+address)[ address ]
+
+ regexes = {
+ '(ht|f)tps?://[^\s\>\]\)]+': link_url,
+ '[\w\.\-\+]+@([0-9a-zA-Z\-]+\.)+[a-zA-Z]+': link_email,
+ }
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/RegexTransform.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/RegexTransform.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Server.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Server.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Server.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,193 @@
+""" LibCIA.Web.Server
+
+General classes for CIA's web server, including subclasses of twisted.web's
+Site and Request classes.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.web import server, static
+from twisted.python import log
+import ServerPages
+import time, gc, os
+
+
+class Request(server.Request):
+ """A Request subclass overriding some default policies
+ of twisted.web, including exception reporting
+ """
+ def processingFailed(self, reason):
+ """A replacement for twisted.web's usual traceback page.
+ We disable the traceback and only report the actual exception,
+ for a few reasons:
+
+ - Often the data structures in use are large, and repr()'ing them
+ for the web page is very slow
+
+ - Showing the values of variables on the stack may disclose capability
+ keys or other values that the web users shouldn't be able to see
+
+ - The tracebacks really aren't any more helpful than the
+ ones reported to twistd.log :)
+ """
+ log.err(reason)
+ page = ServerPages.InternalErrorPage(reason)
+ page.render(self)
+ self.finish()
+ return reason
+
+ def getClientIP(self):
+ """Get the real IP address of our client. This is aware of proxies
+ that support the X-Forwarded-For HTTP header.
+ """
+ xff = self.getHeader('X-Forwarded-For')
+ if xff:
+ return xff.split(',', 1)[0].strip()
+ return server.Request.getClientIP(self)
+
+ def process(self):
+ # Allow environment variables to override the host/port of this machine,
+ # with an extra header to enable SSL. This is an alternative to VHostMonster
+ # that works with the "pound" proxy and load balancer.
+ host = os.getenv("REQUEST_HOST")
+ if host:
+ xfp = self.getHeader('X-Forwarded-Proto')
+ if xfp and xfp.strip().lower() == "https":
+ self.setHost(host, 443)
+ self.isSecure = lambda: 1
+ else:
+ self.setHost(host, int(os.getenv("REQUEST_PORT", 80)))
+ self.isSecure = lambda: 0
+
+ # Count this request, yay
+ server.Request.process(self)
+ self.site.requestCount += 1
+
+
+class File(static.File):
+ """A local subclass of static.File that overrides the default MIME type map
+ and directory listing behaviour. We would rather all the scripts in /clients
+ be given out as text/plain so they're easy to view in a browser, and the directory
+ listings should be given using our own template rather than Twisted's.
+ """
+ contentTypes = {
+ '.png': 'image/png',
+ '.ico': 'image/png',
+ '.jpeg': 'image/jpeg',
+ '.jpg': 'image/jpeg',
+ '.gif': 'image/gif',
+ '.xml': 'text/xml',
+ '.html': 'text/html',
+ '.css': 'text/css',
+ '.js': 'application/x-javascript',
+ '.svg': 'image/svg+xml',
+ }
+
+ def listNames(self):
+ """Override listNames to hide hidden files, like .svn and .xvpics"""
+ listing = static.File.listNames(self)
+ return [item for item in listing if not item.startswith(".")]
+
+ def directoryListing(self):
+ """Use our own directory lister rather than relying on Twisted's default.
+ This lets us keep the site's look more consistent, and doesn't pull
+ in all of Woven just for a silly listing page.
+ """
+ return ServerPages.DirectoryListing(self)
+
+
+class StaticJoiner(File):
+ """This web page acts mostly like a static.File, and all children
+ are files under the given directory- however this page itself
+ renders the provided 'index' page. This can be used to create
+ a dynamically generated front page that references static pages
+ or images as its children.
+ """
+ def __init__(self, path, indexPage,
+ defaultType = "text/plain",
+ ignoredExts = (),
+ registry = None,
+ allowExt = 0):
+ self.indexPage = indexPage
+ static.File.__init__(self, path, defaultType, ignoredExts, registry, allowExt)
+
+ def getChild(self, path, request):
+ if path:
+ return static.File.getChild(self, path, request)
+ else:
+ return self
+
+ def render(self, request):
+ return self.indexPage.render(request)
+
+ def createSimilarFile(self, path):
+ f = File(path, self.defaultType, self.ignoredExts, self.registry)
+ f.processors = self.processors
+ f.indexNames = self.indexNames[:]
+ return f
+
+
+class Component:
+ """A component is some top-level area of the site that is explicitly
+ assigned a URL and may be visible to the user in some sort of
+ site-wide navigation system. Every component must have a root URL
+ and a root resource- indeed, the major reason Components exist is
+ to give a subsystem a way to bind itself to a subset of a site's URL
+ space.
+ """
+ # The component's URL, will be set by the Site
+ url = None
+
+ # The component's resource, will be set by the component's constructor.
+ # Optional. Components implemented in the Django server won't have this.
+ resource = None
+
+ # The component's user-visible name, if it has one
+ name = None
+
+ def __init__(self, name=None):
+ self.name = name
+
+ def __contains__(self, page):
+ """Subclasses must implement this to test whether a page
+ belongs to this component.
+ """
+ return False
+
+
+class Site(server.Site):
+ """A twisted.web.server.Site subclass, to use our modified Request class"""
+ requestFactory = Request
+
+ def __init__(self, resource):
+ # Some extra widgets for tracking server uptime and hit count
+ self.requestCount = 0
+ self.serverStartTime = time.time()
+
+ self.components = []
+ server.Site.__init__(self, resource)
+
+ def putComponent(self, childName, component):
+ """Install the given component instance at 'childName'"""
+ component.url = '/' + childName
+ if component.resource:
+ self.resource.putChild(childName, component.resource)
+ self.components.append(component)
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Server.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Server.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/ServerPages.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/ServerPages.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/ServerPages.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,203 @@
+""" LibCIA.Web.ServerPages
+
+Web pages used internally to the server, including the "internal server error"
+page and a directory listing page.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.web import http
+from Nouvelle import tag, place, Twisted
+from LibCIA import TimeUtil, Units
+import Template
+import time, Nouvelle
+
+
+class InternalErrorPage(Twisted.Page):
+ """An internal server error page, generated when we encounter an exception
+ generating the intended page. This page should be fairly simple, so there's
+ little risk in it causing an exception also.
+ """
+ def __init__(self, failure):
+ self.failure = failure
+
+ def preRender(self, context):
+ request = context['request']
+ request.setHeader('content-type', "text/html")
+ request.setResponseCode(http.INTERNAL_SERVER_ERROR)
+
+ def render_time(self, context):
+ return TimeUtil.formatDateRFC822(time.time())
+
+ def render_excType(self, context):
+ return str(self.failure.value.__class__)
+
+ def render_excValue(self, context):
+ return str(self.failure.value)
+
+ def render_traceback(self, context):
+ return self.failure.getTraceback()
+
+ def render_uri(self, context):
+ return context['request'].uri
+
+ document = tag('html')[
+ tag('head')[
+ tag('title')[ "Internal Server Error" ],
+ ],
+ tag('body')[
+ tag('h2')[ "Internal Server Error" ],
+
+ # Friendly message
+ tag('p')[
+ "Sorry, it looks like you just found a bug. If you would like to "
+ "help us identify the problem, please email a copy of this page to the "
+ "webmaster of this site along with a description of what happened. Thanks!"
+ ],
+
+ # Table of useful values
+ tag('table', cellpadding=5) [
+ tag('tr')[
+ tag('td')[ tag('b')[ 'Current time:' ]],
+ tag('td')[ place('time') ],
+ ],
+ tag('tr')[
+ tag('td')[ tag('b')[ 'Requested path:' ]],
+ tag('td')[ place('uri') ],
+ ],
+ tag('tr')[
+ tag('td')[ tag('b')[ 'Exception type:' ]],
+ tag('td')[ place('excType') ],
+ ],
+ tag('tr')[
+ tag('td')[ tag('b')[ 'Exception value:' ]],
+ tag('td')[ place('excValue') ],
+ ],
+ ],
+
+ # Traceback
+ tag('p')[
+ tag('b')[ 'Traceback:' ],
+ ],
+ tag('p')[
+ tag('pre')[ place('traceback') ],
+ ],
+ ],
+ ]
+
+
+class DirectoryListing(Template.Page):
+ """Generates pretty listings for static directories"""
+ def __init__(self, dir):
+ self.dir = dir
+ self.mainTitle = self.dir.basename()
+ self.subTitle = 'directory listing'
+
+ self.leftColumn = [
+ DirectoryTreeSection(self.dir)
+ ]
+
+ self.mainColumn = [
+ DirectoryContentsSection(self.dir)
+ ]
+
+
+class DirectoryTreeSection(Template.Section):
+ """A section showing this directory's location in the tree"""
+ title = 'tree'
+
+ def __init__(self, dir):
+ self.dir = dir
+
+ def render_rows(self, context):
+ """Assemble our file tree in the form of nested dictionaries,
+ then let Template.FileTree render it.
+ """
+ request = context['request']
+ tree = {}
+ node = tree
+ urlSegments = ['']
+
+ # Walk down the tree putting in our ancestor directories
+ # (not counting the root directory, since navigating to
+ # it is better done using our tabs or site name link)
+ for path in request.prepath:
+ if path:
+ urlSegments.append(path)
+ link = tag('a', href="/".join(urlSegments))[ path ]
+ node = node.setdefault(link, {})
+
+ return [Template.FileTree(tree)]
+
+
+class FileNameColumn(Nouvelle.Column):
+ heading = 'name'
+
+ def getValue(self, file):
+ return file.basename()
+
+ def render_data(self, context, file):
+ """We hyperlink the file name, and give it an icon to indicate directory vs file"""
+ if file.isdir():
+ icon = Template.dirIcon
+ else:
+ icon = Template.fileIcon
+ name = file.basename()
+ return [
+ icon,
+ tag('a', href=name)[ name ],
+ ]
+
+
+class FileSizeColumn(Nouvelle.Column):
+ heading = 'size'
+
+ def __init__(self):
+ self.formatter = Units.StorageUnits().format
+
+ def getValue(self, file):
+ return file.getFileSize()
+
+ def render_data(self, context, file):
+ if file.isfile():
+ return self.formatter(file.getFileSize())
+ else:
+ return ''
+
+
+class DirectoryContentsSection(Template.Section):
+ """A section showing this directory's contents"""
+ title = 'directory contents'
+
+ columns = [
+ FileNameColumn(),
+ FileSizeColumn(),
+ ]
+
+ def __init__(self, dir):
+ self.dir = dir
+
+ def render_rows(self, context):
+ files = self.dir.listEntities()
+ if files:
+ return [Template.Table(files, self.columns, id="contents")]
+ else:
+ return ["No visible files"]
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/ServerPages.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/ServerPages.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Browser.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Stats/Browser.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Stats/Browser.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,377 @@
+""" LibCIA.Web.Stats.Browser
+
+Implements CIA's actual stats browser pages. This uses Sections provided
+by other modules in this package, and detects magic URLs that rediect one
+to metadata or RSS pages.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.python import log
+from twisted.internet import defer
+from twisted.web import error, resource, server, http
+from LibCIA.Web import Template, Info, Server
+from LibCIA import Stats, Message, TimeUtil, Formatters, XML
+from Nouvelle import tag, place
+import Nouvelle, time, sys, posixpath, re
+import Metadata, Catalog, Feed, Link, MessageViewer, Graph
+
+
+class Component(Server.Component):
+ """A server component representing the whole stats interface"""
+ name = "Stats"
+
+ def __init__(self):
+ self.resource = Page(self)
+
+ def __contains__(self, page):
+ for cls in (Page,
+ Feed.CustomizeRSS,
+ MessageViewer.MessagePage,
+ ):
+ if isinstance(page, cls):
+ return True
+ return False
+
+
+class Page(Template.Page):
+ """A web page providing an interface to one StatsTarget.
+ The root of the stats namespace should be created with the
+ capabilities database and StatsStorage. Children will
+ be automatically created with child targets.
+ """
+ def __init__(self, component, target=None):
+ if target is None:
+ target = Stats.Target.StatsTarget()
+ self.component = component
+ self.target = target
+
+ def parent(self):
+ parentTarget = self.target.parent()
+ if parentTarget:
+ return self.__class__(self.component, parentTarget)
+
+ def getURL(self, context):
+ return posixpath.join(self.component.url, self.target.path)
+
+ def getChildWithDefault(self, name, request):
+ """Part of IResource, called by twisted.web to retrieve pages for URIs
+ below this one. This just creates a Page instance for our StatsTarget's child,
+ with a few special cases used for metadata and editing.
+ """
+ childFactories = {
+ '.message': MessageViewer.RootPage,
+ '.rss': Feed.RSSFrontend,
+ '.xml': Feed.XMLFeed,
+ }
+
+ if not name:
+ # Ignore empty path sections
+ return self
+ elif name in childFactories:
+ return childFactories[name](self)
+ else:
+ # Return the stats page for a child
+ try:
+ child = self.target.child(name)
+ except ValueError:
+ # Reserved word in stats target
+ return error.NoResource("Invalid stats path")
+ else:
+ return self.__class__(self.component, child)
+
+ def preRender(self, context):
+ context['component'] = self.component
+
+ def render_mainTitle(self, context):
+ return self.target.getTitle()
+
+ def render_subTitle(self, context):
+ return self.target.metadata.getValue('subtitle', 'Real-time open source activity stats')
+
+ def render_mainColumn(self, context):
+ return [
+ Counters(self.target),
+ Catalog.CatalogSection(self.target),
+ RecentMessages(self.target),
+ ]
+
+ def render_leftColumn(self, context):
+ return [
+ Metadata.Info(self.target),
+ LinksSection(self.target),
+ Graph.RelatedSection(self.target),
+ Info.Clock(),
+ ]
+
+ def render_extraHeaders(self, context):
+ # Add a <link> tag pointing at our RSS feed. Some RSS
+ # aggregators can use this to automatically detect feeds.
+ return tag('link',
+ rel = 'alternate',
+ type = 'application/rss+xml',
+ title = 'RSS',
+ href = Link.RSSLink(self.target).getURL(context),
+ )
+
+
+class Counters(Template.Section):
+ """A Section displaying the counters from a StatsTarget"""
+ title = "event counters"
+
+ rows = [
+ [
+ 'The last message was received ',
+ Template.value[ place('value', 'forever', 'lastEventTime', 'relativeDate') ],
+ ' ago at ',
+ Template.value[ place('value', 'forever', 'lastEventTime', 'date') ],
+ ],
+ [
+ Template.value[ place('value', 'today', 'eventCount') ],
+ ' messages so far today, ',
+ Template.value[ place('value', 'yesterday', 'eventCount') ],
+ ' messages yesterday',
+ ],
+ [
+ Template.value[ place('value', 'thisWeek', 'eventCount') ],
+ ' messages so far this week, ',
+ Template.value[ place('value', 'lastWeek', 'eventCount') ],
+ ' messages last week',
+ ],
+ [
+ Template.value[ place('value', 'thisMonth', 'eventCount') ],
+ ' messages so far this month, ',
+ Template.value[ place('value', 'lastMonth', 'eventCount') ],
+ ' messages last month',
+ ],
+ [
+ Template.value[ place('value', 'forever', 'eventCount') ],
+ ' messages since the first one, ',
+ Template.value[ place('value', 'forever', 'firstEventTime', 'relativeDate') ],
+ ' ago',
+ place('averagePeriod', 'forever'),
+ ],
+ ]
+
+ def __init__(self, target):
+ self.counters = target.counters
+
+ def render_rows(self, context):
+ """If this target has received at least one event, render the rows normally..
+ otherwise, hide this section completely.
+ """
+ result = defer.Deferred()
+ self.counters.getCounter('forever').addCallback(
+ self._render_rows, result).addErrback(result.errback)
+ return result
+
+ def _render_rows(self, foreverCounter, result):
+ if foreverCounter and foreverCounter['eventCount'] > 0:
+ result.callback(self.rows)
+ else:
+ result.callback([])
+
+ def render_value(self, context, counterName, valueName, filter=None):
+ """Fetch a counter value, rendering its value optionally via
+ a filter function. 'filter' is a string which will be used to
+ look up a filter_* method from this class.
+ """
+ result = defer.Deferred()
+ self.counters.getCounter(counterName).addCallback(
+ self._render_value, valueName, filter, result).addErrback(result.errback)
+ return result
+
+ def _render_value(self, counter, valueName, filter, result):
+ if counter:
+ value = counter.get(valueName, 0)
+ else:
+ value = 0
+ if filter:
+ value = getattr(self, 'filter_'+filter)(value)
+ result.callback(value)
+
+ def filter_date(self, value):
+ return TimeUtil.formatDate(value)
+
+ def filter_relativeDate(self, value):
+ return TimeUtil.formatDuration(time.time() - value)
+
+ def render_averagePeriod(self, context, counterName):
+ result = defer.Deferred()
+ self.counters.getCounter(counterName).addCallback(
+ self._render_averagePeriod, result).addErrback(result.errback)
+ return result
+
+ def _render_averagePeriod(self, counter, result):
+ if not counter:
+ result.callback('')
+ return
+ events = counter.get('eventCount', 0)
+ first = counter.get('firstEventTime')
+ if events < 2 or not first:
+ result.callback('')
+ return
+ result.callback([
+ ', for an average of ',
+ Template.value[ TimeUtil.formatDuration( (time.time() - first) / events ) ],
+ ' between messages',
+ ])
+
+
+class MessageDateColumn(Nouvelle.Column):
+ """A column that displays a message's date"""
+ heading = 'date'
+
+ def getValue(self, message):
+ try:
+ return XML.digValue(message.xml, int, "message", "timestamp")
+ except ValueError:
+ return None
+
+ def render_data(self, context, message):
+ value = self.getValue(message)
+ if value:
+ return TimeUtil.formatRelativeDate(value)
+ else:
+ return Template.error[ "Invalid Date" ]
+
+
+class MessageProjectColumn(Nouvelle.Column):
+ """A column that displays the project a message originated from"""
+ heading = 'project'
+
+ def getValue(self, message):
+ project = XML.dig(message.xml, "message", "source", "project")
+ if project:
+ return XML.shallowText(project)
+
+
+class MessageContentColumn(Nouvelle.Column):
+ """A column that displays a message, formatted in XHTML"""
+ heading = 'content'
+
+ def getValue(self, message):
+ try:
+ return Formatters.getFactory().findMedium('xhtml', message).formatMessage(message)
+ except:
+ return Template.error[str(sys.exc_info()[1])]
+
+
+class MessageList(Template.Section):
+ """A list of messages, with metadata and hyperlinks. Must be
+ constructed with a list of Message instances.
+ """
+ title = "messages"
+
+ columns = [
+ MessageDateColumn(),
+ MessageProjectColumn(),
+ MessageContentColumn(),
+ Nouvelle.AttributeColumn('link', 'hyperlink'),
+ ]
+
+ def renderMessages(self, context, messages):
+ return [
+ Template.Table(messages, self.columns,
+ defaultSortReversed = True,
+ id = 'message'),
+ ]
+
+
+class RecentMessages(MessageList):
+ """A section displaying recent messages from a given stats target"""
+ title = "recent messages"
+
+ def __init__(self, target, limit=20):
+ self.target = target
+ self.limit = limit
+
+ def render_rows(self, context):
+ parsed = []
+ for id, xml in self.target.messages.getLatest(self.limit):
+ m = Message.Message(xml)
+ m.hyperlink = Link.MessageLink(self.target, id, text="#")
+ parsed.append(m)
+
+ if parsed:
+ return self.renderMessages(context, parsed)
+ else:
+ return []
+
+
+class LinksSection(Template.Section):
+ """A section displaying useful links for a particular stats target. All links
+ are classes in the Link module that take only our stats target as a constructor argument.
+ We have a list of allowed links and default links, but the exact links we display may
+ be modified by the links-filter metadata key.
+
+ The links-filter format is simple- one entry per line, each entry consists of an
+ action and a link name regex, separated by whitespace. The action can be "+" to add the
+ link(s) or "-" to remove the link(s).
+ """
+ title = 'syndicate'
+
+ availableLinkNames = [
+ "RSSLink",
+ "RSSCustomizer",
+ "XMLLink",
+ ]
+
+ defaultLinkNames = [
+ "RSSLink",
+ "RSSCustomizer",
+ "XMLLink",
+ ]
+
+ def __init__(self, target):
+ self.target = target
+
+ def render_rows(self, context):
+ # First look for a links-filter metadata key for this target.
+ # The default is empty.
+ result = defer.Deferred()
+ self.target.metadata.getValue("links-filter", default="").addCallback(
+ self._render_rows, context, result
+ ).addErrback(result.errback)
+ return result
+
+ def _render_rows(self, linksFilter, context, result):
+ linkNames = list(self.defaultLinkNames)
+
+ for line in linksFilter.split("\n"):
+ line = line.strip()
+ if line:
+ action, name = line.split(None, 1)
+ regex = re.compile(name)
+
+ if action == "+":
+ for available in self.availableLinkNames:
+ if regex.match(available):
+ linkNames.append(available)
+
+ elif action == "-":
+ filtered = []
+ for name in linkNames:
+ if not regex.match(name):
+ filtered.append(name)
+ linkNames = filtered
+
+ result.callback([getattr(Link, linkName)(self.target) for linkName in linkNames])
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Browser.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Stats/Browser.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Catalog.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Stats/Catalog.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Stats/Catalog.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,112 @@
+""" LibCIA.Web.Stats.Catalog
+
+Implements the 'catalog' section in stats, pages, a fast way to
+get an overview of all stats targets directly under the current one.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.internet import defer
+from LibCIA.Web import Template
+from LibCIA import Database
+import Columns
+
+
+class CatalogSection(Template.Section):
+ """A Section displaying links to all children of a StatsTarget, with
+ other information about the children displayed as applicable.
+ """
+ title = "catalog"
+ limit = 100
+
+ query = """
+ SELECT
+ T.target_path,
+ ST.title,
+ ICO.path,
+ C_TODAY.event_count,
+ C_YESTERDAY.event_count,
+ C_FOREVER.event_count,
+ C_FOREVER.last_time,
+ COUNT(CHILD.target_path),
+ ICO.width,
+ ICO.height
+ FROM stats_catalog T
+ LEFT OUTER JOIN stats_catalog CHILD ON (CHILD.parent_path = T.target_path)
+ LEFT OUTER JOIN stats_statstarget ST ON (T.target_path = ST.path)
+ LEFT OUTER JOIN images_imageinstance ICO ON (ICO.source_id = IF(ST.icon_id IS NOT NULL, ST.icon_id, ST.photo_id) AND ICO.thumbnail_size = 32)
+ LEFT OUTER JOIN stats_counters C_TODAY ON (T.target_path = C_TODAY.target_path AND C_TODAY.name = 'today')
+ LEFT OUTER JOIN stats_counters C_YESTERDAY ON (T.target_path = C_YESTERDAY.target_path AND C_YESTERDAY.name = 'yesterday')
+ LEFT OUTER JOIN stats_counters C_FOREVER ON (T.target_path = C_FOREVER.target_path AND C_FOREVER.name = 'forever')
+ WHERE T.parent_path = %(path)s
+ GROUP BY
+ T.target_path,
+ ST.title,
+ ICO.path,
+ C_TODAY.event_count,
+ C_YESTERDAY.event_count,
+ C_FOREVER.event_count,
+ C_FOREVER.last_time
+ ORDER BY NULL LIMIT %(limit)s
+ """
+
+ columns = [
+ Columns.IndexedIconColumn(iconIndex=2, widthIndex=8, heightIndex=9),
+ Columns.TargetTitleColumn(pathIndex=0, titleIndex=1),
+ Columns.IndexedBargraphColumn('events today', 3),
+ Columns.IndexedBargraphColumn('events yesterday', 4),
+ Columns.IndexedBargraphColumn('total events', 5),
+ Columns.IndexedPercentColumn('% total', 5),
+ Columns.TargetLastEventColumn('last event', 6),
+ Columns.IndexedUnitColumn('contents', 7),
+ ]
+
+ def __init__(self, target):
+ self.target = target
+
+ def render_rows(self, context):
+ # First we run a big SQL query to gather all the data for this catalog.
+ # Control is passed to _render_rows once we have the query results.
+ result = defer.Deferred()
+ Database.pool.runQuery(self.query % {
+ 'path': Database.quote(self.target.path, 'varchar'),
+ 'limit': self.limit,
+ }).addCallback(
+ self._render_rows, context, result
+ ).addErrback(result.errback)
+ return result
+
+ def _render_rows(self, queryResults, context, result):
+ if queryResults:
+ content = [Template.Table(list(queryResults), self.columns,
+ id = 'catalog',
+ defaultSortColumnIndex = 1)]
+ if len(queryResults) == self.limit:
+ content.insert(0, Template.longError[
+ "This page has a very large number of child items, "
+ "and CIA can not yet display them all or browse them "
+ "incrementally. Below is an arbitrary set of %d "
+ "items. Sorry for the inconvenience, we're working "
+ "on resolving this issue." % self.limit
+ ])
+ result.callback(content)
+ else:
+ result.callback(None)
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Catalog.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Stats/Catalog.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Columns.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Stats/Columns.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Stats/Columns.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,193 @@
+""" LibCIA.Web.Stats.Columns
+
+A collection of Nouvelle.Column subclasses used
+to visualize tabular data in the stats browser.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from __future__ import division
+from LibCIA.Web import Template
+from LibCIA import Stats, TimeUtil
+import Nouvelle, time, math
+import Link
+
+
+class TargetTitleColumn(Nouvelle.Column):
+ """A Column displaying a target's title as a StatsLink. To avoid having to make
+ a query for every row in the table, this looks for the title to use in the
+ indicated SQL query column number. iconIndex should rever to the column
+ number with the metadata key to use for an icon, or None if the target has
+ no icon.
+ """
+ heading = 'title'
+ def __init__(self, pathIndex, titleIndex):
+ self.pathIndex = pathIndex
+ self.titleIndex = titleIndex
+
+ def _findTitle(self, row):
+ """Use our titleIndex to return the title according to the metadata if that
+ exists, otherwise make up a title based on the path.
+ """
+ title = row[self.titleIndex]
+ if title is None:
+ return Stats.Target.StatsTarget(row[self.pathIndex]).name
+ else:
+ return title
+ # Note that we don't have to worry about the case (name is None)
+ # because this is only used for listing children of some target,
+ # and the root target has no parent.
+
+ def getValue(self, row):
+ # For case-insensitive sorting by title
+ return self._findTitle(row).lower()
+
+ def render_data(self, context, row):
+ # Create a StatsLink. Note that we could leave it up to the StatsLink
+ # to look up the title, but that would end up creating an SQL query
+ # for each row in the table- not good when we're trying to view a page
+ # with 1000 projects without making our server explode.
+ target = Stats.Target.StatsTarget(row[self.pathIndex])
+ return Link.StatsLink(target, text=self._findTitle(row))
+
+
+class IndexedUnitColumn(Nouvelle.IndexedColumn):
+ """A Nouvelle.IndexedColumn that appends a fixed unit qualifier to each rendered item.
+ Hides individual cells when zero, hides the column when all cells are zero.
+ """
+ def __init__(self, heading, index, singularUnit='item', pluralUnit='items'):
+ self.singularUnit = singularUnit
+ self.pluralUnit = pluralUnit
+ Nouvelle.IndexedColumn.__init__(self, heading, index)
+
+ def isVisible(self, context):
+ return context['table'].reduceColumn(self, self._visibilityTest)
+
+ def _visibilityTest(self, seq):
+ """Visibility testing operator for this column's data"""
+ for item in seq:
+ if item != 0:
+ return True
+ return False
+
+ def render_data(self, context, row):
+ value = row[self.index]
+ if value == 0:
+ return ''
+ elif value == 1:
+ unit = self.singularUnit
+ else:
+ unit = self.pluralUnit
+ return "%s %s" % (row[self.index], unit)
+
+
+class RankIndexedColumn(Nouvelle.IndexedColumn):
+ """An IndexedColumn tweaked for numeric rankings.
+ None is treated as zero, sorting is reversed, the column
+ is hidden if all values are zero or less.
+ """
+ def cmp(self, a, b):
+ """Reverse sort"""
+ return -cmp(a[self.index],b[self.index])
+
+ def getValue(self, row):
+ return row[self.index] or 0
+
+ def isVisible(self, context):
+ return context['table'].reduceColumn(self, self._visibilityTest)
+
+ def _visibilityTest(self, seq):
+ """Visibility testing operator for this column's data"""
+ for item in seq:
+ if item != 0:
+ return True
+ return False
+
+
+class IndexedBargraphColumn(RankIndexedColumn):
+ """An IndexedColumn that renders as a logarithmic bar chart"""
+ def render_data(self, context, row):
+ value = row[self.index]
+ if not value:
+ return ''
+ logMax = math.log(context['table'].reduceColumn(self, max))
+ if logMax > 0:
+ fraction = math.log(value) / logMax
+ else:
+ fraction = 1
+ return Template.Bargraph(fraction)[ value ]
+
+
+class IndexedPercentColumn(RankIndexedColumn):
+ """An IndexedColumn that renders itself as a percent of the column's total"""
+ def render_data(self, context, row):
+ value = row[self.index]
+ if not value:
+ return ''
+ total = context['table'].reduceColumn(self, sum)
+ return "%.03f" % (value / total * 100)
+
+
+class IndexedIconColumn(RankIndexedColumn):
+ """An IndexedColumn that, if its value is nonzero, renders an
+ icon from the named metadata key, thumbnailed to the given max size.
+ """
+ def __init__(self, iconIndex, widthIndex, heightIndex, heading='icon'):
+ RankIndexedColumn.__init__(self, heading, iconIndex)
+ self.widthIndex = widthIndex
+ self.heightIndex = heightIndex
+
+ def render_data(self, context, row):
+ path = row[self.index]
+ if path:
+ return Nouvelle.tag('img', src = '/images/db/' + path,
+ width = row[self.widthIndex],
+ height = row[self.heightIndex])
+ else:
+ return ()
+
+
+class TargetLastEventColumn(Nouvelle.IndexedColumn):
+ """A Column displaying the amount of time since the last message was delivered to
+ each target, given a column containing the last event timestamp.
+ """
+ def getValue(self, row):
+ """Returns the number of seconds since the last event"""
+ lastTime = row[self.index]
+ if lastTime:
+ return time.time() - lastTime
+
+ def isVisible(self, context):
+ return context['table'].reduceColumn(self, self._visibilityTest)
+
+ def _visibilityTest(self, seq):
+ """Visibility testing operator for this column's data"""
+ for item in seq:
+ if item:
+ return True
+ return False
+
+ def render_data(self, context, target):
+ value = self.getValue(target)
+ if value is None:
+ return ''
+ else:
+ return "%s ago" % TimeUtil.formatDuration(self.getValue(target))
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Columns.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Stats/Columns.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Feed.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Stats/Feed.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Stats/Feed.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,530 @@
+""" LibCIA.Web.Stats.Feed
+
+Pages for getting real-time message feeds in RSS and unformatted XML
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.internet import defer
+from twisted.web import resource, server, http
+from twisted.python import log
+from LibCIA import Message, Formatters, TimeUtil, XML, Database
+import Nouvelle
+import Nouvelle.Twisted
+from Nouvelle import tag, place, xml, quote
+from LibCIA.Web import Template
+import Link
+import sys, traceback
+
+
+class BaseFeed(Nouvelle.Twisted.Page):
+ """Abstract base classes for XML message feeds, using Nouvelle
+ to render text/xml pages from a list of recent messages.
+ """
+ defaultLimit = 20
+ mimeType = 'text/xml'
+
+ def __init__(self, statsPage, limit=None):
+ if not limit:
+ limit = self.defaultLimit
+ self.limit = limit
+ self.statsPage = statsPage
+ self.target = statsPage.target
+ Nouvelle.Twisted.Page.__init__(self)
+
+ def preRender(self, context):
+ context['component'] = self.statsPage.component
+ context['request'].setHeader('content-type', self.mimeType)
+
+ def render(self, request):
+ """Intercept the normal rendering operations to check our
+ stats target's modification time. If the browser already
+ has a recent copy of this feed, we can get away without
+ rendering at all.
+ """
+ self.target.getMTime().addCallback(
+ self._render, request
+ ).addErrback(request.processingFailed)
+ return server.NOT_DONE_YET
+
+ def _render(self, mtime, request):
+ if (mtime is None) or (request.setLastModified(mtime) is not http.CACHED):
+ # We don't know the mtime or the browser's copy is no good, render as usual
+ Nouvelle.Twisted.Page.render(self, request)
+ else:
+ # Finish without rendering anything
+ request.finish()
+
+ def render_items(self, context):
+ """Renders the most recent commits as items in the feed"""
+ latest = list(self.target.messages.getLatest(self.limit))
+ latest.reverse()
+ return self.formatItems(latest, context)
+
+
+class FormattedFeed(BaseFeed):
+ """Abstract base classes that apply formatters to each message they include"""
+ def __init__(self, statsPage, limit=None, medium='xhtml'):
+ self.medium = medium
+ BaseFeed.__init__(self, statsPage, limit)
+
+ def formatMessage(self, m, medium=None):
+ """Format a message in the requested medium"""
+ if medium is None:
+ medium = self.medium
+
+ # Look for a proper format_* function to handle special feed-only media
+ f = getattr(self, 'format_' + medium, None)
+ try:
+ if f:
+ return f(m)
+ else:
+ # No special formatter, find a normal one
+ return Formatters.getFactory().findMedium(medium, m).formatMessage(m)
+ except Message.NoFormatterError:
+ return self.formatError()
+
+ def formatError(self):
+ """Subclasses should override this to generate error messages for formatMessage"""
+ return "(Unable to format message)"
+
+ def format_unquoted(self, m):
+ """Our 'unquoted' medium is plaintext inside an xml() marker to prevent quoting"""
+ return xml(self.formatMessage(m, 'plaintext'))
+
+
+class RSSFeed(FormattedFeed):
+ """An abstract base class for code shared between versions of the RSS format"""
+ def render_photo(self, context):
+ # First figure out if we have a photo. Actually render it in the Deferred if we do.
+ photo_query = """
+ SELECT IM.path
+ FROM stats_statstarget ST
+ LEFT OUTER JOIN images_imageinstance IM
+ ON (IM.source_id = ST.photo_id AND IM.thumbnail_size = 128)
+ WHERE ST.path = %s
+ """ % Database.quote(self.target.path, 'varchar')
+ result = defer.Deferred()
+ Database.pool.runQuery(photo_query).addCallback(
+ self._render_photo, context, result).addErrback(result.errback)
+ return result
+
+ def _render_photo(self, query_results, context, result):
+ if query_results and query_results[0][0]:
+ result.callback(tag('image')[
+ tag('url')[ '/images/db/' + query_results[0][0] ],
+ tag('title')[ place('title') ],
+ tag('link')[ place('link') ],
+ ])
+ else:
+ result.callback([])
+
+ def render_title(self, context):
+ return self.target.getTitle()
+
+ def render_link(self, context):
+ return self.target.metadata.getValue('url', Link.StatsLink(self.target).getURL(context))
+
+ def render_description(self, context):
+ return self.target.metadata.getValue('description', 'CIA Stats')
+
+
+class RSS2Feed(RSSFeed):
+ """A web resource representing an RSS 2.0 feed for a particular stats target,
+ constructed according to the spec at http://blogs.law.harvard.edu/tech/rss
+ """
+ def formatItems(self, messages, context):
+ items = []
+ for id, content in messages:
+ try:
+ items.append(tag('item')[self.messageToItemContent(context, Message.Message(content), id)])
+ except:
+ log.msg("Exception occurred in %s.formatItems\n%s" % (
+ self.__class__.__name__,
+ "".join(traceback.format_exception(*sys.exc_info()))))
+ return items
+
+ def messageToItemContent(self, context, m, id):
+ """Render an XML message as the content of an RSS <item>"""
+ url = Link.MessageLink(self.target, id).getURL(context)
+ tags = [
+ tag('pubDate')[ TimeUtil.formatDateRFC822(XML.digValue(m.xml, int, "message", "timestamp")) ],
+ tag('guid')[url],
+ tag('link')[url],
+ tag('description')[ quote(self.formatMessage(m)) ],
+ ]
+
+ # Generate a title if we can, but if we can't don't worry too much
+ try:
+ tags.append(tag('title')[ Formatters.getFactory().findMedium('title', m).formatMessage(m) ])
+ except Message.NoFormatterError:
+ pass
+
+ return tags
+
+ def render_cloud(self, context):
+ """Implements the first step of supporting the RSS 2.0 <cloud> tag.
+ This element describes how the RSS aggregator can subscribe for instant
+ notification of changes to this resource.
+ """
+ return tag('cloud',
+ domain = context['request'].getRequestHostname(),
+ port = context['request'].host[2],
+ path = '/RPC2',
+ protocol = 'xml-rpc',
+ registerProcedure = 'stats.subscribe.rss2',
+ )
+
+ document = [
+ xml('<?xml version="1.0"?>\n'),
+ tag('rss', version='2.0')[
+ tag('channel')[
+ tag('title')[ place('title') ],
+ tag('link')[ place('link') ],
+ tag('description')[ place('description') ],
+ place('photo'),
+ place('cloud'),
+ place('items'),
+ ],
+ ],
+ ]
+
+
+class RSS1Feed(RSSFeed):
+ """A web resource representing an RSS 1.0 feed for a particular stats target,
+ constructed according to the standard at http://web.resource.org/rss/1.0/spec
+
+ This uses mostly core RSS 1.0, with the dublin core module for datestamps.
+ """
+ def messageToItemContent(self, context, m, id):
+ return []
+
+ def formatItems(self, messages, context):
+ """This is called after we've retrieved a list of the target's recent messages.
+ Most of the document can't be formatted until this point, as we need to generate
+ both <item> elements and the <items> table of contents.
+ """
+ # Add our channel description to the content of our <rdf>
+ content = [ self.render_channel(context, messages) ]
+
+ # Add <item>s for each message
+ for id, messageContent in messages:
+ try:
+ content.append(self.render_item(context, id, messageContent))
+ except:
+ log.msg("Exception occurred in %s.formatItems\n%s" % (
+ self.__class__.__name__,
+ "".join(traceback.format_exception(*sys.exc_info()))))
+ return content
+
+ def render_channel(self, context, messages):
+ """Generate our <channel> element and all of its children"""
+ # Make a table of contents for the messages
+ toc = []
+ for id, content in messages:
+ url = Link.MessageLink(self.target, id).getURL(context)
+ toc.append(tag('rdf:li', resource=url))
+
+ targetUrl = Link.StatsLink(self.target).getURL(context)
+ return tag('channel', **{
+ 'rdf:about': targetUrl,
+ })[
+ tag('title')[ place('title') ],
+ tag('link')[ targetUrl ],
+ tag('description')[ place('description') ],
+ place('photo'),
+
+ tag('items')[
+ tag('rdf:Seq')[ toc ],
+ ],
+ ]
+
+ def render_item(self, context, id, content):
+ url = Link.MessageLink(self.target, id).getURL(context)
+ m = Message.Message(content)
+ tags = [
+ tag('link')[ url ],
+ tag('dc:date')[ TimeUtil.formatDateISO8601(XML.digValue(m.xml, int, "message", "timestamp")) ],
+ tag('description')[ quote(self.formatMessage(m)) ],
+ ]
+
+ # Generate a title if we can, but if we can't don't worry too much
+ try:
+ tags.append(tag('title')[ Formatters.getFactory().findMedium('title', m).formatMessage(m) ])
+ except Message.NoFormatterError:
+ pass
+
+ return tag('item', **{'rdf:about': url})[tags]
+
+ document = [
+ xml('<?xml version="1.0"?>\n'),
+ tag('rdf:RDF', **{
+ 'xmlns:rdf': "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
+ 'xmlns:dc': "http://purl.org/dc/elements/1.1/",
+ 'xmlns': "http://purl.org/rss/1.0/",
+ })[
+ place('items'),
+ ],
+ ]
+
+
+class XMLFeed(BaseFeed):
+ """A web resource representing a feed of unformatted XML commits for a stats target."""
+ def formatItems(self, messages, context):
+ return [self.formatItem(content) for id, content in messages]
+
+ def formatItem(self, content):
+ # Convert the root node, not the document- we don't want to
+ # be outputting another XML declaration inside our larger document.
+ return xml(XML.toString(content.childNodes[0]).encode('utf8'))
+
+ def render_metadata(self, context):
+ # Look up all the metadata first
+ result = defer.Deferred()
+ self.target.metadata.dict().addCallback(
+ self._render_metadata, context, result).addErrback(result.errback)
+ return result
+
+ def _render_metadata(self, metadict, context, result):
+ result.callback([self.renderMetadataItem(name, t[0], t[1], context)
+ for name, t in metadict.iteritems()])
+
+ def renderMetadataItem(self, name, value, mimeType, context):
+ """Render a single metadata item. If the content is short and in
+ a text format, we include it directly. Otherwise, just link to it.
+ XXX: These links don't really make sense any more, since the metadata
+ format changed.
+ """
+ valueTag = tag('value', _type=mimeType)[ str(value) ]
+ return tag('item', _name=name)[ valueTag ]
+
+ def render_counters(self, context):
+ # Look up all the counters first
+ result = defer.Deferred()
+ self.target.counters.dict().addCallback(
+ self._render_counters, context, result).addErrback(result.errback)
+ return result
+
+ def _render_counters(self, counterdict, context, result):
+ tags = []
+ for name, valueDict in counterdict.iteritems():
+ eventCount = valueDict.get('eventCount', 0)
+ try:
+ del valueDict['eventCount']
+ except KeyError:
+ pass
+ tags.append(tag('counter', _name = name, **valueDict)[ eventCount ])
+ result.callback(tags)
+
+ def render_statsLink(self, context):
+ return Link.StatsLink(self.target).getURL(context)
+
+ document = [
+ xml('<?xml version="1.0"?>\n'),
+ tag('statsTarget')[
+ tag('link')[ place('statsLink') ],
+ tag('counters')[ place('counters') ],
+ tag('metadata')[ place('metadata') ],
+ tag('recentMessages') [ place('items') ],
+ ],
+ ]
+
+
+class CustomizeRSS(Template.Page):
+ """A web page that lets the user generate a customized RSS feed for a particular
+ stats target. This can change the format, message style, number of messages, and such.
+ """
+ def __init__(self, statsPage):
+ Template.Page.__init__(self)
+ self.statsPage = statsPage
+
+ def parent(self):
+ return self.statsPage
+
+ def preRender(self, context):
+ context['component'] = self.statsPage.component
+
+ def render_subTitle(self, context):
+ return ["for ", self.statsPage.render_mainTitle(context)]
+
+ def render_form(self, context):
+ return tag('form',
+ action = Link.RSSLink(self.statsPage.target).getURL(context),
+ )[place('formContent')]
+
+ mainTitle = "Customized RSS"
+
+ leftColumn = [
+ Template.StaticSection('information')[
+ "This page is a form you can use to tweak everything tweakable about "
+ "the way CIA generates RSS feeds. After finding the settings you want, "
+ "the submission button at the bottom will redirect you to the customized "
+ "RSS feed."
+ ],
+ ]
+
+ mainColumn = [
+ Template.pageBody[ place('form') ],
+ ]
+
+ formContent = [
+ tag('h1')[ "RSS Format" ],
+ tag('p')[
+ "There are two current RSS format specifications. Both are named RSS, but "
+ "they are actually very different formats with different goals. RSS 2.0 is not "
+ "'newer' or 'better' than RSS 1.0 just because 2 is greater than 1; they are "
+ "just two separate specifications. CIA gives you the choice of either."
+ ],
+ tag('div', _class='formChoice')[
+ tag('input', _type='radio', value='2', _name='ver', checked='checked'),
+ tag('strong')[ " RSS 2.0 " ],
+ tag('p')[
+ "The default format. RSS 2.0 is simple, and has a publish/subscribe "
+ "system that can make it possible to receive updates immediately without "
+ "polling. Unfortunately, very few news aggregators currently implement "
+ "the <cloud> tag necessary for publish/subscribe. "
+ ],
+ tag('p')[
+ "The CIA server implements the <cloud> tag, so if you have a compatible "
+ "aggregator and a globally routable IP address you should see RSS feed "
+ "updates almost instantly. However, the only compatible aggregator we're "
+ "aware of at the moment is Radio Userland, and it is a commercial product. "
+ "The ability to receive RSS updates in real-time would go a long way toward "
+ "CIA's goals, so please let us know if you have seen other RSS aggregators "
+ "supporting the <cloud> tag."
+ ],
+ ],
+ tag('div', _class='formChoice')[
+ tag('input', _type='radio', value='1', _name='ver'),
+ tag('strong')[ " RSS 1.0 " ],
+ tag('p')[
+ "RSS 1.0 is more of an attempt to rethink RSS and design it with extensibility "
+ "in mind. It makes use of XML namespaces to provide a core set of functionality "
+ "along with 'modules' that can add domain-specific elements or attributes. RSS 1.0 "
+ "is based on the RDF (Resource Description Framework) W3C reccomendation, giving "
+ "it a rich and well-defined way to represent metadata. "
+ ],
+ tag('p')[
+ "CIA doesn't yet make use of an RSS 1.0 module to provide full metadata on commits. "
+ "Until an RSS 1.0 module for commit messages is designed, the best way to get all "
+ "possible information from CIA is to use the raw XML feeds."
+ ],
+ tag('p')[
+ "There was a 'changedpage' module in the works for RSS 1.0 that would have provided "
+ "features similar to the <cloud> element mentioned above for RSS 2.0, however the page "
+ "for it seems to have disappeared. Please contact us with more information if you have any."
+ ],
+ ],
+
+ tag('h1')[ "Messages" ],
+ tag('p')[
+ "This section controls which medium CIA tries to format messages in before embedding "
+ "them in the RSS feed. The default of XHTML is optimal, but alternatives are provided "
+ "if you need them."
+ ],
+ tag('div', _class='formChoice')[
+ tag('input', _type='radio', value='xhtml', _name='medium', checked='checked'),
+ tag('strong')[ " XHTML " ],
+ tag('p')[
+ "Format messages as XHTML with embedded CSS styles. This should "
+ "work and look good in most RSS aggregators. "
+ ],
+ ],
+ tag('div', _class='formChoice')[
+ tag('input', _type='radio', value='plaintext', _name='medium'),
+ tag('strong')[ " Plain Text " ],
+ tag('p')[
+ "Format messages in plain text, properly quoted for inclusion in RSS. "
+ "This is the preferred choice if your RSS aggregator runs on a text-only "
+ "console or can't handle HTML. "
+ ],
+ ],
+ tag('div', _class='formChoice')[
+ tag('input', _type='radio', value='unquoted', _name='medium'),
+ tag('strong')[ " Unquoted Text " ],
+ tag('p')[
+ "Format messages as plain text, but instead of quoting them twice (once "
+ "on account of the RSS feed being in XML, once because the content is "
+ "interpreted as HTML) this only quotes the text once. This should "
+ "only be used if your RSS aggregator is buggy and does not follow the "
+ "specification!"
+ ],
+ ],
+ tag('p')[
+ "You can optionally change the maximum number of messages a feed will contain "
+ "at once. Leave it blank to use the default of %s. There is no explicit upper limit, "
+ "But the database does store a finite number of messages for each stats target. "
+ "Please be reasonable. " % BaseFeed.defaultLimit
+ ],
+ tag('div', _class='formChoice')[
+ tag('p')[ "Retrieve at most: " ],
+ tag('p')[ tag('input', _type='text', _name='limit', size=10), " messages" ],
+ ],
+
+ tag('h1')[ "Your RSS Feed" ],
+ tag('p')[
+ "This button will now redirect you to an RSS feed with the settings above, "
+ "ripe for opening in your favorite RSS aggregator or copying and pasting somewhere useful."
+ ],
+ tag('p')[
+ tag('input', _type='submit', value='Get my customized RSS feed'),
+ ],
+ ]
+
+
+class RSSFrontend(resource.Resource):
+ """A web resource representing an RSS feed, the format and content of which depends
+ on parameters passed to us. Children are supported- for now this includes the
+ 'customize' page that helps you build RSS URLs with non-default options.
+ """
+ def __init__(self, statsPage):
+ resource.Resource.__init__(self)
+ self.statsPage = statsPage
+ self.putChild('customize', CustomizeRSS(statsPage))
+
+ def render(self, request):
+ # Check the requested RSS version
+ version = request.args.get('ver')
+ if version and version[-1]:
+ # Pick a feed factory based on the RSS version
+ factory = {
+ '1': RSS1Feed,
+ '2': RSS2Feed,
+ }[version[-1]]
+ else:
+ # Use RSS 2 by default
+ factory = RSS2Feed
+
+ # The rest of the arguments get transformed into keyword args
+ kwargs = {}
+
+ # Check the requested message medium, defaulting to XHTML
+ medium = request.args.get('medium')
+ if medium and medium[-1]:
+ kwargs['medium'] = medium[-1]
+
+ # Get the message limit, with BaseFeed's default
+ limit = request.args.get('limit')
+ if limit and limit[-1]:
+ kwargs['limit'] = int(limit[-1])
+
+ # Now construct the proper feed object and render it
+ return factory(self.statsPage, **kwargs).render(request)
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Feed.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Stats/Feed.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Graph.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Stats/Graph.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Stats/Graph.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,242 @@
+""" LibCIA.Web.Stats.Graph
+
+Implements web interfaces based on the stats_relations graph.
+This includes the 'related' section on all stats targets, and
+visually graphing the relationships between stats targets.
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.internet import defer
+from LibCIA.Web import Template
+from LibCIA import Stats, Database
+from Nouvelle import tag
+import Nouvelle
+from twisted.web import resource, server
+import LibCIA.Stats.Graph
+import Link, Columns
+
+
+class RelatedTable(Nouvelle.BaseTable):
+ tableTag = tag('table', _class='related')
+
+
+class RelatedFilter(Database.Filter):
+ """An XML filter that can control what's displayed in the 'related' box
+ on a per-stats-target basis. These filters come from stats metadata,
+ so we need to restrict the available SQL variables.
+ """
+ def varLookup(self, var):
+ # Map from some abstract variable names to concrete variable names.
+ # The returned variables must match the ones in RelatedSection.query below.
+ return {
+ 'parent_path': 'C.parent_path',
+ 'target_path': 'C.target_path',
+ }[var]
+
+
+class RelatedSection(Template.Section):
+ """A section showing links to related stats targets. This works by looking for
+ nodes connected to this one in the stats_relations graph. The paths and
+ titles of related nodes are fetched using one SQL query, for efficiency.
+
+ The query sorts first by parent path, so we can extract each section in one
+ piece, and second in descending order by freshness. This way, instead of
+ constantly having the strongest associations at the top, every time an
+ association is reinforced it pops up to the top, showing our visitors what's
+ cool and hip.
+ """
+ title = 'related'
+
+ query = """
+ SELECT
+ C.parent_path,
+ R.freshness,
+ PARENT_ST.title,
+ C.target_path,
+ ST.title,
+ ICO.path,
+ ICO.width,
+ ICO.height
+ FROM stats_relations R
+ LEFT OUTER JOIN stats_catalog C
+ ON (C.target_path = target_%(otherSide)s_path)
+ LEFT OUTER JOIN stats_statstarget ST ON (C.target_path = ST.path)
+ LEFT OUTER JOIN images_imageinstance ICO ON (ICO.source_id = IF(ST.icon_id IS NOT NULL, ST.icon_id, ST.photo_id) AND ICO.thumbnail_size = 32)
+ LEFT OUTER JOIN stats_statstarget PARENT_ST ON (C.parent_path = PARENT_ST.path)
+ WHERE R.target_%(thisSide)s_path = %(path)s
+ AND C.parent_path != %(path)s
+ AND %(filter)s
+ ORDER BY NULL
+ """
+
+ sectionLimit = 15
+
+ columns = [
+ Columns.IndexedIconColumn(iconIndex=4, widthIndex=5, heightIndex=6),
+ Columns.TargetTitleColumn(pathIndex=2, titleIndex=3),
+ ]
+
+ def __init__(self, target):
+ self.target = target
+
+ def makeLink(self, path, title):
+ """Link to a stats target when we already know the title"""
+ target = Stats.Target.StatsTarget(path)
+ if title is None:
+ title = target.name
+ return Link.StatsLink(target, text=title)
+
+ def render_rows(self, context):
+ # First look for a related-filter metadata key for this target.
+ # The default allows all related items to be displayed.
+ result = defer.Deferred()
+ self.target.metadata.getValue("related-filter", default="<true/>").addCallback(
+ self._startQuery, context, result
+ ).addErrback(result.errback)
+ return result
+
+ def _startQuery(self, filter, context, result):
+ # We have our related-filter, start a db interaction to do our actual
+ # queries then pass the results on to _render_rows.
+ Database.pool.runInteraction(self._runQuery, filter).addCallback(
+ self._render_rows, context, result
+ ).addErrback(result.errback)
+
+ def _runQuery(self, cursor, filter):
+ # Set up and run two SQL queries, one for each side of the graph link that this
+ # target may be on. We can't do this in one step and still have the server use
+ # its indexes effectively.
+
+ filterSql = RelatedFilter(filter).sql
+ sections = {}
+
+ for thisSide, otherSide in ( ('a','b'), ('b', 'a') ):
+ cursor.execute(self.query % dict(
+ path = Database.quote(self.target.path, 'varchar'),
+ filter = filterSql,
+ thisSide = thisSide,
+ otherSide = otherSide,
+ ))
+
+ while 1:
+ row = cursor.fetchone()
+ if not row:
+ break
+ # Drop rows into sections according to their parent path
+ try:
+ sections[row[0]].append(row[1:])
+ except KeyError:
+ sections[row[0]] = [row[1:]]
+
+ # Sort sections descending by freshness
+ for items in sections.itervalues():
+ items.sort()
+ items.reverse()
+ return sections
+
+ def _render_rows(self, queryResults, context, result):
+ # Sort sections by decreasing size. We want the most interesting ones at
+ # the top, and those are usually the biggest.
+ sections = queryResults.keys()
+ sections.sort(lambda a,b: cmp(len(queryResults[b]), len(queryResults[a])))
+ result.callback([self.render_section(section, queryResults[section]) for section in sections])
+
+ def render_section(self, section, rows):
+ """Given a heading renderable and a list of rows for that
+ heading, render one section of the 'related' box.
+ """
+ # Truncate the rows if we need to
+ if len(rows) > self.sectionLimit:
+ footer = tag('div', _class='relatedFooter')[
+ '(%d others)' % (len(rows) - self.sectionLimit)
+ ]
+ rows = rows[:self.sectionLimit]
+ else:
+ footer = ()
+
+ parentTitle = rows[0][1]
+ return [
+ tag('div', _class='relatedHeading')[ self.makeLink(section, parentTitle) ],
+ RelatedTable(rows, self.columns, showHeading=False),
+ footer,
+ ]
+
+
+class GraphPage(resource.Resource):
+ """Implements a web resource that generates rasterized and cached
+ graphs from the stats_relations table.
+ """
+ def __init__(self, *args, **kwargs):
+ self.grapher = Stats.Graph.RelationGrapher(*args, **kwargs)
+
+ def render(self, request):
+ """Figure out what format the user wants this graph in, and
+ call one of our format_* functions.
+
+ Each format function must return an object with a render(f)
+ method that writes the finished page to a file-like object
+ and returns a Deferred signalling completion.
+
+ The format function probably also should be setting the
+ content-type header to an appropriate value.
+ """
+ format = request.args.get('format', ['png'])[0]
+ render = getattr(self, 'format_'+format)(request).render
+ render(request).addCallback(
+ self._renderDone, request).addErrback(
+ request.processingFailed)
+ return server.NOT_DONE_YET
+
+ def _renderDone(self, x, request):
+ request.finish()
+
+ def format_dot(self, request):
+ """Return the graph in its original .dot source format"""
+ request.setHeader('content-type', 'text/plain')
+ return Stats.Graph.RenderCache(self.grapher)
+
+ def format_svg(self, request):
+ """Perform graph layout but not rasterization, and return the SVG"""
+ request.setHeader('content-type', 'image/svg+xml')
+ graphCache = Stats.Graph.RenderCache(self.grapher)
+ layout = Stats.Graph.GraphLayout(graphCache)
+ return Stats.Graph.RenderCache(layout)
+
+ def format_png(self, request, defaultWidth=600, maxWidth=2000):
+ """Return a fully rasterized graph image at a specified
+ size, with caching at every interesting stage.
+ """
+ # Let our user specify a width, up to a preset maximum.
+ # Height will follow automatically to keep the aspect ratio correct.
+ width = min(int(request.args.get('width', [defaultWidth])[0]), maxWidth)
+
+ # Let the user specify a background color, or the empty string
+ # to leave the background transparent.
+ bg = request.args.get('bg', ['white'])[0]
+ if not bg:
+ bg = None
+
+ request.setHeader('content-type', 'image/png')
+ graphCache = Stats.Graph.RenderCache(self.grapher)
+ layoutCache = Stats.Graph.RenderCache(Stats.Graph.GraphLayout(graphCache))
+ raster = Stats.Graph.SvgRasterizer(layoutCache, width=width, background=bg)
+ return Stats.Graph.RenderCache(raster)
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Graph.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Stats/Graph.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Link.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Stats/Link.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Stats/Link.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,147 @@
+""" LibCIA.Web.Stats.Link
+
+Classes for forming hyperlinks between stats browser pages
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from Nouvelle import tag
+from urllib import quote
+from LibCIA.Web import Template
+import posixpath
+
+
+class TargetRelativeLink:
+ """Abstract base class for a link to a stats target or something relative to it"""
+ def __init__(self, target, relativePathSegments=()):
+ self.target = target
+ self.relativePathSegments = tuple(relativePathSegments)
+
+ def getURL(self, context):
+ # Make this an absolute URL- currently it's required for
+ # links placed in the RSS and XML feeds, and won't
+ # hurt elsewhere.
+ req = context['request']
+ port = req.host[2]
+ hostname = req.getRequestHostname()
+ if req.isSecure():
+ default = 443
+ else:
+ default = 80
+ if port == default:
+ hostport = ''
+ else:
+ hostport = ':%d' % port
+ path = posixpath.join('/stats',
+ *(tuple(self.target.pathSegments) + self.relativePathSegments))
+ return quote('http%s://%s%s%s' % (
+ req.isSecure() and 's' or '',
+ hostname,
+ hostport,
+ path), "/:")
+
+
+class StatsLink(TargetRelativeLink):
+ """An anchor tag linking to the given stats target.
+ Text for the link may be specified, but by default the
+ target's title is used.
+ """
+ def __init__(self, target, tagFactory=tag('a'), text=None):
+ TargetRelativeLink.__init__(self, target)
+ self.tagFactory = tagFactory
+ self.text = text
+
+ def render(self, context):
+ text = self.text
+ if text is None:
+ text = self.target.getTitle()
+ return self.tagFactory(href=self.getURL(context))[text]
+
+
+class MessageLink(TargetRelativeLink):
+ """A link to a particular message delivered to a stats target"""
+ def __init__(self, target, id, extraSegments=(), tagFactory=tag('a'), text=None):
+ TargetRelativeLink.__init__(self, target, ('.message', "%x" % id) + extraSegments)
+ self.tagFactory = tagFactory
+ self.text = text
+
+ def render(self, context):
+ return self.tagFactory(href=self.getURL(context))[self.text]
+
+
+class MetadataLink(TargetRelativeLink):
+ """An anchor tag linking to an item in the given stats target's metadata.
+ Text for the link may be specified, but by default the key is used.
+
+ This class only works for keys that are strings. A key of None links
+ to the metadata index.
+ """
+ def __init__(self, target, key=None, tagFactory=tag('a'), text=None):
+ segments = ['.metadata']
+ if key:
+ segments.append(key)
+ TargetRelativeLink.__init__(self, target, segments)
+
+ self.tagFactory = tagFactory
+ self.key = key
+ self.text = text
+
+ def render(self, context):
+ text = self.text
+ if text is None:
+ if self.key is None:
+ text = "View/Edit Metadata"
+ else:
+ text = self.key
+ return self.tagFactory(href=self.getURL(context))[text]
+
+
+class RSSLink(TargetRelativeLink):
+ """An anchor tag linking to the default RSS feed for a particular stats target"""
+ def __init__(self, target, text="RSS 2.0 Feed", extraSegments=()):
+ TargetRelativeLink.__init__(self, target, ('.rss',) + extraSegments)
+ self.text = text
+
+ def render(self, context):
+ return Template.SubscriptionLink(self.getURL(context), self.text)
+
+
+class RSSCustomizer(RSSLink):
+ """An anchor tag leading to a page that lets the user customize the generated RSS"""
+ def __init__(self, target, text="Customized RSS"):
+ RSSLink.__init__(self, target, text, ('customize',))
+
+ def render(self, context):
+ return tag('a', href=self.getURL(context))[self.text]
+
+
+class XMLLink(TargetRelativeLink):
+ """An anchor tag linking to the XML feed for a given stats target"""
+ def __init__(self, target, tagFactory=tag('a'), text=None):
+ TargetRelativeLink.__init__(self, target, ('.xml',))
+ self.tagFactory = tagFactory
+ self.text = text
+
+ def render(self, context):
+ text = self.text
+ if text is None:
+ text = "Unformatted XML"
+ return self.tagFactory(href=self.getURL(context))[text]
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Link.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Stats/Link.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Stats/MessageViewer.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Stats/MessageViewer.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Stats/MessageViewer.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,250 @@
+""" LibCIA.Web.Stats.MessageViewer
+
+An interface for viewing the individual messages stored by the stats subsystem
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.web import resource, error, server
+from twisted.internet import defer
+from LibCIA.Web import Template, Info
+from Nouvelle import tag, place
+from LibCIA import Formatters, Message, XML, TimeUtil
+import Link
+
+
+class RootPage(resource.Resource):
+ """A page that doesn't generate any interesting output, but whose children are message IDs"""
+ def __init__(self, statsPage):
+ self.statsPage = statsPage
+ resource.Resource.__init__(self)
+
+ def render(self, request):
+ return error.NoResource("There's no index here, you need a message ID").render(request)
+
+ def getChildWithDefault(self, name, request):
+ if not name:
+ # Ignore empty path sections
+ return self
+ else:
+ try:
+ return MessagePage(self.statsPage, int(name, 16))
+ except ValueError:
+ return error.NoResource("Message ID is invalid")
+
+
+class EnvelopeSection(Template.Section):
+ """A section displaying general information contained in a message's envelope"""
+ title = 'envelope'
+
+ def __init__(self, target, message):
+ self.target = target
+ self.message = message
+
+ def render_rows(self, context):
+ if not self.message:
+ return []
+ rows = []
+
+ timestamp = XML.dig(self.message.xml, "message", "timestamp")
+ if timestamp:
+ rows.append(self.format_timestamp(timestamp))
+
+ generator = XML.dig(self.message.xml, "message", "generator")
+ if generator:
+ rows.append(self.format_generator(generator))
+
+ return rows
+
+ def format_generator(self, gen):
+ """Format the information contained in this message's <generator> tag"""
+ name = XML.digValue(gen, str, "name")
+ url = XML.digValue(gen, str, "url")
+ version = XML.digValue(gen, str, "version")
+
+ if url:
+ name = tag('a', href=url)[ name ]
+ items = ["Generated by ", Template.value[ name ]]
+ if version:
+ items.extend([" version ", version])
+ return items
+
+ def format_timestamp(self, stamp):
+ return ["Received ", Template.value[TimeUtil.formatRelativeDate(int(XML.shallowText(stamp)))]]
+
+
+class LinksSection(Template.Section):
+ """A section displaying useful links for a particular message"""
+ title = 'links'
+
+ def __init__(self, target, messageId):
+ self.target = target
+ self.messageId = messageId
+
+ def render_rows(self, context):
+ return [
+ Link.MessageLink(self.target, self.messageId,
+ extraSegments = ('xml',),
+ text = 'Unformatted XML',
+ ),
+ ]
+
+
+class UnformattedMessagePage(resource.Resource):
+ """A page that sends back the raw XML text of a particular message"""
+ def __init__(self, target, id):
+ self.target = target
+ self.id = id
+ resource.Resource.__init__(self)
+
+ def render(self, request):
+ xml = self.target.messages.getMessageById(self.id)
+ if xml:
+ request.setHeader('content-type', 'text/xml')
+ request.write(unicode(XML.toString(xml)).encode('utf-8'))
+ request.finish()
+ else:
+ request.write(error.NoResource("Message not found").render(request))
+ request.finish()
+ return ""
+
+
+class HTMLPrettyPrinter(XML.XMLObjectParser):
+ """An object parser that converts arbitrary XML to pretty-printed
+ representations in the form of Nouvelle-serializable tag trees.
+ """
+ def parseString(self, s):
+ s = s.strip()
+ if s:
+ return tag('p', _class='xml-text')[ s ]
+ else:
+ return ()
+
+ def unknownElement(self, element):
+ # Format the element name and attributes
+ elementName = tag('span', _class="xml-element-name")[ element.nodeName ]
+ elementContent = [ elementName ]
+ for attr in element.attributes.values():
+ elementContent.extend([
+ ' ',
+ tag('span', _class='xml-attribute-name')[ attr.name ],
+ '="',
+ tag('span', _class='xml-attribute-value')[ attr.value ],
+ '"',
+ ])
+
+ # Now the contents...
+ if element.hasChildNodes():
+ completeElement = [
+ "<", elementContent, ">",
+ tag('blockquote', _class='xml-element-content')[
+ [self.parse(e) for e in element.childNodes],
+ ],
+ "</", elementName, ">",
+ ]
+ else:
+ completeElement = ["<", elementContent, "/>"]
+
+ return tag('div', _class='xml-element')[ completeElement ]
+
+htmlPrettyPrint = HTMLPrettyPrinter().parse
+
+
+class MessagePage(Template.Page):
+ """A page that views one message from the stats database"""
+ mainTitle = 'Archived Message'
+
+ def __init__(self, statsPage, id):
+ self.statsPage = statsPage
+ self.id = id
+ Template.Page.__init__(self)
+ self.putChild('xml', UnformattedMessagePage(self.statsPage.target, self.id))
+
+ def parent(self):
+ return self.statsPage
+
+ def preRender(self, context):
+ # Load the message once, so multiple components can share it
+ xml = self.statsPage.target.messages.getMessageById(self.id)
+ if xml:
+ self.message = Message.Message(xml)
+ else:
+ self.message = None
+
+ context['component'] = self.statsPage.component
+
+ def render_subTitle(self, context):
+ return ["for ",
+ self.statsPage.render_mainTitle(context)]
+
+ def render_message(self, context):
+ if not self.message:
+ context['request'].setResponseCode(404)
+ return self.notFoundMessage
+
+ # Try to format it using several media, in order of decreasing preference.
+ # The 'xhtml-long' formatter lets messages define a special formatter to
+ # use when an entire page is devoted to their one message, possibly showing
+ # it in greater detail. 'xhtml' is the formatter most messages should have.
+ # 'plaintext' is a nice fallback.
+ #
+ # This default list of media to try can be overridden with an argument in our URL.
+
+ if 'media' in context['args']:
+ mediaList = context['args']['media'][0].split()
+ else:
+ mediaList = ('xhtml-long', 'xhtml', 'plaintext')
+
+ for medium in mediaList:
+ try:
+ formatted = Formatters.getFactory().findMedium(
+ medium, self.message).formatMessage(self.message)
+ except Message.NoFormatterError:
+ continue
+ return formatted
+
+ # Still no luck? Display a warning message and a pretty-printed XML tree
+ return [
+ tag('h1')[ "No formatter available" ],
+ XML.htmlPrettyPrint(self.message.xml),
+ ]
+
+ def render_leftColumn(self, context):
+ return [
+ EnvelopeSection(self.statsPage.target, self.message),
+ LinksSection(self.statsPage.target, self.id),
+ Info.Clock(),
+ ]
+
+ notFoundMessage = [
+ tag('h1')[ "Not Found" ],
+ tag('p')[
+ "This message was not found in our database. The number could be "
+ "incorrect, or the message may be old enough that it was deleted "
+ "from the database. Each stats target archives a fixed number of messages, "
+ "so you might be able to find another copy of it on a different stats "
+ "target with less traffic."
+ ]
+ ]
+
+ mainColumn = [
+ Template.pageBody[ place('message') ],
+ ]
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Stats/MessageViewer.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Stats/MessageViewer.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Metadata.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Stats/Metadata.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Stats/Metadata.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,103 @@
+""" LibCIA.Web.Stats.Metadata
+
+Viewers and editors for the metadata associated with each stats target
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from twisted.internet import defer
+from LibCIA.Web import Template
+from LibCIA import Units, Database
+from Nouvelle import tag
+
+
+class Info(Template.Section):
+ """A section that displays a StatsTarget's miscellaneous metadata"""
+ title = "information"
+
+ def __init__(self, target):
+ self.target = target
+ self.metadata = target.metadata
+
+ def render_rows(self, context):
+ photo_query = """
+ SELECT IM.path, IM.width, IM.height
+ FROM stats_statstarget ST
+ LEFT OUTER JOIN images_imageinstance IM
+ ON (IM.source_id = ST.photo_id AND IM.thumbnail_size = 256)
+ WHERE ST.path = %s
+ """ % Database.quote(self.target.path, 'varchar')
+
+ # XXX: This is hacky. Search for exclusive owners of this target.
+ owner_query = """
+ SELECT UA.id, UA.access, UA.user_id
+ FROM stats_statstarget ST
+
+ LEFT OUTER JOIN accounts_project PROJ ON (PROJ.target_id = ST.id)
+ LEFT OUTER JOIN accounts_author AUTH ON (AUTH.target_id = ST.id)
+
+ LEFT OUTER JOIN django_content_type CT_AUTH
+ ON (CT_AUTH.app_label = 'accounts' AND CT_AUTH.model = 'author')
+ LEFT OUTER JOIN django_content_type CT_PROJ
+ ON (CT_PROJ.app_label = 'accounts' AND CT_PROJ.model = 'project')
+
+ LEFT OUTER JOIN accounts_userasset UA
+ ON ( (UA.content_type_id = CT_AUTH.id AND UA.object_id = AUTH.id)
+ OR (UA.content_type_id = CT_PROJ.id AND UA.object_id = PROJ.id))
+
+ WHERE ST.path = %s AND UA.access > 1
+ """ % Database.quote(self.target.path, 'varchar')
+
+ # Grab the metadata keys we'll need and wait for them to become available
+ result = defer.Deferred()
+ defer.gatherResults([
+ self.metadata.getValue('url'),
+ self.metadata.getValue('description'),
+ Database.pool.runQuery(photo_query),
+ Database.pool.runQuery(owner_query),
+ ]).addCallback(self._render_rows, context, result).addErrback(result.errback)
+ return result
+
+ def _render_rows(self, metadata, context, result):
+ url, description, photo_results, owner_results = metadata
+ rows = []
+ if url:
+ rows.append(tag('a', href=url)[url])
+ if photo_results and photo_results[0][0]:
+ path, width, height = photo_results[0]
+ rows.append(Template.Photo('/images/db/' + path, width=width, height=height))
+ if description:
+ rows.append(description)
+
+ # XXX: This is kind of a hack, but it should improve usability
+ # between the old and new sites. Show 'edit' links if
+ # this is something users can edit (projects or authors)
+ # and if it isn't claimed exclusively already.
+
+ if (not owner_results and
+ len(self.target.pathSegments) >= 2 and
+ self.target.pathSegments[0] in ('project', 'author')):
+ rows.append(tag('a', href='/account/%ss/add/%s/' % (
+ self.target.pathSegments[0], '/'.join(self.target.pathSegments[1:])))
+ ["Edit..."])
+
+ result.callback(rows)
+
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Stats/Metadata.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Stats/Metadata.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Stats/__init__.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Stats/__init__.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Stats/__init__.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,28 @@
+""" LibCIA.Web.Stats
+
+Implements the web interface to CIA's stats subsystem
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+# Convenience imports
+import Browser
+from Browser import Component
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Stats/__init__.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Stats/__init__.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/Template.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/Template.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/Template.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,451 @@
+""" LibCIA.Web.Template
+
+Template classes for building web pages using our particular style
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import Nouvelle
+import Nouvelle.Twisted
+from twisted.internet import defer
+from Nouvelle import tag, place, xml, subcontext
+import types, random
+
+# Verify we have a new enough Nouvelle
+if (not hasattr(Nouvelle, "version_info")) or Nouvelle.version_info < (0, 92, 1):
+ raise Exception("The installed copy of Nouvelle is too old")
+
+# Tag templates
+catalogList = tag('ul', _class="catalog")
+value = tag('strong')
+error = tag('span', _class="error")
+longError = tag('p', _class="error")
+unableToFormat = error[ "Unable to format data" ]
+breadcrumbSeparator = xml(" » ")
+pageBody = tag('div', _class="pageBody")
+
+# Stock images
+fileIcon = tag('img', src="/images/bullet.png", _class="left-icon", width=9, height=9, alt="file")
+dirIcon = tag('img', src="/images/folder.png", _class="left-icon", width=14, height=12, alt="directory")
+
+
+def Photo(url, **attrs):
+ """A factory for images presented as a photo"""
+ return tag('div', _class='photo')[ tag('img', _class='photo', src=url, alt="Photo", **attrs) ]
+
+
+def Bargraph(value, width=4, padding=0.2):
+ """A factory for tags that use their size to express a value between 0 and 1"""
+ return tag('span', _class='bargraph',
+ style="padding: 0em %.4fem" % (value * width + padding))
+
+
+def SubscriptionLink(url, content, icon="/images/rss.png", iconSize=(36,14)):
+ """An anchor tag that can be used to link to RSS feeds.
+ """
+ return tag('a', href = url)[
+ tag('img', src=icon, _class="left-icon", alt="RSS",
+ width=iconSize[0], height=iconSize[1]),
+ content,
+ ]
+
+
+def SectionGrid(*rows):
+ """Create a grid of sections, for layouts showing a lot of small boxes
+ in a regular pattern.
+ """
+ return tag('table', _class="sectionGrid")[[
+ tag('tr', _class="sectionGrid")[[
+ tag('td', _class="sectionGrid")[
+ cell
+ ] for cell in row
+ ]] for row in rows
+ ]]
+
+
+def MessageHeaders(d):
+ """A factory for displaying message headers from a dictionary-like object.
+ If order is important (it probably is) use twisted.python.util.OrderedDict.
+ """
+ return tag('table', _class="messageHeaders")[[
+ tag('tr')[
+ tag('td', _class='name')[ name, ":" ],
+ tag('td', _class='value')[ value ],
+ ]
+ for name, value in d.iteritems()
+ ]]
+
+
+def randomlySubdivide(string, minLength, maxLength):
+ """Randomly break a string into pieces not smaller than minLength
+ or longer than maxLength.
+ """
+ parts = []
+ while string:
+ l = random.randint(minLength, maxLength)
+ parts.append(string[:l])
+ string = string[l:]
+ return parts
+
+
+def treeReplace(tree, a, b):
+ """Replace any occurrance of 'a' with 'b' recursively in a tree of strings"""
+ if type(tree) in (list, tuple):
+ return [treeReplace(i, a, b) for i in tree]
+ elif type(tree) in (str, unicode):
+ return tree.replace(a, b)
+ else:
+ return tree
+
+
+class EmailLink:
+ """This is a tag-like class for generating obfuscated links to email addresses,
+ using the javascript-based technique described by Kieth Bell at:
+ http://www.december14.net/ways/js/nospam.shtml
+ """
+ def __init__(self, href):
+ self.href = href
+ self.content = []
+
+ def __getitem__(self, content):
+ self.content = content
+ return self
+
+ def render(self, context):
+ # Obfuscate the content
+ content = treeReplace(self.content, "@", " at ")
+ content = treeReplace(content, ".", " dot ")
+
+ # Create an 'onmouseover' script that will replace our link's
+ # href with the correct one. We start by randomly dividing the
+ # email address into small chunks.
+ parts = randomlySubdivide(self.href, 1, 5)
+ script = "this.href=" + "+".join(["'%s'" % part for part in parts])
+
+ # Assemble and render the final link
+ return tag('a', href="/doc/mail", onmouseover=script)[ content ].render(context)
+
+
+def FileTree(tree):
+ """A factory for rendering file trees from a tree of nested dictionaries.
+ Dictionaries with no children are treated as files, dictionaries with
+ children are treated as directories.
+ """
+ keys = tree.keys()
+ keys.sort()
+ items = []
+ for key in keys:
+ if not key:
+ # This can happen when there's a trailing slash, for example because a new directory
+ # was added. (in clients that are smart enough to detect that) Ignore it here for now.
+ continue
+
+ if tree[key]:
+ items.append( tag('li', _class='directory')[ key, FileTree(tree[key]) ])
+ else:
+ items.append( tag('li', _class='file')[ key ])
+
+ return tag('ul', _class='fileTree')[ items ]
+
+
+class Section:
+ """A renderable portion of the web page with a title and a body,
+ that may be placed in any of the page's columns.
+
+ A section can be made up of any number of 'rows', which manifest
+ themselves as whitespace-padded sections. A section without any
+ rows is completely hidden.
+ """
+ def render_title(self, context):
+ return self.title
+
+ def render_rows(self, context):
+ return self.rows
+
+ def render(self, context):
+ result = defer.Deferred()
+ defer.maybeDeferred(self.render_rows, context).addCallback(
+ self._render, result).addErrback(result.errback)
+ # Optimize out Deferreds where we can
+ if result.called:
+ return result.result
+ return result
+
+ def _render(self, rows, result):
+ """The backend for render(), called once our rows list is known"""
+ if rows:
+ result.callback(subcontext(owner=self)[
+ tag('span', _class="section")[ place("title") ],
+ tag('div', _class="section")[
+ tag('div', _class="sectionTop")[" "],
+ [tag('div', _class="row")[r] for r in rows],
+ ],
+ ])
+ else:
+ result.callback([])
+
+
+class Table(Nouvelle.ResortableTable):
+ """Add sorting indicators to Nouvelle's normal ResortableTable"""
+ reversedSortIndicator = tag('img', _class='sortIndicator', width=11, height=7,
+ src="/images/sort_up.png", alt="Reversed sort column")
+ sortIndicator = tag('img', _class='sortIndicator', width=11, height=7,
+ src="/images/sort_down.png", alt="Sort column")
+
+
+class HideFromSpiders:
+ """Hides its contents when isWebSpider is true. This can be used to show
+ content that shouldn't be indexed by search engines if possible.
+ """
+ def __init__(self):
+ self.content = []
+
+ def __getitem__(self, content):
+ self.content = content
+ return self
+
+ def render(self, context):
+ if context['request'].isWebSpider():
+ return []
+ else:
+ return self.content
+
+
+class StaticSection(Section):
+ """A section containing static content, usable with tag-like syntax:
+ StaticSection(title)[body]
+ """
+ def __init__(self, title=None, rows=None):
+ if rows is None:
+ # This is important- remember that default values are only created
+ # once and reused forever. A list specified as a default value would
+ # be shared by all instances that don't specify their own list.
+ rows = []
+ self.title = title
+ self.rows = rows
+
+ def __call__(self, title=None, rows=None):
+ n = StaticSection()
+ if title is None:
+ n.title = self.title
+ else:
+ n.title = title
+ if rows is None:
+ n.rows = list(self.rows)
+ else:
+ n.rows = rows
+ return n
+
+ def __getitem__(self, rows):
+ return self.__class__(self.title, [rows])
+
+
+class Page(Nouvelle.Twisted.Page):
+ """A template for pages using our CSS- all pages have a heading with
+ title, subtitle, and site name. The content of each page is in
+ columns holding sections, while each page shares certain navigation
+ features- section tabs at the top, and a 'breadcrumbs' display
+ linking to and showing a page's parent pages.
+ """
+ siteName = "CIA.vc"
+ mainTitle = None
+ subTitle = []
+ leftColumn = []
+ mainColumn = []
+ extraHeaders = []
+
+ # Placeholders for site-specific customization
+ site_belowLeftColumn = []
+ site_bottomOfFooter = []
+
+ titleElements = [
+ tag('div', _class="mainTitle")[ place("mainTitle") ],
+ tag('div', _class="subTitle")[ place("subTitle") ],
+ ]
+
+ logoElements = [
+ tag('a', _class="sitename", href="/")[
+ tag('img', src='/media/img/nameplate-24.png', width=87, height=24,
+ alt='CIA.vc'),
+ ],
+ ]
+
+ document = [
+ xml('<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" '
+ '"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">\n'),
+
+ tag('html', xmlns="http://www.w3.org/1999/xhtml")[
+ tag('head')[
+ tag('title')[ place("pageTitle") ],
+ place('baseTag'),
+ xml('<meta http-equiv="Content-Type" content="text/html;charset=UTF-8" />'),
+ tag('link', rel='stylesheet', href='/media/css/old-site.css', type='text/css'),
+ tag('link', rel='shortcut icon', href='/favicon.ico', type='image/png'),
+ tag('script', type='text/javascript', src='/media/js/base.js')[ " " ],
+ place('extraHeaders'),
+ ],
+ tag('body')[
+
+ tag('div', _class="heading")[
+ tag('div', _class="topRight")[
+ tag('input', type='text', id='search'),
+ place("tabs"),
+ place('logoElements'),
+ ],
+ tag('div', _class="topLeft")[
+ place('titleElements'),
+ ],
+ tag('div', _class="tabBar")[ place("breadcrumbs") ],
+ ],
+
+ # The page body. We really shouldn't still be using a table for this...
+ tag('table', _class="columns")[ tag('tr')[
+ tag('td', _class="left")[
+ place("templateLeftColumn"),
+ ],
+ tag('td', _class="main")[
+ place("mainColumn")
+ ],
+ ]],
+
+ tag('div', _class="footer")[
+
+ # Legal goop
+ tag('p', _class='smallprint')[
+ xml("The CIA.vc server is Copyright © 2003-2007 "),
+ EmailLink('mailto:micah at navi.cx')["Micah Dowty"],
+ ", and released under the ",
+ tag('a', _href='/doc/COPYING')["GNU GPL"], ".", tag('br'),
+ "All hosted messages and metadata are owned by their respective authors.",
+ ],
+
+ # More optional text
+ place("site_bottomOfFooter"),
+
+ ],
+
+ xml('<script type="text/javascript">'
+ 'CIASearch.init("/api/search/", "search", "Search CIA.vc");'
+ '</script>'),
+ ],
+ ],
+ ]
+
+ def render_pageTitle(self, context):
+ # Wait for the title and site name to resolve into strings so we can mess with them a bit more
+ result = defer.Deferred()
+ defer.gatherResults([
+ defer.maybeDeferred(self.render_mainTitle, context),
+ defer.maybeDeferred(self.render_siteName, context),
+ ]).addCallback(
+ self._render_pageTitle, context, result
+ ).addErrback(result.errback)
+ return result
+
+ def _render_pageTitle(self, titleAndSite, context, result):
+ # Now that the title and site name have fully resolved, we can apply some heuristics...
+ title, siteName = titleAndSite
+
+ if title is None:
+ result.callback(siteName)
+ return
+
+ if type(title) in types.StringTypes and type(siteName) in types.StringTypes:
+ # The title and site are plain strings. If it starts with or ends with the site name,
+ # just use it as-is to avoid being overly redundant.
+ if title == siteName or title.startswith(siteName + " ") or title.endswith(" " + siteName):
+ result.callback(title)
+ return
+
+ # Otherwise, stick the title and site name together
+ result.callback([title, ' - ', siteName])
+
+ def render_templateLeftColumn(self, context):
+ """A sneaky little rendering function that runs render_leftColumn,
+ but then sticks in the site_* modifiers where possible. Note that
+ this won't work if render_leftColumn returns a Deferred, but
+ nothing in the CIA server does this yet.
+ """
+ return self.render_leftColumn(context) + self.site_belowLeftColumn
+
+ def render_siteName(self, context):
+ return self.siteName
+
+ def render_mainTitle(self, context):
+ return self.mainTitle
+
+ def render_subTitle(self, context):
+ return self.subTitle
+
+ def render_leftColumn(self, context):
+ return self.leftColumn
+
+ def render_mainColumn(self, context):
+ return self.mainColumn
+
+ def render_breadcrumbs(self, context):
+ places = [self.render_mainTitle(context)]
+ node = self.parent()
+ # If we don't at least have a parent node, breadcrumbs
+ # are going to be pretty useless. Just stick in a
+ # non-breaking space as a placeholder.
+ if not node:
+ return xml(" ")
+ while node:
+ places.insert(0, breadcrumbSeparator)
+ places.insert(0, node.render_link(context))
+ node = node.parent()
+ return places
+
+ def render_baseTag(self, context):
+ """Return an HTML <base> tag pointing at this page's original URL.
+ This keeps the page from breaking if it's saved to disk or copied elsewhere.
+ """
+ return tag('base', href = context['request'].prePathURL())
+
+ def render_link(self, context):
+ """Return a serializable object that should be used to link to this page.
+ By default, this returns a plain link with the page's title, pointing
+ to the page's URL.
+ """
+ return tag('a', href=self.getURL(context))[self.render_mainTitle(context)]
+
+ def parent(self):
+ """Pages must implement this to return their parent page.
+ This is used for the default implementation of breadcrumbs.
+ """
+ pass
+
+ def render_tabs(self, context):
+ """The page's tabs show all named components"""
+ tabs = []
+ for component in context['request'].site.components:
+ if component.name:
+ tabs.append(tag('li')[
+ xml('» '),
+ tag('a', href=component.url)[ component.name ],
+ ])
+
+ return tag('ul', _class='heading')[ tabs ]
+
+ def getURL(self, context):
+ """Retrieve a URL suitable for linking to this page."""
+ pass
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/Template.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/Template.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/Web/__init__.py
===================================================================
--- trunk/community/infrastructure/LibCIA/Web/__init__.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/Web/__init__.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,29 @@
+""" LibCIA.Web
+
+twisted.web resources and templates implementing CIA's web interface
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+# Convenience imports
+import Stats
+import Server
+import Overview
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/Web/__init__.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/Web/__init__.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/XML.py
===================================================================
--- trunk/community/infrastructure/LibCIA/XML.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/XML.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,397 @@
+""" LibCIA.XML
+
+Classes and utility functions to make the DOM suck less. CIA has been
+ported across DOM implementations multiple times, and may need to be
+ported again in the future. This file, in addition to making life easier,
+should hide quirks of particular DOM implementations as best as possible.
+
+This implementation uses Minidom.
+
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import re
+from xml.dom import minidom
+from xml.sax import SAXParseException as ParseException
+
+
+class XMLObject(object):
+ """An object based on an XML document tree. This class provides
+ methods to load it from a string or a DOM tree, and convert
+ it back to an XML string.
+
+ 'doc' is either a DOM node, a string containing
+ the message in XML, or a stream-like object.
+ """
+ # Subclasses can set this to enable some caches
+ immutable = False
+ _xpcache = None
+
+ def __init__(self, doc=None, uri=None):
+ if type(doc) in (str, unicode):
+ self.loadFromString(doc, uri)
+ elif hasattr(doc, 'read'):
+ self.loadFromStream(doc, uri)
+ elif hasattr(doc, 'nodeType'):
+ self.loadFromDom(doc)
+
+ def __str__(self):
+ return toString(self.xml)
+
+ def loadFromString(self, string, uri=None):
+ """Parse the given string as XML and set the contents of the message"""
+ self.loadFromDom(parseString(string))
+
+ def loadFromStream(self, stream, uri=None):
+ """Parse the given stream as XML and set the contents of the message"""
+ self.loadFromDom(parseStream(stream))
+
+ def loadFromDom(self, root):
+ """Set the contents of the Message from a parsed DOM tree"""
+ if hasattr(root, "documentElement"):
+ self.xml = root
+ else:
+ # Encase the given tree fragment in a Document
+ self.xml = createRootNode()
+ self.xml.appendChild(self.xml.importNode(root, True))
+ self.preprocess()
+
+ def preprocess(self):
+ """A hook where subclasses can add code to inspect a freshly
+ loaded XML document and/or fill in any missing information.
+ """
+ pass
+
+
+class XMLObjectParser(XMLObject):
+ """An XMLObject that is parsed recursively on creation into any
+ python object, stored in 'resultAttribute'. parse() dispatches
+ control to an element_* method when it finds an element, and
+ to parseString when it comes to character data.
+ """
+ requiredRootElement = None
+ resultAttribute = 'result'
+
+ def preprocess(self):
+ """Upon creating this object, parse the XML tree recursively.
+ The result returned from parsing the tree's root element
+ is set to our resultAttribute.
+ """
+ # Validate the root element type if the subclass wants us to.
+ # This is hard to do elsewhere, since the element handlers don't
+ # know where they are in the XML document.
+ if self.requiredRootElement is not None:
+ rootElement = None
+ if self.xml.nodeType == self.xml.DOCUMENT_NODE:
+ rootElement = self.xml.documentElement
+ elif self.xml.nodeType == self.xml.ELEMENT_NODE:
+ rootElement = self.xml
+
+ if (not rootElement) or rootElement.nodeName != self.requiredRootElement:
+ raise UnknownElementError("Missing a required %r root element" %
+ self.requiredRootElement)
+
+ setattr(self, self.resultAttribute, self.parse(self.xml))
+
+ def parse(self, node, *args, **kwargs):
+ """Given a DOM node, finds an appropriate parse function and invokes it"""
+ if node.nodeType == node.TEXT_NODE:
+ return self.parseString(node.data, *args, **kwargs)
+
+ elif node.nodeType == node.ELEMENT_NODE:
+ f = getattr(self, "element_" + node.nodeName, None)
+ if f:
+ return f(node, *args, **kwargs)
+ else:
+ return self.unknownElement(node, *args, **kwargs)
+
+ elif node.nodeType == node.DOCUMENT_NODE:
+ return self.parse(node.documentElement, *args, **kwargs)
+
+ def childParser(self, node, *args, **kwargs):
+ """A generator that parses all relevant child nodes, yielding their return values"""
+ parseableTypes = (node.TEXT_NODE, node.ELEMENT_NODE)
+ for child in node.childNodes:
+ if child.nodeType in parseableTypes:
+ yield self.parse(child, *args, **kwargs)
+
+ def parseString(self, s):
+ """The analogue to element_* for character data"""
+ pass
+
+ def unknownElement(self, element):
+ """An unknown element was found, by default just generates an exception"""
+ raise UnknownElementError("Invalid element in %s: '%s'" % (self.__class__.__name__, element.nodeName))
+
+
+class XMLFunction(XMLObjectParser):
+ """An XMLObject that is parsed on creation into a function,
+ making this class callable. The parser relies on member functions
+ starting with 'element_' to recursively parse each element of the XML
+ tree, returning a function implementing it.
+ """
+ resultAttribute = 'f'
+
+ def __call__(self, *args, **kwargs):
+ return self.f(*args, **kwargs)
+
+
+class XMLValidityError(Exception):
+ """This error is raised by subclasses of XMLObject that encounter problems
+ in the structure of XML documents presented to them. Normally this should
+ correspond with the document not being valid according to its schema,
+ but we don't actually use a validating parser.
+ """
+ pass
+
+class UnknownElementError(XMLValidityError):
+ pass
+
+
+def allTextGenerator(node):
+ """A generator that, given a DOM tree, yields all text fragments in that tree"""
+ if node.nodeType == node.TEXT_NODE:
+ yield node.data
+ for child in node.childNodes:
+ for text in allTextGenerator(child):
+ yield text
+
+
+def allText(node):
+ """Concatenate all text under the given element recursively, and return it"""
+ return "".join(allTextGenerator(node))
+
+
+def shallowTextGenerator(node):
+ """A generator that, given a DOM tree, yields all text fragments contained immediately within"""
+ for child in node.childNodes:
+ if child.nodeType == child.TEXT_NODE:
+ yield child.data
+
+
+def shallowText(node):
+ """Concatenate all text immediately within the given node"""
+ return "".join(shallowTextGenerator(node))
+
+
+def dig(node, *subElements):
+ """Search for the given named subelements inside a node. Returns
+ None if any subelement isn't found.
+ """
+ if not node:
+ return None
+ for name in subElements:
+ nextNode = None
+ for child in node.childNodes:
+ if child.nodeType == child.ELEMENT_NODE and child.nodeName == name:
+ nextNode = child
+ break
+ if nextNode:
+ node = nextNode
+ else:
+ return None
+ return node
+
+
+def digValue(node, _type, *subElements):
+ """Search for a subelement using 'dig', then retrieve all contained
+ text and convert it to the given type. Strips extra whitespace.
+ """
+ foundNode = dig(node, *subElements)
+ if foundNode:
+ return _type(shallowText(foundNode).strip())
+
+
+def bury(node, *subElements):
+ """Search for the given named subelements inside a node,
+ creating any that can't be found.
+ """
+ for name in subElements:
+ nextNode = None
+ for child in node.childNodes:
+ if child.nodeType == child.ELEMENT_NODE and child.nodeName == name:
+ nextNode = child
+ break
+ if nextNode:
+ node = nextNode
+ else:
+ node = addElement(node, name)
+ return node
+
+
+def buryValue(node, value, *subElements):
+ """Search for a subelement using 'bury' then store the given value,
+ overwriting the previous content of that node.
+ """
+ node = bury(node, *subElements)
+
+ for child in list(node.childNodes):
+ if child.nodeType == child.TEXT_NODE:
+ node.removeChild(child)
+
+ node.appendChild(node.ownerDocument.createTextNode(str(value)))
+
+
+def addElement(node, name, content=None, attributes={}):
+ """Add a new child element to the given node, optionally containing
+ the given text and attributes. The attributes are specified as a
+ simple mapping from name string to value string. This function
+ does not support namespaces.
+ """
+ if node.nodeType == node.DOCUMENT_NODE:
+ doc = node
+ else:
+ doc = node.ownerDocument
+
+ newElement = doc.createElementNS(None, name)
+ if content:
+ newElement.appendChild(doc.createTextNode(content))
+ for attrName, attrValue in attributes.iteritems():
+ newElement.setAttributeNS(None, attrName, attrValue)
+ node.appendChild(newElement)
+ return newElement
+
+
+parseStream = minidom.parse
+
+def parseString(string):
+ # Grarr.. minidom can't directly parse Unicode objects
+ if type(string) is unicode:
+ string = string.encode('utf-8')
+
+ return minidom.parseString(string)
+
+
+def createRootNode():
+ return minidom.getDOMImplementation().createDocument(None, None, None)
+
+def toString(doc):
+ """Convert a DOM tree back to a string"""
+ return doc.toxml()
+
+
+def getChildElements(doc):
+ """A generator that returns all child elements of a node"""
+ for child in doc.childNodes:
+ if child.nodeType == child.ELEMENT_NODE:
+ yield child
+
+
+def firstChildElement(doc):
+ try:
+ return getChildElements(doc).next()
+ except StopIteration:
+ return None
+
+
+def hasChildElements(doc):
+ # Force a boolean result
+ return firstChildElement(doc) is not None
+
+
+class XPathBase:
+ """Abstract base XPath implementation"""
+
+ def queryObject(self, obj):
+ return self.queryForNodes(obj.xml)
+
+
+class XPathFull(XPathBase):
+ """Full XPath implementation, using 4XPath. Caches all xpaths by
+ default, in a global dictionary. XPaths are never evicted from
+ the cache.
+ """
+ cache = {}
+ enableCache = True
+
+ def __init__(self, path):
+ if self.enableCache:
+ try:
+ self.compiled = self.cache[path]
+ except KeyError:
+ self.compiled = self._compile(path)
+ self.cache[path] = self.compiled
+ else:
+ self.compiled = self._compile(path)
+
+ def _compile(self, path):
+ # xpath.Compile() seems to have broken error handling-
+ # it's reporting syntax errors as RuntimeException.INTERNAL.
+ # Work around this by instantiating the parser directly.
+
+ import xml.xpath
+
+ try:
+ return xml.xpath.parser.new().parse(path)
+ except xml.xpath.yappsrt.SyntaxError, e:
+ raise XMLValidityError('XPath syntax error in "%s" at char %d: %s' % (
+ path, e.pos, e.msg))
+
+ def queryForNodes(self, doc):
+ """Query an XML DOM, returning the result set"""
+ return self.compiled.evaluate(xml.xpath.CreateContext(doc))
+
+
+class XPathTiny(XPathBase):
+ """Implements a very tiny XPath subset, which only supports absolute
+ paths using only the default (child::) axis.
+ """
+ _validation_re = re.compile(r"^(/[a-zA-Z_][a-zA-Z0-9_\.-]*)+$")
+
+ def __init__(self, path):
+ if not self._validation_re.match(path):
+ raise XMLValidityError('"%s" is not valid in the supported subset of XPath' % path)
+ self.compiled = path.split("/")[1:]
+
+ def queryForNodes(self, doc):
+ ELEMENT = doc.ELEMENT_NODE
+ current = [doc]
+ for name in self.compiled:
+ next = []
+ for el in current:
+ for child in el.childNodes:
+ if child.nodeType == ELEMENT and child.nodeName == name:
+ next.append(child)
+ current = next
+ return current
+
+
+# Our big switch to choose the default XPath implementation
+XPath = XPathTiny
+
+
+# The general form of "path" is an XPath, but we actually only support
+# a very tiny subset of XPath. To make filters a little less overly verbose
+# without making our XPaths (and eventually our Esquilax index) less
+# efficient, we'll define some shortcuts for common paths.
+
+pathShortcuts = {
+ 'branch': '/message/source/branch',
+ 'project': '/message/source/project',
+ 'module': '/message/source/module',
+ 'revision': '/message/body/commit/revision',
+ 'version': '/message/body/commit/version',
+ 'author': '/message/body/commit/author',
+ 'log': '/message/body/commit/log',
+ 'files': '/message/body/commit/files',
+ 'url': '/message/body/commit/url',
+ }
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/XML.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/XML.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/LibCIA/__init__.py
===================================================================
--- trunk/community/infrastructure/LibCIA/__init__.py (rev 0)
+++ trunk/community/infrastructure/LibCIA/__init__.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,48 @@
+""" LibCIA
+
+This is a Python package providing modules used to implement the CIA
+open source notification system.
+
+CIA provides a way for projects to send messages from their version
+control and bug tracking systems to anyone interested- mainly a
+network of IRC bots and a web site.
+
+Where does the name CIA come from? It was originally designed to
+monitor commits from PicoGUI's Subversion repository, and Lalo came
+up with the name CIA: it was a brainless entity designed to keep
+an eye on Subversion ;)
+"""
+#
+# CIA open source notification system
+# Copyright (C) 2003-2007 Micah Dowty <micah at navi.cx>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+__version__ = "0.91svn"
+
+
+# Check the python version here before we proceed further
+requiredPythonVersion = (2,2,1)
+import sys, string
+if sys.version_info < requiredPythonVersion:
+ raise Exception("%s requires at least Python %s, found %s instead." % (
+ name,
+ string.join(map(str, requiredPythonVersion), "."),
+ string.join(map(str, sys.version_info), ".")))
+del sys
+del string
+
+### The End ###
Added: trunk/community/infrastructure/LibCIA/__init__.pyc
===================================================================
(Binary files differ)
Property changes on: trunk/community/infrastructure/LibCIA/__init__.pyc
___________________________________________________________________
Name: svn:mime-type
+ application/octet-stream
Added: trunk/community/infrastructure/Nouvelle/BaseHTTP.py
===================================================================
--- trunk/community/infrastructure/Nouvelle/BaseHTTP.py (rev 0)
+++ trunk/community/infrastructure/Nouvelle/BaseHTTP.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,127 @@
+""" Nouvelle.BaseHTTP
+
+Glue for using Nouvelle with Python's builtin BaseHTTPServer module.
+This provides a Page class that lets objects be attached as children
+to it, a RequestHandler that dispatches HTTP requests to a root Page,
+and a simple main function that makes it quick and easy to start a
+server with a particular Page at its root.
+"""
+#
+# Nouvelle web framework
+# Copyright (C) 2003-2005 Micah Dowty <micahjd at users.sourceforge.net>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import Nouvelle
+from Nouvelle import tag
+import BaseHTTPServer, urlparse
+
+
+class Page:
+ """A web resource that renders a tree of tag instances from its 'document'
+ attribute, and can have child resources attached to it.
+ """
+ serializerFactory = Nouvelle.Serializer
+ responseCode = 200
+
+ def handleRequest(self, request, args):
+ """Given a RequestHandler instance, send back an HTTP response code,
+ headers, and a rendition of this page.
+ """
+ request.send_response(self.responseCode)
+ self.sendHeaders(request)
+
+ context = {
+ 'owner': self,
+ 'request': request,
+ 'args': args,
+ }
+ self.preRender(context)
+
+ rendered = str(self.serializerFactory().render(self.document, context))
+ request.wfile.write(rendered)
+
+ def sendHeaders(self, request):
+ """Send back HTTP headers for a given request"""
+ request.send_header('Content-Type', 'text/html')
+ request.end_headers()
+
+ def preRender(self, context):
+ """Called prior to rendering each request, subclasses can use this to annotate
+ 'context' with extra information or perform other important setup tasks.
+ """
+ pass
+
+ def addChild(self, name, page):
+ """Add the given Page instance as a child under this one in the URL tree"""
+ if not hasattr(self, 'children'):
+ self.children = {}
+ self.children[name] = page
+
+ def findChild(self, name):
+ """Return the named child of this Page. By default this looks in
+ self.children, and if a page isn't found returns Error404.
+ """
+ if not name:
+ # Ignore empty path segments
+ return self
+ if hasattr(self, 'children') and self.children.has_key(name):
+ return self.children[name]
+ return Error404()
+
+
+class Error404(Page):
+ """A 404 error, resource not found"""
+ responseCode = 404
+ document = tag('html')[
+ tag('head')[
+ tag('title')[ "404 - Resource not found" ],
+ ],
+ tag('body')[
+ tag('h1')[ "404" ],
+ tag('h3')[ "Resource not found" ],
+ ],
+ ]
+
+
+class RequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
+ def do_GET(self):
+ # Parse the path we were given as a URL...
+ scheme, host, path, parameters, query, fragment = urlparse.urlparse(self.path)
+
+ # Find the page corresponding with our URL's path
+ page = self.rootPage
+ for segment in path.split("/"):
+ page = page.findChild(segment)
+
+ # Split the query into key-value pairs
+ args = {}
+ for pair in query.split("&"):
+ if pair.find("=") >= 0:
+ key, value = pair.split("=", 1)
+ args.setdefault(key, []).append(value)
+ else:
+ args[pair] = []
+
+ page.handleRequest(self, args)
+
+
+def main(rootPage, port=8080):
+ handler = RequestHandler
+ handler.rootPage = rootPage
+ BaseHTTPServer.HTTPServer(('', port), handler).serve_forever()
+
+### The End ###
Added: trunk/community/infrastructure/Nouvelle/ModPython.py
===================================================================
--- trunk/community/infrastructure/Nouvelle/ModPython.py (rev 0)
+++ trunk/community/infrastructure/Nouvelle/ModPython.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,124 @@
+""" Nouvelle.ModPython
+
+Glue for using Nouvelle with mod_python. Page objects here, once instantiated,
+can be rendered by the publisher handler using their 'publish' method or 'handler'
+can be used to implement a standalone request handler.
+"""
+#
+# Nouvelle web framework
+# Copyright (C) 2003-2005 Micah Dowty <micahjd at users.sourceforge.net>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import Nouvelle
+from Nouvelle import tag
+from mod_python import apache
+
+class Page:
+ """A web resource that renders a tree of tag instances from its 'document'
+ attribute when called.
+ """
+ serializerFactory = Nouvelle.Serializer
+ contentType = "text/html"
+ isLeaf = True
+
+ def publish(self, **kwargs):
+ """Render this page via the publisher handler"""
+ request = kwargs['req']
+ del kwargs['req']
+ context = dict(owner=self, request=request, args=kwargs)
+ self.preRender(context)
+ return self.render(context)
+
+ def handler(self, req):
+ """Implement a standalone request handler. This can handle child requests
+ through the getChild() method.
+ """
+ # Recurse into child pages as necessary
+ current = self
+ for seg in req.path_info.split('/'):
+ if not seg:
+ continue
+ try:
+ current = current.getChild(seg)
+ except KeyError:
+ return apache.HTTP_NOT_FOUND
+ if not current:
+ return apache.HTTP_NOT_FOUND
+
+ # Split the query into key-value pairs
+ args = {}
+ if req.args:
+ for pair in req.args.split("&"):
+ if pair.find("=") >= 0:
+ key, value = pair.split("=", 1)
+ args.setdefault(key, []).append(value)
+ else:
+ args[pair] = []
+
+ context = dict(owner=current, request=req, args=args)
+ current.preRender(context)
+ return current.render(context)
+
+ def render(self, context):
+ """Render the current page, writing the results via the context's request object"""
+ req = context['request']
+
+ # If this isn't a leaf resource and doesn't end with a slash, redirect there
+ if not self.isLeaf and not req.uri.endswith("/"):
+ self.redirect(req, req.uri + "/")
+
+ page = str(self.serializerFactory().render(self.document, context))
+ req.content_type = self.contentType
+ req.write(page)
+ return apache.OK
+
+ def redirect(self, req, url, temporary=False, seeOther=False):
+ """Immediately redirects the request to the given url. If the
+ seeOther parameter is set, 303 See Other response is sent, if the
+ temporary parameter is set, the server issues a 307 Temporary
+ Redirect. Otherwise a 301 Moved Permanently response is issued.
+
+ This function is from Stian Soiland's post to mod_python's mailing list at:
+ http://www.modpython.org/pipermail/mod_python/2004-January/014865.html
+ """
+ from mod_python import apache
+
+ if seeOther:
+ status = apache.HTTP_SEE_OTHER
+ elif temporary:
+ status = apache.HTTP_TEMPORARY_REDIRECT
+ else:
+ status = apache.HTTP_MOVED_PERMANENTLY
+
+ req.headers_out['Location'] = url
+ req.status = status
+ raise apache.SERVER_RETURN, status
+
+ def preRender(self, context):
+ """Called prior to rendering each request, subclasses can use this to annotate
+ 'context' with extra information or perform other important setup tasks.
+ """
+ pass
+
+ def getChild(self, name):
+ """Return a child resource. The default implementation looks in a 'children' database,
+ but this can be overridden by subclasses to implement more dynamic behavior.
+ """
+ d = getattr(self, 'children', {})
+ return d.get(name)
+
+### The End ###
Added: trunk/community/infrastructure/Nouvelle/Serial.py
===================================================================
--- trunk/community/infrastructure/Nouvelle/Serial.py (rev 0)
+++ trunk/community/infrastructure/Nouvelle/Serial.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,246 @@
+""" Nouvelle.Serial
+
+Core functionality for Nouvelle's object serialization system, including
+the tag, place, and Serializer objects.
+
+The lowercase 'place', 'xml', and 'tag' classes break my naming
+convention, but since they aren't really used like conventional classes
+I think lowercase makes more sense.
+"""
+#
+# Nouvelle web framework
+# Copyright (C) 2003-2005 Micah Dowty <micahjd at users.sourceforge.net>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+__all__ = ['place', 'xml', 'subcontext', 'tag', 'quote', 'Serializer', 'DocumentOwner']
+
+# Use psyco if we can, as this module is the bottleneck in most web apps
+# and the code is small enough that the memory penalty is minimal
+try:
+ from psyco.classes import __metaclass__
+except ImportError:
+ pass
+
+
+class place:
+ """A placeholder for data that can be rendered by a document's owner.
+ For example, place('title') calls render_title() in the object owning
+ the current document, context['owner']. If there is not render_title
+ function, this will look for a 'title' attribute to return verbatim.
+ """
+ def __init__(self, name, *args, **kwargs):
+ self.name = name
+ self.args = args
+ self.kwargs = kwargs
+
+ def render(self, context):
+ try:
+ f = getattr(context['owner'], 'render_' + self.name)
+ except AttributeError:
+ return getattr(context['owner'], self.name)
+ return f(context, *self.args, **self.kwargs)
+
+
+class xml(str):
+ """A marker indicating data that is already represented in raw XML and
+ needs no further processing.
+ """
+ __slots__ = []
+
+
+class quote:
+ """A wrapper for any serializable object that fully serializes it then
+ leaves the result as a string rather than an xml() object, so it is quoted.
+ This is useful for rendering HTML as quoted text inside of other XML
+ documents, for example. If this is used on normal text, note that the
+ text will be quoted twice.
+ """
+ def __init__(self, item):
+ self.item = item
+
+
+class tag:
+ """A renderable XHTML tag, containing other renderable objects.
+ If an object enclosed doesn't have a 'render' method, it is casted
+ to a string and quoted.
+
+ This class was inspired by nevow, but much simpler. In CIA, dynamically
+ rebuilding the code to implement changes at runtime is more important
+ than speed, and this avoids much of the complexity (and power) of
+ nevow's ISerializable.
+
+ Attributes are quoted and converted to tag attributes. Leading underscores
+ are removed, so they can be used to define attributes that are reserved
+ words in Python.
+ """
+ def __init__(self, name, **attributes):
+ self.name = name
+ self.content = []
+ self._setAttributes(attributes)
+
+ def __call__(self, name=None, **attributes):
+ """A tag instance can be called just like the tag
+ class, so any tag instance can be used in place
+ of 'tag' to provide default values.
+ """
+ if name is None:
+ name = self.name
+ attrs = dict(self.attributes)
+ attrs.update(attributes)
+ return tag(name, **attrs)[ self.content ]
+
+ def _stringizeAttributes(self, attributes):
+ """Return a string representation of the given attribute dictionary,
+ with a leading space if there are any attributes. Returns the empty
+ string if no attributes were given.
+ """
+ s = ''
+ for key, value in attributes.iteritems():
+ if key[0] == '_':
+ key = key[1:]
+ if value is not None:
+ s += ' %s="%s"' % (key, escapeToXml(str(value), True))
+ return s
+
+ def _setAttributes(self, attributes):
+ """Change this tag's attributes, rerendering the opening and closing text"""
+ self.attributes = attributes
+ attrString = self._stringizeAttributes(attributes)
+ self.renderedOpening = xml('<%s%s>' % (self.name, attrString))
+ self.renderedEmpty = xml('<%s%s />' % (self.name, attrString))
+ self.renderedClosing = xml('</%s>' % self.name)
+
+ def __getitem__(self, content):
+ """Overloads the [] operator used, in Tag, to return a new tag
+ with the given content
+ """
+ newTag = tag(self.name, **self.attributes)
+ newTag.content = content
+ return newTag
+
+ def render(self, context=None):
+ if self.content in ('', [], ()):
+ return self.renderedEmpty
+ else:
+ return [self.renderedOpening, self.content, self.renderedClosing]
+
+
+class subcontext:
+ """A wrapper for any serializable object that makes note of a modified
+ context to serialize all content in. This can be used like a tag,
+ with content in [] and context changes in ().
+
+ It is not enforced, but normally subcontexts should be immutable.
+ """
+ def __init__(self, content=None, **modifications):
+ self.content = content
+ self.modifications = modifications
+
+ def __call__(self, content=None, **modifications):
+ """Like tags, instances can be used as templates for creating other
+ subcontexts. This returns a new distinct subcontext based on
+ the current one.
+ """
+ if content is None:
+ content = self.content
+ newMods = dict(self.modifications)
+ newMods.update(modifications)
+ return subcontext(content, **newMods)
+
+ def __getitem__(self, content):
+ """Returns a new subcontext, like this one but with the indicated content"""
+ return subcontext(content, **self.modifications)
+
+
+def escapeToXml(text, isAttrib=0):
+ text = text.replace("&", "&")
+ text = text.replace("<", "<")
+ text = text.replace(">", ">")
+ if isAttrib == 1:
+ text = text.replace("'", "'")
+ text = text.replace("\"", """)
+ return text
+
+
+class Serializer:
+ """Convert arbitrary objects to xml markers recursively. Renderable objects
+ are allowed to render themselves, other types must have renderers looked
+ up in this class's render_* methods.
+ """
+ def __init__(self):
+ self.cache = {}
+
+ def render(self, obj, context={}):
+ # We cache rendering functions based on obj.__class__.
+ # This assumes that classes won't spontaneously
+ # grow a render() method, which is safe enough for our purposes.
+ # If you know otherwise, you can just create a new Serializer or
+ # invalidate the cache on this one.
+ try:
+ f = self.cache[obj.__class__]
+ except KeyError:
+ f = self.lookupRenderer(obj.__class__)
+ self.cache[obj.__class__] = f
+ return f(obj, context)
+
+ def lookupRenderer(self, cls):
+ """Look up a rendering function for the given class"""
+ if hasattr(cls, 'render'):
+ return self.render_renderable
+ try:
+ return getattr(self, 'render_' + cls.__name__)
+ except AttributeError:
+ return self.render_other
+
+ def render_xml(self, obj, context):
+ return obj
+
+ def render_list(self, obj, context):
+ return xml(''.join([self.render(o, context) for o in obj]))
+
+ def render_tuple(self, obj, context):
+ return self.render_list(obj, context)
+
+ def render_quote(self, obj, context):
+ return xml(escapeToXml(str(self.render(obj.item, context))))
+
+ def render_function(self, obj, context):
+ return self.render(obj(context), context)
+
+ def render_instancemethod(self, obj, context):
+ return self.render(obj(context), context)
+
+ def render_renderable(self, obj, context):
+ return self.render(obj.render(context), context)
+
+ def render_subcontext(self, obj, context):
+ newContext = dict(context)
+ newContext.update(obj.modifications)
+ return self.render(obj.content, newContext)
+
+ def render_other(self, obj, context):
+ return xml(escapeToXml(str(obj)))
+
+
+class DocumentOwner(object):
+ """A base class defining a 'render' function for objects that own a document"""
+ serializerFactory = Serializer
+
+ def render(self, context={}):
+ return subcontext(self.document, owner=self)
+
+### The End ###
Added: trunk/community/infrastructure/Nouvelle/Table.py
===================================================================
--- trunk/community/infrastructure/Nouvelle/Table.py (rev 0)
+++ trunk/community/infrastructure/Nouvelle/Table.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,270 @@
+""" Nouvelle.Table
+
+A fancy table renderable for Nouvelle. The dataset is represented by a
+sequence containing arbitrary objects representing rows. Columns
+are classes that know how to retrieve a particular piece of data
+from the table and optionally format it.
+
+ResortableTable is a Table in which each column heading is a link that
+toggles sorting by that column and the direction of the sort.
+"""
+#
+# Nouvelle web framework
+# Copyright (C) 2003-2005 Micah Dowty <micahjd at users.sourceforge.net>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+from __future__ import generators
+from Serial import tag, xml
+import re
+
+__all__ = ['BaseTable', 'Column', 'AttributeColumn', 'IndexedColumn',
+ 'ResortableTable']
+
+
+class Column:
+ """A column provides a way to view and sort by some aspect of
+ the data stored in a row. This is an abstract base class defining
+ a Column's interface.
+ """
+ def render_heading(self, context):
+ """Returns a serializable object representing this column in a table's heading"""
+ return self.heading
+
+ def render_data(self, context, row):
+ """Returns a serializable object representing this column's data for a particular row.
+ Normally this will be a visible representation of getValue's results.
+ By default it's just getValue's result without any extra formatting.
+ """
+ return self.getValue(row)
+
+ def getValue(self, row):
+ """Return a single value representing this column's perspective of the row.
+ This is used by reduceColumn, and for sorting.
+ """
+ pass
+
+ def cmp(self, a, b):
+ """This column's comparison function, used for sorting a table by this column.
+ By default this does an ascending sort using getValue.
+ """
+ return cmp(self.getValue(a), self.getValue(b))
+
+ def isVisible(self, context):
+ """Subclasses can override this to hide the entire column in some circumstances"""
+ return True
+
+
+class AttributeColumn(Column):
+ """A Column that has a fixed heading and returns some attribute from each row,
+ with no special formatting.
+ """
+ def __init__(self, heading, attribute):
+ self.heading = heading
+ self.attribute = attribute
+
+ def getValue(self, row):
+ return getattr(row, self.attribute)
+
+
+class IndexedColumn(Column):
+ """A Column that has a fixed index it uses to retrieve an item from the row
+ and return it with no special formatting.
+ """
+ def __init__(self, heading, index):
+ self.heading = heading
+ self.index = index
+
+ def getValue(self, row):
+ return row[self.index]
+
+
+class BaseTable:
+ """A renderable that creates an XHTML table viewing some dataset using
+ a list of Column instances. The Columns are used to generate headings
+ and data cells for each column of the table.
+
+ Subclasses can override the tag factories here to change the generated
+ XHTML, possibly by adding CSS 'class' attributes to the tags.
+ """
+ tableTag = tag('table')
+ rowTag = tag('tr')
+ headingTag = tag('th')
+ dataTag = tag('td')
+
+ def __init__(self, rows, columns, showHeading=True):
+ self.rows = rows
+ self.columns = columns
+ self.showHeading = showHeading
+ self._reduceColumnCache = {}
+
+ def render(self, context={}):
+ """Render the entire table, returning an unserialized tag tree"""
+ context = dict(context)
+ context['table'] = self
+
+ body = []
+ if self.showHeading:
+ body.append(self.rowTag[self.render_headings(context)])
+ for row in self.rows:
+ body.append(self.rowTag[self.render_row(context, row)])
+ return self.tableTag[body]
+
+ def render_headings(self, context):
+ """Return a list of headingTag instances labelling each column"""
+ return [self.headingTag[self.render_heading(context, c)] for c in self.getVisibleColumns(context)]
+
+ def render_heading(self, context, column):
+ """Return a serializable representation of one column heading"""
+ return column.render_heading(context)
+
+ def getVisibleColumns(self, context):
+ """Return an iterator over all visible columns"""
+ for column in self.columns:
+ if column.isVisible(context):
+ yield column
+
+ def render_row(self, context, row):
+ """Return a list of dataTag instances for each column in the given row"""
+ return [self.dataTag[c.render_data(context, row)] for c in self.getVisibleColumns(context)]
+
+ def getColumnValues(self, column):
+ """Return an iterator that iterates over all values in the specified column"""
+ for row in self.rows:
+ yield column.getValue(row)
+
+ def reduceColumn(self, column, operation):
+ """Calls the provided 'operation' with a generator that returns the getValue()
+ result for the given column on every row. The result is cached.
+ This is good for summing columns in the table, for example, by
+ passing the Python builtin 'sum' as the operation.
+ """
+ key = (column, operation)
+ if not self._reduceColumnCache.has_key(key):
+ self._reduceColumnCache[key] = operation(self.getColumnValues(column))
+ return self._reduceColumnCache[key]
+
+ def sortByColumn(self, column, reverse=False):
+ if reverse:
+ self.rows.sort(lambda a,b: -column.cmp(a,b))
+ else:
+ self.rows.sort(column.cmp)
+
+
+class ResortableTable(BaseTable):
+ """A Table with hyperlinks on each table column that set that column
+ as the sort column, or if it's already the sort column reverse the
+ order.
+
+ By default, this generates link URLs and retrieves the user's sort
+ setting by assuming context['args'] is a dictionary of arguments
+ passed in the query section of our URL, mapping key names to lists
+ of values.
+
+ Subclasses can override getCookieFromContext and getCookieHyperlink
+ to change this behaviour. The 'cookie' is an opaque piece of information
+ used to keep the table's state across page views.
+ """
+ headingLinkTag = tag('a')
+ sortIndicator = xml(" ↓")
+ reversedSortIndicator = xml(" ↑")
+ sortArgPrefix = "s_"
+
+ def __init__(self, rows, columns,
+ defaultSortColumnIndex = 0,
+ defaultSortReversed = False,
+ id = None,
+ ):
+ self.defaultSortColumnIndex = defaultSortColumnIndex
+ self.defaultSortReversed = defaultSortReversed
+ BaseTable.__init__(self, rows, columns)
+ self.sortArgName = self.sortArgPrefix + (id or '')
+
+ def render(self, context={}):
+ cookie = self.getCookieFromContext(context)
+ if cookie:
+ self.setSortFromCookie(cookie)
+ else:
+ self.sortColumnIndex = self.defaultSortColumnIndex
+ self.sortReversed = self.defaultSortReversed
+ self.sortByColumn(self.columns[self.sortColumnIndex], self.sortReversed)
+ return BaseTable.render(self, context)
+
+ def render_heading(self, context, column):
+ """Override render_heading to insert hyperlinks generated with createSortCookie,
+ and indicators on the current sort column.
+ """
+ url = self.getCookieHyperlink(context, self.getSortCookie(column))
+ heading = self.headingLinkTag(href=url)[column.render_heading(context)]
+ if self.columns[self.sortColumnIndex] is column:
+ heading = self.addSortIndicator(heading, self.sortReversed)
+ return heading
+
+ def addSortIndicator(self, heading, reversed=False):
+ if reversed:
+ indicator = self.reversedSortIndicator
+ else:
+ indicator = self.sortIndicator
+ return [heading, indicator]
+
+ def setSortFromCookie(self, cookie):
+ """Set our current sort using the given cookie.
+ Our cookies are just the column index (in the original
+ column list, not just visible columns) optionally
+ followed by 'R' for reversed sorts.
+ """
+ match = re.match("(?P<column>[0-9]+)(?P<reversed>R)?", str(cookie))
+ self.sortColumnIndex = int(match.group('column'))
+ self.sortReversed = bool(match.group('reversed'))
+
+ def getSortCookie(self, column):
+ """Return the cookie that should be used after the user clicks the given column.
+ If this is the current sort column already, we reverse the sorting direction.
+ If not, we set the column.
+ """
+ index = self.columns.index(column)
+ if index == self.sortColumnIndex and not self.sortReversed:
+ return "%dR" % index
+ else:
+ return str(index)
+
+ def getCookieFromContext(self, context):
+ try:
+ return context['args'][self.sortArgName][0]
+ except:
+ return None
+
+ def getCookieHyperlink(self, context, cookie):
+ """Build a new hyperlink including the new sort cookie and all arguments
+ that are safe to include in the new hyperlink.
+ """
+ newArgs = self.getCookieHyperlinkArgs(context)
+ newArgs[self.sortArgName] = [cookie]
+ pairs = []
+ for key in newArgs:
+ for value in newArgs[key]:
+ pairs.append('%s=%s' % (key, value))
+ return '?' + '&'.join(pairs)
+
+ def getCookieHyperlinkArgs(self, context):
+ """Return a new dictionary of arguments that are safe to include in our hyperlinks"""
+ d = {}
+ for key, values in context['args'].iteritems():
+ if key.startswith(self.sortArgPrefix):
+ d[key] = values
+ return d
+
+### The End ###
Added: trunk/community/infrastructure/Nouvelle/Twisted.py
===================================================================
--- trunk/community/infrastructure/Nouvelle/Twisted.py (rev 0)
+++ trunk/community/infrastructure/Nouvelle/Twisted.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,175 @@
+""" Nouvelle.Twisted
+
+Glue to help interface Nouvelle with twisted.web.
+This includes a twisted.web.resource that renders a Nouvelle document,
+and support for asynchronous rendering using Deferred.
+"""
+#
+# Nouvelle web framework
+# Copyright (C) 2003-2005 Micah Dowty <micahjd at users.sourceforge.net>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+import Nouvelle
+from Nouvelle import xml
+from twisted.internet import defer
+from twisted.web import server, error, resource, http
+
+
+class TwistedSerializer(Nouvelle.Serializer):
+ """A subclass of Nouvelle.Serializer that understands twisted's Deferred
+ objects and can render web pages asynchronously. If at any time a Deferred
+ object is encountered, a new Deferred object is created to represent the
+ serialized form of the original Deferred. If a Deferred is left over after
+ rendering the complete document, the document is rendered in the Deferred's
+ callback and our render function returns server.NOT_DONE_YET.
+ """
+ def render_Deferred(self, obj, context):
+ """A handler for rendering Deferred instances. This returns a new Deferred
+ instance representing the first deferred's serialized form.
+ """
+ result = defer.Deferred()
+ obj.addCallbacks(self.deferredRenderCallback,
+ result.errback,
+ callbackArgs = (context, result),
+ )
+ return result
+
+ def deferredRenderCallback(self, obj, context, result):
+ """A Deferred callback that renders data when it becomes available, adding the
+ result to our 'result' deferred.
+ """
+ r = self.render(obj, context)
+ # If the render result was a deferred, chain it to our result
+ if isinstance(r, defer.Deferred):
+ r.addCallback(result.callback).addErrback(result.errback)
+ else:
+ result.callback(r)
+ return obj
+
+ def render_list(self, obj, context):
+ """A new version of render_list that returns a Deferred
+ if any item in the list is a Deferred.
+ """
+ # Render each item in the list, noting whether we have any Deferreds
+ renderedItems = []
+ hasDeferreds = False
+ for item in obj:
+ rendered = self.render(item, context)
+ renderedItems.append(rendered)
+ if isinstance(rendered, defer.Deferred):
+ hasDeferreds = True
+
+ # If we had any deferred items, we need to create a DeferredList
+ # so we get notified once all of our content is rendered. Otherwise
+ # we can join them now.
+ if hasDeferreds:
+ result = defer.Deferred()
+
+ # Convert every result to a Deferred, wrapping non-deferred
+ # objects in deferreds that are already completed.
+ deferreds = []
+ for item in renderedItems:
+ if not isinstance(item, defer.Deferred):
+ d = defer.Deferred()
+ d.callback(item)
+ item = d
+ deferreds.append(item)
+
+ # Propagate errors to our parent. This is necessary
+ # even with an errback in our DeferredList, as the
+ # DeferredList doesn't absorb errors, it only detects them.
+ item.addErrback(self.listRenderErrback, result)
+
+ # Wait until all deferreds are completed
+ dl = defer.DeferredList(deferreds)
+ dl.addCallbacks(self.listRenderCallback,
+ self.listRenderErrback,
+ callbackArgs = (context, result),
+ errbackArgs = (result,),
+ )
+ return result
+ else:
+ return xml(''.join(renderedItems))
+
+ def listRenderCallback(self, obj, context, result):
+ results = []
+ if result.called:
+ return obj
+ for success, item in obj:
+ if success:
+ results.append(str(item))
+ else:
+ # If we had any failures, our listRenderErrback
+ # should have already been called. Ignore this.
+ return
+ result.callback(xml(''.join(results)))
+ return obj
+
+ def listRenderErrback(self, failure, result):
+ # Propagate this error to our result if it hasn't
+ # already seen an error. (This ignores all but the first
+ # error if we had multiple failures)
+ if not result.called:
+ result.errback(failure)
+
+
+class Page(resource.Resource):
+ """A web resource that renders a tree of tag instances from its 'document' attribute"""
+ serializer = TwistedSerializer()
+
+ def render(self, request):
+ context = {
+ 'owner': self,
+ 'request': request,
+ 'args': request.args, # For compatibility across systems utilizing Nouvelle
+ }
+ defer.maybeDeferred(self.preRender, context).addCallback(
+ self._afterPreRender, context).addErrback(self.pageErrorCallback, context)
+ return server.NOT_DONE_YET
+
+ def _afterPreRender(self, preRenderResult, context):
+ if preRenderResult is not None:
+ if type(preRenderResult) in (str, unicode):
+ context['request'].write(preRenderResult)
+ context['request'].finish()
+ else:
+ assert preRenderResult == server.NOT_DONE_YET
+ return
+
+ request = context['request']
+ defer.maybeDeferred(self.serializer.render, self.document, context).addCallback(
+ self.pageFinishedCallback, context).addErrback(self.pageErrorCallback, context)
+
+ def pageFinishedCallback(self, obj, context):
+ """Callback for asynchronous page rendering from a Deferred object"""
+ context['request'].write(str(obj))
+ context['request'].finish()
+
+ def pageErrorCallback(self, failure, context):
+ """Error handler for pages rendered asynchronously"""
+ context['request'].processingFailed(failure)
+
+ def preRender(self, context):
+ """Called prior to rendering each request, subclasses can use this to annotate
+ 'context' with extra information or perform other important setup tasks.
+ If this returns a Deferred, rendering will be delayed until it is resolved.
+ If it returns anything but None, normal rendering is aborted and render() returns
+ that value.
+ """
+ pass
+
+### The End ###
Added: trunk/community/infrastructure/Nouvelle/__init__.py
===================================================================
--- trunk/community/infrastructure/Nouvelle/__init__.py (rev 0)
+++ trunk/community/infrastructure/Nouvelle/__init__.py 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,140 @@
+""" Nouvelle
+
+Nouvelle is a simple web framework, similar to the 'nevow' package
+in Quotient. Nouvelle has many similarities to nevow, but several
+key differences:
+
+ - nevow is based on Twisted, while Nouvelle can work in any web
+ server environment without requiring more than Python's standard
+ library. Nouvelle supports twisted.web and deferred rendering,
+ but this is in an add-on rather than in the core of Nouvelle.
+
+ - While nevow is a relatively simple web framework, Nouvelle is
+ even simpler, making it very easy to understand and quick to load.
+
+ - nevow keeps as much separation of content and presentation as
+ possible by using a registry of ISerializable adaptors for various
+ data types. Nouvelle takes a simpler approach in which a Serializer
+ object looks up a method to handle each type. The default Serializer
+ includes methods for important builtin types, but instead of
+ encouraging the user to create ISerializable adaptors for their
+ data, Nouvelle encourages the user to create objects with a 'render'
+ method that may wrap a data model object if necessary.
+
+ - When a tag from nevow's 'stan' module is called or indexed into, to
+ change its attributes or content, the tag is modified and returns
+ itself. This causes problems when you want to create one tag as
+ a template then create tags based on it without modifying the template.
+ Nouvelle's tags are normally immutable, so this sort of template
+ system is quite easy.
+
+Like nevow's 'stan' module, Nouvelle has a simple pure-python syntax
+for expressing XHTML documents efficiently. The 'tag' object is a class,
+which must be instantiated with a tag name. Keyword arguments are converted
+to tag attributes. Leading underscores are stripped, so it's still possible
+to represent attributes that are reserved words in Python. Once a tag object
+is instantiated, it can be called again to instantiate a new tag with a name
+and/or attributes based on the original tag. Finally, using square brackets
+a new tag may be instantiated with the same name and attributes but new
+content, described in the brackets. Some examples...
+
+ uvLink = tag('a', _class='ultraviolet')
+ warning = tag('strong')[
+ 'Your spam is on ',
+ uvLink(href='fire_safety.html')[ 'fire' ],
+ ]
+
+ page = tag('html')[
+ tag('head')[
+ tag('title')[ 'Slinkies Around the World' ],
+ ],
+ tag('body')[
+ warning,
+ ],
+ ]
+
+All tag objects are renderable- that is, they have a 'render' method that
+can be called to convert that tag and its contents recursively to an 'xml'
+object. Normally, Nouvelle quotes all strings appropriately. xml objects
+are just special string objects that signify data that's already in XML.
+
+Any python object can be embedded inside a tag. Renderable objects will
+have their 'render' method called, and the returned data will be processed
+in place of the renderable objects. Lists and tuples will have their contents
+concatenated. Strings are quoted and inserted into the resulting document.
+Other objects are casted to strings.
+
+It is common to include tag trees as class attributes. This is efficient,
+as the tag tree only needs to be constructed once, and during initialization
+the 'tag' object can pre-serialize itself. However, it also means the tags
+can't contain any dynamic content. This is where the 'place' renderable
+comes in handy. It acts as a placeholder for dynamic information.
+
+A place instance is created with the name of a piece of information, and
+optionally with extra parameters. If the object owning the current tag tree
+(context['owner']) has a render_* method for the given name, it is called
+to return a value for the placeholder. If not, the placeholder will look
+for an attribute by the given name and call it. This is helpful when you
+want to insert information that is usually static, but should be overridable
+by subclasses. For example:
+
+class WebThingy(DocumentOwner):
+ title = 'Default Title'
+
+ def render_body(self, context):
+ return fetchSomeData()
+
+ document = [
+ tag('h3')[ place('title') ],
+ tag('p')[ place('body') ],
+ ]
+
+The DocumentOwner class our hypothetical WebThingy subclasses can be anything
+that context['owner'] to itself before serializing self.document. The 'context'
+dictionary is passed to all render() functions, starting with that of the
+Serializer. It keeps track of important extra data, without requiring rendering
+functions to know what that data might be. It can be used to hold information
+about the current web request, for example.
+"""
+#
+# Nouvelle web framework
+# Copyright (C) 2003-2005 Micah Dowty <micahjd at users.sourceforge.net>
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+
+__version__ = "1.0"
+
+# The third number will be zero for releases, and an increasing
+# number for development versions.
+version_info = (1, 0, 0)
+
+# Check the python version here before we proceed further
+requiredPythonVersion = (2,2,1)
+import sys, string
+if sys.version_info < requiredPythonVersion:
+ raise Exception("%s requires at least Python %s, found %s instead." % (
+ name,
+ string.join(map(str, requiredPythonVersion), "."),
+ string.join(map(str, sys.version_info), ".")))
+del sys
+del string
+
+# Convenience imports
+import Serial, Table
+from Serial import *
+from Table import *
+
+### The End ###
Added: trunk/community/infrastructure/check-static
===================================================================
--- trunk/community/infrastructure/check-static (rev 0)
+++ trunk/community/infrastructure/check-static 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+BASE=/var/lib/gforge/chroot/home/groups/debian-med
+
+cp -rf $BASE/static/* $BASE/htdocs/
Property changes on: trunk/community/infrastructure/check-static
___________________________________________________________________
Name: svn:executable
+ *
Added: trunk/community/infrastructure/update-bugs
===================================================================
--- trunk/community/infrastructure/update-bugs (rev 0)
+++ trunk/community/infrastructure/update-bugs 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,276 @@
+#!/usr/bin/python
+
+#
+# This Python script is:
+# (C) 2007, David Paleino <d.paleino at gmail.com>
+#
+# It is licensed under the terms of GNU General Public License (GPL) v2 or later.
+#
+
+import SOAPpy
+import HTMLTemplate
+import os
+import re
+from datetime import datetime
+from email.Utils import formatdate
+import time
+from Tools import parseTasks
+
+url = "http://bugs.debian.org/cgi-bin/soap.cgi"
+base = "/var/lib/gforge/chroot/home/groups/debian-med"
+namespace = "Debbugs/SOAP"
+maint = "debian-med-packaging at lists.alioth.debian.org"
+tasks_repos = "svn://svn.debian.org/svn/cdd/projects/med/trunk/debian-med/tasks/"
+
+soap = SOAPpy.SOAPProxy(url, namespace)
+
+def getStatus(*args):
+ return soap.get_status(*args)
+
+def getBugs(*args):
+ """
+ Possible keys are:
+ package, submitter, maint, src, severity, status, tag, owner, bugs
+
+ To be called like getBugs(key, value), i.e.:
+
+ getBugs("maint", "my at address.com")
+ """
+ return soap.get_bugs(*args)
+
+def getField(bug, field):
+ return getStatus(bug).__getitem__('item').__getitem__('value').__getitem__(field)
+
+#<SOAPpy.Types.structType s-gensym3 at -1216426292>: {'item':
+#<SOAPpy.Types.structType item at -1217077972>: {'value':
+#<SOAPpy.Types.structType value at -1217031444>: {
+#'fixed_versions': [],
+#'blockedby': '',
+#'keywords': 'unreproducible moreinfo',
+#'done': '"Nelson A. de Oliveira" <naoliv at gmail.com>',
+#'unarchived': '',
+#'owner': '',
+#'id': 444343,
+#'subject': 'mummer: needs build dependency on libc headers and C++ compiler',
+#'forwarded': '',
+#'msgid': '<2007092721230716338.7665.reportbug at toblerone.sgn.cornell.edu>',
+#'location': 'db-h',
+#'pending': 'done',
+#'found_date': [],
+#'originator': 'Robert Buels <rmb32 at cornell.edu>',
+#'blocks': '',
+#'tags': 'unreproducible moreinfo',
+#'date': 1190928242,
+#'mergedwith': '',
+#'severity': 'important',
+#'package': 'mummer',
+#'log_modified': 1191328025,
+#'fixed_date': [],
+#'found_versions': ['mummer/3.19-1'],
+#'found': <SOAPpy.Types.structType found at -1216488532>: {'item':
+#<SOAPpy.Types.structType item at -1216488340>: {'value': None, 'key':
+#'mummer/3.19-1'}}, 'fixed': ''}, 'key': 444343}}
+
+def getSubject(bug):
+ return getField(bug, "subject")
+
+def getOriginator(bug):
+ return getField(bug, "originator")
+
+def getCloser(bug):
+ return getField(bug, "done")
+
+def getTags(bug):
+ return getField(bug, "tags")
+
+def getDate(bug):
+ return getField(bug, "date")
+
+def getSeverity(bug):
+ return getField(bug, "severity")
+
+def getPackage(bug):
+ return getField(bug, "package")
+
+def getAffectedVersions(bug):
+ return getField(bug, "found_versions")["data"]
+
+def getFixedVersions(bug):
+ return getField(bug, "fixed_versions")["data"]
+
+def getPending(bug):
+ return getField(bug, "pending")
+
+#
+# We're going to handle the HTML template here
+#
+
+# this is used later, for single-package pages
+pairs = []
+
+def renderTemplate(node, package, bugs):
+ global pairs
+ node.package.content = package
+ node.count.content = str(len(bugs))
+
+ t = datetime.now()
+ node.date.content = formatdate(time.mktime(t.timetuple()))
+ #node.date.content = datetime.datetime.now().strftime("%a, %d %b %Y %H:%M:%S %Z")
+ #Sat, 13 Oct 2007 10:39:27 +0200
+
+ bugs.sort()
+ openbugs = {}
+ fixedbugs = {}
+ opencount = 0
+ fixedcount = 0
+ for tuple in bugs:
+ name, id = tuple
+ if getPending(id) != "done":
+ if name not in openbugs:
+ openbugs[name] = []
+ openbugs[name].append(id)
+ opencount += 1
+ else:
+ if name not in fixedbugs:
+ fixedbugs[name] = []
+ fixedbugs[name].append(id)
+ fixedcount += 1
+
+ openlist = openbugs.items()
+ openlist.sort()
+ node.opencount.content = str(opencount)
+
+ fixedlist = fixedbugs.items()
+ fixedlist.sort()
+ node.fixedcount.content = str(fixedcount)
+
+ tmp = openlist + fixedlist
+ dict = {}
+ for tuple in tmp:
+ name, bugs = tuple
+ if name not in dict:
+ dict[name] = []
+ for bug in bugs:
+ dict[name].append(bug)
+ for item in dict:
+ pairs.append((item, dict[item]))
+
+ node.openlist.repeat(renderPackage, openlist)
+ node.fixedlist.repeat(renderPackage, fixedlist, True)
+
+def renderPackage(node, buglist, showfixed = False):
+ name, bugs = buglist
+ node.name.content = name
+ node.count.content = str(len(bugs))
+ node.numbers.repeat(renderBugs, bugs, name, showfixed)
+
+def renderBugs(node, number, package, showfixed):
+ #url = "bug_details.php?id=%s" % str(number)
+ url = "pkgs/%s.php#%s" % (package, str(number))
+
+ severity = getSeverity(number)
+ subject = getSubject(number)
+
+ node.id.atts["class"] = "bugid %s" % severity
+ node.id.raw = '<a href="%s">%s</a>' % (url, str(number))
+ node.summary.atts["class"] = "summary %s" % severity
+ node.summary.raw = '<a href="%s">%s</a>' % (url, subject)
+ node.severity.atts["class"] = "severity %s" % severity
+ node.severity.content = severity
+
+ if showfixed:
+ node.fixed.atts["class"] = "fixed %s" % severity
+ value = "<br />".join(getFixedVersions(number)).replace("%s/" % package, "")
+ node.fixed.raw = value
+
+#
+# These are the single-package pages method
+#
+
+def renderStatic(node, package, bugs):
+ node.package.content = package
+ node.count.content = str(len(bugs))
+
+ t = datetime.now()
+ node.date.content = formatdate(time.mktime(t.timetuple()))
+
+ node.allbugs.repeat(renderStaticBugs, bugs, package)
+
+def renderStaticBugs(node, bug, package):
+ node.id.content = str(bug)
+ node.id.atts["href"] = "http://bugs.debian.org/%s" % str(bug)
+ node.id.atts["name"] = str(bug)
+ fields = ["Subject",
+ "Sender",
+ "Tags",
+ "Date",
+ "Severity",
+ "Found in",
+ "Fixed in"]
+ node.singlebug.repeat(renderStaticBugDetails, fields, bug, package)
+
+def renderStaticBugDetails(node, field, bug, package):
+ node.field.raw = '<?=_("%s")?>' % field
+ if field == "Subject":
+ value = getSubject(bug)
+ elif field == "Sender":
+ value = getOriginator(bug)
+ elif field == "Tags":
+ value = getTags(bug)
+ elif field == "Date":
+ value = datetime.fromtimestamp(getDate(bug)).strftime("%c")
+ elif field == "Severity":
+ value = getSeverity(bug)
+ elif field == "Found in":
+ value = " ".join(getAffectedVersions(bug)).replace("%s/" % package, "")
+ elif field == "Fixed in":
+ value = " ".join(getFixedVersions(bug)).replace("%s/" % package, "")
+
+ node.value.content = str(value).strip()
+
+#
+# Let's parse our task files (from Tools.py)
+#
+
+tasks_packages = parseTasks()
+
+#
+# Let's get our bugs
+#
+
+bugs = []
+for package in tasks_packages:
+ for bug in getBugs("package", package):
+ bugs.append((package, bug))
+
+#bugs = [("a",1),("b",2),("b",3),("c",0),("c",4),("a",67809),("c",1208)]
+
+#
+# Let's render bugs.php
+#
+
+f = open("%s/htdocs/bugs.tmpl" % base)
+tmpl = HTMLTemplate.Template(renderTemplate, f.read())
+f.close()
+
+package = "all"
+
+f = open("%s/static/bugs.php" % base, "w")
+f.write(tmpl.render(package, bugs))
+f.close()
+
+#
+# Let's do the <package>.php pages
+#
+
+for pair in pairs:
+ name, bugs = pair
+ outfile = "%s.php" % name
+
+ f = open("%s/htdocs/bug_details.tmpl" % base)
+ tmpl = HTMLTemplate.Template(renderStatic, f.read())
+ f.close
+
+ f = open("%s/static/pkgs/%s" % (base, outfile), "w")
+ f.write(tmpl.render(name, bugs))
+ f.close()
Property changes on: trunk/community/infrastructure/update-bugs
___________________________________________________________________
Name: svn:executable
+ *
Added: trunk/community/infrastructure/update-ddtp
===================================================================
--- trunk/community/infrastructure/update-ddtp (rev 0)
+++ trunk/community/infrastructure/update-ddtp 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,194 @@
+#!/usr/bin/python
+
+#
+# This Python script is:
+# (C) 2007, David Paleino <d.paleino at gmail.com>
+#
+# It is licensed under the terms of GNU General Public License (GPL)
+# v3, or any later revision.
+#
+
+from urllib import urlopen, urlretrieve
+import re
+import HTMLTemplate
+from datetime import datetime
+from email.Utils import formatdate
+import time
+from Tools import parseTasks, grep
+
+base = "/var/lib/gforge/chroot/home/groups/debian-med"
+#base = "/home/neo/tmp/debmed"
+ddtp_url = "http://ddtp.debian.net/debian/dists/sid/main/i18n/Translation-%s"
+fetch_url = "http://kleptog.org/cgi-bin/ddtss2-cgi/%s/fetch?package=%s"
+trans_url = "http://kleptog.org/cgi-bin/ddtss2-cgi/%s/translate/%s"
+
+# Should we dinamically get this list from PO translations
+# of the website?
+langs = ["de", "fr", "it", "pt_BR", "es", "ja"]
+langs.sort()
+
+dict = {}
+longs = {}
+shorts = {}
+
+#['adun.app', 'amap-align', 'amide', 'arb', 'bioimagesuite', 'bioperl', 'biosquid', 'blast2', 'boxshade', 'chemtool',
+#'cimg-dev', 'clustalw', 'clustalw-mpi', 'clustalx', 'ctn', 'ctn-dev', 'ctsim', 'cycle', 'dcmtk', 'dialign',
+#'dialign-t', 'dicomnifti', 'emboss', 'fastdnaml', 'fastlink', 'garlic', 'gdpc', 'gff2aplot', 'gff2ps', 'ghemical',
+#'gnumed-client', 'gromacs', 'hmmer', 'imagej', 'kalign', 'libbio-ruby', 'libfslio0', 'libfslio0-dev',
+#'libinsighttoolkit-dev', 'libmdc2-dev', 'libminc0-dev', 'libncbi6-dev', 'libniftiio0', 'libniftiio0-dev', 'loki',
+#'mcl', 'medcon', 'melting', 'mencal', 'minc-tools', 'mipe', 'molphy', 'mummer', 'muscle', 'ncbi-epcr',
+#'ncbi-tools-bin', 'ncbi-tools-x11', 'nifti-bin', 'njplot', 'octave', 'octave2.1', 'paw++', 'perlprimer', 'phylip',
+#'poa', 'primer3', 'probcons', 'proda', 'pymol', 'python-biopython', 'python-nifti', 'r-base', 'r-base-core',
+#'r-cran-qtl', 'rasmol', 'readseq', 'seaview', 'sibsim4', 'sigma-align', 'sim4', 't-coffee', 'tigr-glimmer',
+#'tm-align', 'tree-ppuzzle', 'tree-puzzle', 'treetool', 'treeviewx', 'wise', 'xmedcon', 'zope-zms']
+
+def force_fetch(packages, lang = langs):
+ global langs
+
+ # If it's a single language to be updated...
+ #
+ # NOT USED CURRENTLY.
+ if type(lang) == "str":
+ for name in packages:
+ urlopen(fetch_url % (lang, name))
+ else:
+ for lang_name in lang:
+ for pkg_name in packages:
+ #print "Fetching %s - %s" % (lang_name, pkg_name)
+ urlopen(fetch_url % (lang_name, pkg_name))
+
+def get_status(package):
+ global dict
+ global langs
+ status = {}
+
+ for lang in langs:
+# print "Parsing %s - %s" % (package, lang)
+ if grep("Package: %s\n" % package, "%s/data/ddtp/Translation-%s" % (base, lang)):
+ status[lang] = 1
+ else:
+ status[lang] = 0
+
+ dict[package] = status
+
+def parse_description(package):
+ global dict
+ global longs
+ global shorts
+
+ tmp_long = {}
+ tmp_short = {}
+ for lang in langs:
+ if dict[package][lang]:
+ f = open("%s/data/ddtp/Translation-%s" % (base, lang), "r")
+ # Package: 9wm\nDescription-md5: (?P<md5sum>.{32})\nDescription-\w+: (?P<short>.*)\n(?P<long>.*\n)\n
+ # TODO: Fix the regex. (it worked in Kodos :()
+ regex = r"""Package: %s
+Description-md5: (?P<md5sum>.{32})
+Description-\w+: (?P<short>.*)
+(?P<long>(^ .*$\n)+\n?)""" % package
+
+ p = re.compile(regex, re.MULTILINE)
+ m = p.search(f.read())
+ if m:
+ tmp_long[lang] = m.group("long").replace("\n ", "<br />").replace("<br />.<br />", "<br /><br />")
+ tmp_short[lang] = m.group("short")
+ f.close()
+
+ longs[package] = tmp_long
+ shorts[package] = tmp_short
+
+def renderTemplate(node, langs, statuses):
+ node.langs.repeat(renderLangs, langs)
+
+ names = statuses.keys()
+ names.sort()
+
+ node.packages.repeat(renderStatuses, names, statuses)
+
+ t = datetime.now()
+ node.date.content = formatdate(time.mktime(t.timetuple()))
+
+def renderLangs(node, lang):
+ node.code.raw = "<img src=\"/img/langs/%s.png\" alt=\"%s\" />" % (lang, lang)
+
+def renderStatuses(node, package, status):
+ node.name.raw = '<a href="/ddtp/%s.php">%s</a>' % (package, package)
+
+ langs = dict[package].keys()
+ langs.sort()
+
+ node.translations.repeat(renderTrans, langs, package, dict[package])
+
+def renderTrans(node, lang, package, status):
+ if status[lang] == 0:
+ node.status.raw = '<a href="%s"><img src="/img/no.png" alt="no" title="%s - <?=_("translation not available")?>" /></a>' % (trans_url % (lang, package), lang)
+ else:
+ node.status.raw = '<a href="%s"><img src="/img/ok.png" alt="<?=_("yes")?>" title="%s - <?=_("translated")?>" /></a><a href="%s"><img src="/img/go.png" alt="<?=_("edit")?>" title="%s - <?=_("edit translation")?>" /></a>' % ("/ddtp/%s.php" % package, lang, trans_url % (lang, package), lang)
+
+#tem:
+# con:package
+# rep:langs
+# con:name
+# con:short
+# con:long
+def renderStatic(node, package):
+ node.package.content = package
+ node.langs.repeat(renderSingleTrans, dict[package], package)
+
+ t = datetime.now()
+ node.date.content = formatdate(time.mktime(t.timetuple()))
+
+def renderSingleTrans(node, lang, package):
+ node.name.raw = '<img src="/img/langs/%s.png" alt="%s" title="%s"/>' % (lang, lang, lang)
+ if dict[package][lang]:
+ node.short.raw = shorts[package][lang]
+ node.long.raw = longs[package][lang]
+ else:
+ node.short.raw = '<?=_("untranslated")?>'
+ node.long.raw = '<?=_("untranslated")?><br /><?=_("Please follow the link below to start translating")?>:<br /><br /><a href="%s">%s</a>' % (trans_url % (lang, package), trans_url % (lang, package))
+
+for lang in langs:
+ url = ddtp_url % lang
+ pos = url.rfind("/")
+ name = "%s/data/ddtp/%s" % (base, url[pos + 1:])
+ urlretrieve(url, name)
+
+packages = parseTasks()
+#packages = ["adun.app", "gcc", "emboss-explorer"]
+packages.sort()
+
+# XXX TODO HACK NEWS
+# Re-enable this to force-fetch the packages.
+# Shouldn't be done unless having talked to DDTSS admins.
+# -- David
+#force_fetch(packages)
+for item in packages:
+ get_status(item)
+ parse_description(item)
+
+#print dict
+#print longs
+#print shorts
+
+# Let's generate the overview page
+f = open("%s/htdocs/ddtp.tmpl" % base)
+tmpl = HTMLTemplate.Template(renderTemplate, f.read())
+f.close()
+
+f = open("%s/static/ddtp.php" % base, "w")
+f.write(tmpl.render(langs, dict))
+f.close()
+
+# Let's generate nice static overview per-package pages
+for name in packages:
+ outfile = "%s/static/ddtp/%s.php" % (base, name)
+ f = open("%s/htdocs/ddtp_details.tmpl" % base)
+ tmpl = HTMLTemplate.Template(renderStatic, f.read())
+ f.close()
+
+# print tmpl.structure()
+
+ f = open(outfile, "w")
+ f.write(tmpl.render(name))
+ f.close()
Property changes on: trunk/community/infrastructure/update-ddtp
___________________________________________________________________
Name: svn:executable
+ *
Added: trunk/community/infrastructure/update-tasks
===================================================================
--- trunk/community/infrastructure/update-tasks (rev 0)
+++ trunk/community/infrastructure/update-tasks 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,319 @@
+#!/usr/bin/python -W ignore
+
+#
+# This Python script is:
+# (C) 2007, David Paleino <d.paleino at gmail.com>
+#
+# It is licensed under the terms of GNU General Public License (GPL)
+# v3, or any later revision.
+#
+
+import apt
+import apt_pkg
+import apt_inst
+import HTMLTemplate
+import re
+import sys
+import time
+from datetime import datetime
+from email.Utils import formatdate
+from Tools import *
+
+base = "/var/lib/gforge/chroot/home/groups/debian-med"
+tasks = "%s/scripts/tasks" % base
+
+official = {} # Official packages
+todo = {} # Packages not in repositories, nor unofficial,
+ # nor prospected. They will eventually go into
+ # "unavailable".
+det = {} # Official Packages details
+
+# let's get our nice dict in the form:
+# { 'task_foo': ['package1', 'package2', '...'],
+# 'task_bar': ['package3', 'package4', '...']}
+
+packages = parseTasks(None, True)
+unofficial = parseTasksNonOff()
+unavailable = parseTasksUnavail()
+task_details = parseTaskDetails()
+
+tasks = packages.keys()
+tasks.sort()
+
+apt_pkg.init()
+#~ apt_pkg.Config.Set("APT::Acquire::Translation", "it")
+
+cache = apt_pkg.GetCache()
+depcache = apt_pkg.GetDepCache(cache)
+aptcache = apt.Cache()
+
+###
+# Wrappers around apt_pkg
+###
+
+def __getRecords(package):
+ ### TODO: convert to Python API
+ (f, index) = depcache.GetCandidateVer(cache[package]).TranslatedDescription.FileList.pop(0)
+ records = apt_pkg.GetPkgRecords(cache)
+ records.Lookup ((f, index))
+ return records
+
+def __getDesc(package):
+ regex = r"""(?P<short>.*)
+(?P<long>(^ .*$\n)+\n?)"""
+
+ p = re.compile(regex, re.MULTILINE)
+ m = p.match(getSections(package)['Description'])
+ if m:
+ return {'ShortDesc': m.group("short"), 'LongDesc': m.group("long")}
+ else:
+ return False
+
+def getShort(package):
+ ### TODO: convert to Python API
+ desc = __getDesc(package)
+ if desc:
+ return desc['ShortDesc']
+ else:
+ # Fallback to the C++ wrapper
+ return __getRecords(package).ShortDesc
+
+def getLong(package):
+ ### TODO: convert to Python API
+ desc = __getDesc(package)
+ if desc:
+ return desc['LongDesc']
+ else:
+ # Fallback to the C++ wrapper
+ return __getRecords(package).LongDesc
+
+def getHomepage(package):
+ sect = getSections(package)
+ try:
+ return sect['Homepage']
+ # Fallback to the old " Homepage: ..." pseudo-field
+ # TODO: also renders _wrong_ "URL" pseudo-fields! Fix the packages!
+ except:
+ p = re.compile(".*(?P<field>Homepage|URL): (?P<url>.*)", re.DOTALL)
+ m = p.match(getLong(package))
+ if m:
+ tmp = det[package]['LongDesc']
+ det[package]['LongDesc'] = tmp.replace(m.group("field") + ": " + m.group("url"), "")
+ return m.group("url")
+ else:
+ # We don't have any valid field for homepage,
+ # return a suitable "#" for <a href>s.
+ return "#"
+
+def getDebUrl(package):
+ try:
+ return getSections(package)["Filename"]
+ except:
+ return "#"
+
+def getVersion(package):
+ try:
+ return getSections(package)["Version"]
+ except:
+ # Fallback to the C++ wrapper
+ for pkg in cache.Packages:
+ if pkg.Name in det:
+ if not pkg.VersionList:
+ return "N/A"
+ else:
+ return pkg.VersionList[0].VerStr
+
+def getLicense(package):
+ ### FIX
+ return "GPL-foo"
+
+def getSections(package):
+ pkg = aptcache[package]
+ pkg._lookupRecord(True)
+ return apt_pkg.ParseSection(pkg._records.Record)
+
+###
+# Template handlers
+###
+
+def renderIndex(node, tasks):
+ node.tasks.repeat(renderTaskList, tasks)
+ t = datetime.now()
+ node.date.content = formatdate(time.mktime(t.timetuple()))
+
+def renderTaskList(node, task):
+ node.task.raw = """<a href="/tasks/%s.php" name="%s" id="%s">%s</a>""" % (task, task, task, task.capitalize())
+
+def renderTasks(node, task, packages, details):
+ node.task.content = details['Task']
+ node.shortdesc.content = details['ShortDesc']
+ node.heading.content = details['ShortDesc']
+ node.longdesc.content = details['LongDesc']
+
+ t = datetime.now()
+ node.date.content = formatdate(time.mktime(t.timetuple()))
+
+ # Let's separate official packages from others
+ for pkg in packages:
+ # If the package has a short description in cache,
+ # there's an high chance it is an official package.
+ # Probably we can use a better algorithm? (I believe
+ # Alioth's APT cache won't be contaminated by external
+ # repositories)
+ try:
+ short = getShort(pkg)
+ if not task in official:
+ official[task] = []
+ official[task].append(pkg)
+ det[pkg] = {}
+ det[pkg]['ShortDesc'] = short
+ det[pkg]['LongDesc'] = getLong(pkg).replace("%s\n" % short, "")
+ det[pkg]['LongDesc'] = det[pkg]['LongDesc'].replace("<", "<").replace(">", ">")
+ # getHomepage() does some magic on ['LongDesc']
+ det[pkg]['Homepage'] = getHomepage(pkg)
+ det[pkg]['LongDesc'] = det[pkg]['LongDesc'].replace("\n .\n", "<br /><br />").replace("\n", "")
+ det[pkg]['Version'] = getVersion(pkg)
+ det[pkg]['License'] = getLicense(pkg)
+ det[pkg]['Task'] = task
+ det[pkg]['Pkg-URL'] = "http://packages.debian.org/unstable/%s/%s" % (getSections(pkg)['Section'], pkg)
+ ### BUG: some packages don't get the right Filename
+ ### see, for example, "treeviewx": it has a Filename
+ ### field, but doesn't get parsed.
+ ### FIX: installed packages with versions newer than the
+ ### one in repositories don't have that field. :@!
+ det[pkg]['Deb-URL'] = "http://ftp.debian.org/%s" % getDebUrl(pkg)
+ except:
+ pass
+
+ if task in official:
+ node.official_head.raw = """<h2>
+<a id="official-debs" name="official-debs"></a>
+ Official Debian packages
+</h2>"""
+ node.official.repeat(renderOfficial, official[task], task)
+
+ if task in todo:
+ error = True
+ else:
+ error = False
+
+ if task in unofficial:
+ node.unofficial_head.raw = """<h2>
+<a id="inofficial-debs" name="inofficial-debs"></a>
+ Inofficial Debian packages
+</h2>"""
+ node.unofficial.repeat(renderUnofficial, unofficial[task])
+ error = False
+
+ if task in unavailable:
+ node.unavailable_head.raw = """<h2>
+<a id="debs-not-available" name="debs-not-available"></a>
+ Debian packages not available
+</h2>"""
+ node.unavailable.repeat(renderUnavailable, unavailable[task])
+ error = False
+
+ if error:
+ # The package probably needs a proper prospective entry in the
+ # task files. Write it to stdout.
+ print "Error: problems with %s" % task
+
+def renderOfficial(node, package, task):
+ # Here we parse just official packages
+ node.shortdesc.content = det[package]['ShortDesc']
+ node.project.raw = "<table class=\"project\" summary=\"%s\">" % package
+ node.anchor.atts['name'] = package
+ node.anchor.atts['id'] = package
+ node.name.content = package.capitalize()
+ node.url.atts['href'] = det[package]['Homepage']
+ if det[package]['Homepage'] == "#":
+ node.url.content = "Homepage not available"
+ else:
+ node.url.content = det[package]['Homepage']
+
+ node.longdesc.raw = det[package]['LongDesc']
+ node.version.content = "Version: %s" % det[package]['Version']
+ node.license.content = "License: %s" % det[package]['License']
+ node.pkgurl.atts['href'] = det[package]['Pkg-URL']
+ node.pkgurl.content = "Official Debian package"
+ node.deburl.atts['href'] = det[package]['Deb-URL']
+ #~ node.deburl.content = "X" ### TODO: add a nice icon here to download the .deb package
+ node.deburl.raw = "<img src=\"/img/deb-icon.png\" />"
+
+def renderUnofficial(node, package):
+ # Here we parse just unofficial packages
+ node.shortdesc.content = package['ShortDesc']
+ node.longdesc.raw = package['LongDesc']
+ node.project.raw = "<table class=\"project\" summary=\"%s\">" % package['Package']
+ node.anchor.atts['name'] = package['Package']
+ node.anchor.atts['id'] = package['Package']
+ node.name.content = package['Package'].capitalize()
+ node.url.atts['href'] = package['Homepage']
+ node.url.content = package['Homepage']
+ node.license.content = "License: %s" %package['License']
+ node.pkgurl.atts['href'] = package['Pkg-URL']
+ node.pkgurl.content = "Inofficial Debian package"
+
+ # Let's try to get the version from the package name
+ # (following Debian standards: <name>_<ver>_<arch>.deb)
+ regex = ".*/%s_(?P<version>.*)_.*\.deb$" % package['Package']
+ p = re.compile(regex)
+ m = p.search(package['Pkg-URL'])
+ if m:
+ node.version.content = "Version: %s" % m.group("version")
+ else:
+ node.version.content = "Version: N/A"
+
+
+def renderUnavailable(node, package):
+ # Parsing unavailable packages :(
+ # PACKAGE THEM! :)
+ name = package['Package']
+ if package['ShortDesc']:
+ node.shortdesc.content = package['ShortDesc']
+ else:
+ node.shortdesc.content = "N/A"
+ node.longdesc.raw = package['LongDesc']
+ node.project.raw = "<table class=\"project\" summary=\"%s\">" % name
+ if package['Responsible']:
+ node.responsible.content = package['Responsible']
+ else:
+ node.responsible.raw = "no one"
+ if package['WNPP']:
+ node.wnpp.raw = " — <a href=\"http://bugs.debian.org/%s\">wnpp</a>" % package['WNPP']
+ node.anchor.atts['name'] = name
+ node.anchor.atts['id'] = name
+ node.name.content = name.capitalize()
+ if package['Homepage']:
+ node.url.atts['href'] = package['Homepage']
+ node.url.content = package['Homepage']
+ else:
+ node.url.atts['href'] = "#"
+ node.url.content = "N/A"
+ if package['License']:
+ node.license.raw = "<?=_('License')?>: %s" % package['License']
+ else:
+ node.license.raw = "<?=_('License')?>: N/A"
+
+# Let's render the Tasks Page index, first
+f = open("%s/htdocs/tasks_idx.tmpl" % base)
+tmpl = HTMLTemplate.Template(renderIndex, f.read())
+f.close()
+f = open("%s/static/tasks/index.php" % base, "w")
+f.write(tmpl.render(tasks))
+f.close()
+
+# Let's render single pages now.
+f = open("%s/htdocs/tasks.tmpl" % base)
+tmpl = HTMLTemplate.Template(renderTasks, f.read())
+f.close()
+
+for task in tasks:
+ f = open("%s/static/tasks/%s.php" % (base, task), "w")
+
+ # This is to avoid useless <br>eaks before closing the cell
+ source = tmpl.render(task, packages[task], task_details[task])
+ f.write(re.sub(r"<br /><br />[ ]*</td>", "</td>", source))
+
+ f.close()
+
Property changes on: trunk/community/infrastructure/update-tasks
___________________________________________________________________
Name: svn:executable
+ *
Added: trunk/community/infrastructure/update-tasks-wrapper
===================================================================
--- trunk/community/infrastructure/update-tasks-wrapper (rev 0)
+++ trunk/community/infrastructure/update-tasks-wrapper 2007-12-30 16:44:44 UTC (rev 1047)
@@ -0,0 +1,4 @@
+#!/bin/bash
+SCRIPT=/var/lib/gforge/chroot/home/groups/debian-med/scripts/update-tasks
+
+python $SCRIPT 2&>/dev/null
Property changes on: trunk/community/infrastructure/update-tasks-wrapper
___________________________________________________________________
Name: svn:executable
+ *
More information about the debian-med-commit
mailing list