PyMOTW: SimpleXMLRPCServer

The SimpleXMLRPCServer module makes it easy to use remote procedure calls in Python that work with many other languages, too.

Module: SimpleXMLRPCServer
Purpose: Implements an XML-RPC server.
Python Version: 2.2 and later

Description:

The SimpleXMLRPCServer module contains classes for creating your own cross-platform, language-independent server using the XML-RPC protocol. Client libraries exist for many other languages, making XML-RPC an easy choice for building RPC-style services.

Note: All of the examples provided here include a client module as well to interact with the demonstration server. If you want to download the code and run the examples, you will want to use 2 separate shell windows, one for the server and one for the client.

A Simple Server:

This simple server example exposes a single function that takes the name of a directory and returns the contents (obviously a security issue, but useful for demonstration purposes). The first step is to create the SimpleXMLRPCServer instance and tell it where to listen for incoming requests ('localhost' port 9000 in this case). Then we define a function to be part of the service, and register the function so the server knows how to call it. The final step is to put the server into an infinite loop receiving and responding to requests.

from SimpleXMLRPCServer import SimpleXMLRPCServer
import logging
import os

# Set up logging
logging.basicConfig(level=logging.DEBUG)

server = SimpleXMLRPCServer(('localhost', 9000), logRequests=True)

# Expose a function
def list_contents(dir_name):
    logging.debug('list_contents(%s)', dir_name)
    return os.listdir(dir_name)
server.register_function(list_contents)

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

The server can be accessed at the URL http://localhost:9000 using xmlrpclib. This client code illustrates how to call the list_contents() service from Python.

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print proxy.list_contents('/tmp')

Notice that we simply connect the ServerProxy to the server using its base URL, and then call methods directly on the proxy. Each method invoked on the proxy is translated into a request to the server. The arguments are formatted using XML, and then POSTed to the server. The server unpacks the XML and figures out what function to call based on the method name invoked from the client. The arguments are passed to the function, and the return value is translated back to XML to be returned to the client.

Starting the server gives:

$ python SimpleXMLRPCServer_function.py 
Use Control-C to exit

Running the client in a second window shows the contents of my /tmp directory:

$ python SimpleXMLRPCServer_function_client.py 
['.s.PGSQL.5432', '.s.PGSQL.5432.lock', '.X0-lock', '.X11-unix', 'ccc_exclude.1mkahl', 
'ccc_exclude.BKG3gb', 'ccc_exclude.M5jrgo', 'ccc_exclude.SPecwL', 'com.hp.launchport', 'emacs527',
'hsperfdata_dhellmann', 'launch-8hGHUp', 'launch-RQnlcc', 'launch-trsdly', 'launchd-242.T5UzTy', 
'var_backups']

and after the request is finished you'll see log output in the server window:

$ python SimpleXMLRPCServer_function.py 
Use Control-C to exit
DEBUG:root:list_contents(/tmp)
localhost - - [29/Jun/2008 09:32:07] "POST /RPC2 HTTP/1.0" 200 -

The first line of output is from the logging.debug() call inside list_contents(). The second line is from the server logging the request because I set logRequests=True.

Alternate Names:

Sometimes the function names you use inside your modules or libraries are not the names you want to use in your external API. You might need to load a platform-specific implementation, build the service API dynamically based on a configuration file, or replace real functions with stubs for testing. If you want to register a function with an alternate name, pass the name as the second argument to register_function(), like this:

from SimpleXMLRPCServer import SimpleXMLRPCServer
import os

server = SimpleXMLRPCServer(('localhost', 9000))

# Expose a function with an alternate name
def list_contents(dir_name):
    return os.listdir(dir_name)
server.register_function(list_contents, 'dir')

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

The client should now use the name 'dir()' instead of 'list_contents()':

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print 'dir():', proxy.dir('/tmp')
print 'list_contents():', proxy.list_contents('/tmp')

Calling list_contents() results in an error, since the server no longer has a handler registered by that name.

$ python SimpleXMLRPCServer_alternate_name_client.py
dir(): ['.s.PGSQL.5432', '.s.PGSQL.5432.lock', '.X0-lock', '.X11-unix', 'ccc_exclude.1mkahl', 'ccc_exclude.BKG3gb', 'ccc_exclude.M5jrgo', 'ccc_exclude.SPecwL', 'com.hp.launchport', 'emacs527', 'hsperfdata_dhellmann', 'launch-8hGHUp', 'launch-RQnlcc', 'launch-trsdly', 'launchd-242.T5UzTy', 'temp_textmate.V6YKzm', 'var_backups']
list_contents():
Traceback (most recent call last):
  File "/Users/dhellmann/Documents/PyMOTW/in_progress/SimpleXMLRPCServer/SimpleXMLRPCServer_alternate_name_client.py", line 15, in <module>
    print 'list_contents():', proxy.list_contents('/tmp')
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1147, in __call__
    return self.__send(self.__name, args)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1437, in __request
    verbose=self.__verbose
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1201, in request
    return self._parse_response(h.getfile(), sock)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1340, in _parse_response
    return u.close()
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 787, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 1: '<type \'exceptions.Exception\'>:method "list_contents" is not supported'>

Dotted Names:

Individual functions can be registered with names that are not normally legal for Python identifiers. For example, you can include '.' in your names to separate the namespace in the service. This example extends our "directory" service to add "create" and "remove" calls. All of the functions are registered using the prefix "dir." so that the same server can provide other services using a different prefix. One other difference in this example is that some of the functions return None, so we have to tell the server to translate the None values to a nil value (see XML-RPC Extensions).

from SimpleXMLRPCServer import SimpleXMLRPCServer
import os

server = SimpleXMLRPCServer(('localhost', 9000), allow_none=True)

server.register_function(os.listdir, 'dir.list')
server.register_function(os.mkdir, 'dir.create')
server.register_function(os.rmdir, 'dir.remove')

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

To call the service functions in the client, simply refer to them with the dotted name, like so:

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print 'BEFORE       :', 'EXAMPLE' in proxy.dir.list('/tmp')
print 'CREATE       :', proxy.dir.create('/tmp/EXAMPLE')
print 'SHOULD EXIST :', 'EXAMPLE' in proxy.dir.list('/tmp')
print 'REMOVE       :', proxy.dir.remove('/tmp/EXAMPLE')
print 'AFTER        :', 'EXAMPLE' in proxy.dir.list('/tmp')

and (assuming you don't have a /tmp/EXAMPLE on your system) the output for the sample client script looks like:

$ python SimpleXMLRPCServer_dotted_name_client.py
BEFORE       : False
CREATE       : None
SHOULD EXIST : True
REMOVE       : None
AFTER        : False

Arbitrary Names:

A less useful, but potentially interesting feature is the ability to register functions with names that are otherwise invalid attribute names. This example service registers a function with the name "multiply args".

from SimpleXMLRPCServer import SimpleXMLRPCServer

server = SimpleXMLRPCServer(('localhost', 9000))

def my_function(a, b):
    return a * b
server.register_function(my_function, 'multiply args')

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

Since the registered name contains a space, we can't use dot notation to access it directly from the proxy. We can, however, use getattr().

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print getattr(proxy, 'multiply args')(5, 5)

This example is provided not necessarily because it is a good idea, but because you may encounter existing services with arbitrary names and need to be able to call them.

$ python SimpleXMLRPCServer_arbitrary_name_client.py
25

Exposing Methods of Objects:

We've talked about establishing APIs using good naming conventions. Another way to do that is to use instances of classes and expose their methods. We can recreate the first example using an instance with a single method.

from SimpleXMLRPCServer import SimpleXMLRPCServer
import os
import inspect

server = SimpleXMLRPCServer(('localhost', 9000), logRequests=True)

class DirectoryService:
    def list(self, dir_name):
        return os.listdir(dir_name)

server.register_instance(DirectoryService())

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

A client can call the method directly:

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print proxy.list('/tmp')

and receive output like:

$ python SimpleXMLRPCServer_instance_client.py
['.s.PGSQL.5432', '.s.PGSQL.5432.lock', '.X0-lock', '.X11-unix', 'ccc_exclude.1mkahl', 
'ccc_exclude.BKG3gb', 'ccc_exclude.M5jrgo', 'ccc_exclude.SPecwL', 'com.hp.launchport', 
'emacs527', 'hsperfdata_dhellmann', 'launch-8hGHUp', 'launch-RQnlcc', 'launch-trsdly', 
'launchd-242.T5UzTy', 'temp_textmate.XNiIdy', 'var_backups']

We've lost the 'dir.' prefix for the service, though, so let's define a class to let us set up a service tree that can be invoked from clients.

from SimpleXMLRPCServer import SimpleXMLRPCServer
import os
import inspect

server = SimpleXMLRPCServer(('localhost', 9000), logRequests=True)

class ServiceRoot:
    pass

class DirectoryService:
    def list(self, dir_name):
        return os.listdir(dir_name)

root = ServiceRoot()
root.dir = DirectoryService()

server.register_instance(root, allow_dotted_names=True)

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

By registering the instance of ServiceRoot with allow_dotted_names=True, we give the server permission to walk the tree of objects when a request comes in to find the named method using getattr().

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print proxy.dir.list('/tmp')

$ python SimpleXMLRPCServer_instance_dotted_names_client.py
['.s.PGSQL.5432', '.s.PGSQL.5432.lock', '.X0-lock', '.X11-unix', 'ccc_exclude.1mkahl', 'ccc_exclude.BKG3gb', 'ccc_exclude.M5jrgo', 'ccc_exclude.SPecwL', 'com.hp.launchport', 'emacs527', 'hsperfdata_dhellmann', 'launch-8hGHUp', 'launch-RQnlcc', 'launch-trsdly', 'launchd-242.T5UzTy', 'temp_textmate.adghkQ', 'var_backups']

Dispatching Calls Yourself:

By default register_instance() finds all callable attributes of the instance with names not starting with '_' and registers them with their name. If you want to be more careful about the exposed methods, you could provide your own dispatching logic. For example:

from SimpleXMLRPCServer import SimpleXMLRPCServer
import os
import inspect

server = SimpleXMLRPCServer(('localhost', 9000), logRequests=True)

def expose(f):
    "Decorator to set exposed flag on a function."
    f.exposed = True
    return f

def is_exposed(f):
    "Test whether another function should be publicly exposed."
    return getattr(f, 'exposed', False)

class MyService:
    PREFIX = 'prefix'

    def _dispatch(self, method, params):
        # Remove our prefix from the method name
        if not method.startswith(self.PREFIX + '.'):
            raise Exception('method "%s" is not supported' % method)
        
        method_name = method.partition('.')[2]
        func = getattr(self, method_name)            
        if not is_exposed(func):
            raise Exception('method "%s" is not supported' % method)
        
        return func(*params)

    @expose
    def public(self):
        return 'This is public'
        
    def private(self):
        return 'This is private'

server.register_instance(MyService())

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

The public() method of MyService is marked as exposed to the XML-RPC service while private() is not. The _dispatch() method is invoked when the client tries to access a function that is part of MyService. It first enforces the use of a prefix ("prefix." in this case, but you can use anything, of course). Then it requires the function to have an attribute called "exposed" with a true value. The exposed flag is set on a function using a decorator for convenience.

Here are a few sample client calls:

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
print 'public():', proxy.prefix.public()
try:
    print 'private():', proxy.prefix.private()
except Exception, err:
    print 'ERROR:', err
try:
    print 'public() without prefix:', proxy.public()
except Exception, err:
    print 'ERROR:', err

and the resulting output, with the expected error messages trapped and reported:

$ python SimpleXMLRPCServer_instance_with_prefix_client.py
public(): This is public
private(): ERROR: <Fault 1: '<type \'exceptions.Exception\'>:method "prefix.private" is not supported'>
public() without prefix: ERROR: <Fault 1: '<type \'exceptions.Exception\'>:method "public" is not supported'>

There are several other ways to override the dispatching mechanism, including subclassing directly from SimpleXMLRPCServer. Check out the docstrings in the module for more details.

Introspection API:

As with many network services, it is possible to query an XML-RPC server to ask it what methods it supports and learn how to use them. SimpleXMLRPCServer includes a set of public methods for performing this introspection. By default they are turned off, but can be enabled with register_introspection_functions(). You can add explicit support for system.listMethods() and system.methodHelp() by defining _listMethods() and _methodHelp() on your service class. For example,

from SimpleXMLRPCServer import SimpleXMLRPCServer, list_public_methods
import os
import inspect

server = SimpleXMLRPCServer(('localhost', 9000), logRequests=True)
server.register_introspection_functions()

class DirectoryService:
    
    def _listMethods(self):
        return list_public_methods(self)

    def _methodHelp(self, method):
        f = getattr(self, method)
        return inspect.getdoc(f)
    
    def list(self, dir_name):
        """list(dir_name) => [<filenames>]
        
        Returns a list containing the contents of the named directory.
        """
        return os.listdir(dir_name)

server.register_instance(DirectoryService())

try:
    print 'Use Control-C to exit'
    server.serve_forever()
except KeyboardInterrupt:
    print 'Exiting'

In this case, the convenience function list_public_methods() scans an instance to return the names of callable attributes that do not start with '_'. You can, of course, redefine _listMethods() to apply whatever rules you like. Similarly, for this basic example _methodHelp() returns the docstring of the function, but could be written to build a help string from any information you like.

This client queries the server and reports on all of the publicly callable methods.

import xmlrpclib

proxy = xmlrpclib.ServerProxy('http://localhost:9000')
for method_name in proxy.system.listMethods():
    print '=' * 60
    print method_name
    print '-' * 60
    print proxy.system.methodHelp(method_name)
    print

Notice that the system methods are included in the results.

$ python SimpleXMLRPCServer_introspection_client.py
============================================================
list
------------------------------------------------------------
list(dir_name) => [<filenames>]

Returns a list containing the contents of the named directory.

============================================================
system.listMethods
------------------------------------------------------------
system.listMethods() => ['add', 'subtract', 'multiple']

Returns a list of the methods supported by the server.

============================================================
system.methodHelp
------------------------------------------------------------
system.methodHelp('add') => "Adds two integers together"

Returns a string containing documentation for the specified method.

============================================================
system.methodSignature
------------------------------------------------------------
system.methodSignature('add') => [double, int, int]

Returns a list describing the signature of the method. In the
above example, the add method takes two integers as arguments
and returns a double result.

This server does NOT support system.methodSignature.

References:

XML-RPC How To
XML-RPC Extensions
Python Module of the Week Home
Download Sample Code

Technorati Tags:
python, PyMOTW

Updated 10 June to fix a typo.

PyWorks 2008: Call for papers

MTA, publishers of Python Magazine, are pleased to announce the first PyWorks conference to be held in Atlanta, GA on November 12-14, 2008.

We are currently looking for speakers and tutorial instructors on a wide range of topics. The deadline for proposals is July 25, so check out the site for details about how to submit yours today!

Python Magazine for June 2008

The June issue is available for download now.

Grig Gheorghiu's cover story on Pybots talks about the fantastic automation system that has been put in place to make sure new releases of Python software are as robust and stable as possible.

In the second part of his PyGame series, Terry Hancock adds some interactivity to the simple game he started building in the May issue. This tutorial is shaping up to be a great introduction for new programmers.

JC Cruz returns this month with another article on combining Python and OS X. This time, he shows us how to create a Cocoa application with PyObjC and Xcode 3.0. This is a topic I've been wanting to dig into myself for some time, so I was glad to have a chance to read his article.

And rounding out the features this month, Massimo Di Pierro introduces web2py, a relative newcomer on the web framework front. While originally designed primarily as a teaching tool, Massimo explains why that includes many features that make it attractive for rapid application development situations, too.

Our columns this month are all about the future:

Mark Mruss takes us quite literally into the __future__ by talking about how to take advantage of some features of Python 3.0 that are already available now in Python 2.5.

Jesse Noller talks with Adam Olsen about the "safe thread" project, a set of patches for the C interpreter that eliminate the Global Interpreter Lock and that may eventually be included in the Python core.

Steve Holden responds to Tim Bray's post Multi-Inflection-Point Alert by fixing his attention on a few technology trends and looking for portents in their convergence.

And finally, this issue also marks the transition as I take over from Brian Jones as Editor in Chief. I want to thank Brian for all he has done to teach me about the position and magazines in general. I'm looking forward to working with the excellent team of reviewers and editors we have put together as we continue to shape the magazine to be something we can all be proud to be a part of.

I'll give more details about our road ahead in another post, but for the time being go download the June issue and enjoy!

svnbackup news

I've received several patches for svnbackup.sh recently, and an offer to host an RPM version of the code. I decided it probably made sense to go ahead and open the project up to contributors, so I created a project on code.google.com.

Unfortunately, the name svnbackup was already taken, and svnbackup.sh wasn't a valid project name. I'm linking from my site to the new project, though, so hopefully the minor name change won't be too confusing.

Since I had the project on my local svn repository, and not in its own repo, importing the history into google's svn repository was something of a pain. I'm glad there are only a few files in the source tree.

There's a new release today, as well. Version 1.4 includes patches from Arjen Van Drie to fix unary
operator problems and an empty history file bug. See the downloads page for a tarball.

PyMOTW: warnings

Manage non-error alerts through the warnings module.

Module: warnings
Purpose: Deliver non-fatal alerts to the user about issues encountered when running a program.
Python Version: 2.1 and later

Description:

The warnings module was introduced in PEP 230 as a way to warn programmers about changes in language or library features in anticipation of backwards incompatible changes coming with Python 3.0. Since warnings are not fatal, a program may encounter the same warn-able situation many times in the course of running. The warnings module suppresses repeated warnings from the same source to cut down on the annoyance factor of showing the same message over and over. You can control the messages printed on a case-by-case basis using the -W option to the interpreter or by calling functions inside warnings from your code.

Categories and Filtering:

Warnings are categorized using subclasses of the builtin exception class Warning. Several standard values are described in the documentation, and you can add your own by subclassing from Warning to create a new class.

Messages are filtered using settings controlled through the -W option to the interpreter. A filter consists of 5 parts, the action, message, category, module, and line number. When a warning is generated, it is compared against all of the registered filters. The first filter that matches controls the action taken for the warning. If no filter matches, the default action is taken.

The actions understood by the filtering mechanism are:

error	Turn the warning into an exception.
ignore	Discard the warning.
always	Always emit a warning.
default	Print the warning the first time it is generated from each location.
module	Print the warning the first time it is generated from each module.
once	Print the warning the first time it is generated.

The message portion of the filter is a regular expression that is used to match the warning text.

The category is a name of an exception class, as described above.

The module contains a regular expression to be matched against the module name generating the warning.

And the line number can be used to change the handling on specific occurrences of a warning. Use 0 for the line number to have the filter apply to all occurrences.

Generating Warnings:

The simplest way to emit a warning from your own code is to just call warnings.warn() with the message as an argument:

import warnings

print 'Before the warning'
warnings.warn('This is a warning message')
print 'After the warning'

Then when your program runs, the message is printed:

$ python warnings_warn.py
/Users/dhellmann/Documents/PyMOTW/in_progress/warnings/warnings_warn.py:14: UserWarning: This is a warning message
  warnings.warn('This is a warning message')
Before the warning
After the warning

Even though the warning is printed, the default behavior is to continue past the warning and run the rest of the program. We can change that behavior with a filter:

import warnings

warnings.simplefilter('error', UserWarning)

print 'Before the warning'
warnings.warn('This is a warning message')
print 'After the warning'

This filter tells the warnings module to raise an exception when the warning is issued.

$ python warnings_warn_raise.py
Before the warning
Traceback (most recent call last):
  File "/Users/dhellmann/Documents/PyMOTW/in_progress/warnings/warnings_warn_raise.py", line 16, in <module>
    warnings.warn('This is a warning message')
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/warnings.py", line 62, in warn
    globals)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/warnings.py", line 102, in warn_explicit
    raise message
UserWarning: This is a warning message

We can, of course, also control the filter behavior from the command line. For example, if we go back to warnings_warn.py and set the filter to raise an error on UserWarning, we see the exception:

$ python -W 'error::UserWarning::0' warnings_warn.py 
Before the warning
Traceback (most recent call last):
  File "warnings_warn.py", line 14, in 
    warnings.warn('This is a warning message')
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/warnings.py", line 62, in warn
    globals)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/warnings.py", line 102, in warn_explicit
    raise message
UserWarning: This is a warning message

Since I left the fields for message and module blank, they were interpreted as matching anything.

Filtering with Patterns:

To filter on more complex rules programmatically, use filterwarnings(). For example, to filter based on the content of the message text:

import warnings

warnings.filterwarnings('ignore', '.*do not.*',)

warnings.warn('Show this message')
warnings.warn('Do not show this message')

Notice that I used "do not" in the pattern, but "Do not" in the warning. The regular expression is always compiled to look for case insensitive matches.

$ python warnings_filterwarnings_message.py
/Users/dhellmann/Documents/PyMOTW/in_progress/warnings/warnings_filterwarnings_message.py:15: UserWarning: Show this message
  warnings.warn('Show this message')

Running this source from the command line:

import warnings

warnings.warn('Show this message')
warnings.warn('Do not show this message')

yields:

$ python -W 'ignore:do not:UserWarning::0' warnings_filtering.py 
warnings_filtering.py:13: UserWarning: Show this message
  warnings.warn('Show this message')

The same pattern matching rules apply to the name of the source module containing the warning call. To suppress all warnings from the warnings_filtering module:

import warnings

warnings.filterwarnings('ignore', 
                        '.*', 
                        UserWarning,
                        'warnings_filtering',
                        )

import warnings_filtering

Since the filter is in place, no warnings are emitted when warnings_filtering is imported:

$ python warnings_filterwarnings_module.py

To suppress only the warning on line 14 of warnings_filtering:

import warnings

warnings.filterwarnings('ignore', 
                        '.*', 
                        UserWarning,
                        'warnings_filtering',
                        14)

import warnings_filtering

$ python warnings_filterwarnings_lineno.py
/Users/dhellmann/Documents/PyMOTW/in_progress/warnings/warnings_filtering.py:13: UserWarning: Show this message
  warnings.warn('Show this message')

Repeated Warnings:

By default, most types of warnings are only printed the first time they occur in a given location, where location is defined as the combination of module and line number.

import warnings

def function_with_warning():
    warnings.warn('This is a warning!')
    
function_with_warning()
function_with_warning()
function_with_warning()

$ python warnings_repeated.py
/Users/dhellmann/Documents/PyMOTW/in_progress/warnings/warnings_repeated.py:14: UserWarning: This is a warning!
  warnings.warn('This is a warning!')

The "once" action can be used to suppress instances of the same message from different locations.

import warnings

warnings.simplefilter('once', UserWarning)

warnings.warn('This is a warning!')
warnings.warn('This is a warning!')
warnings.warn('This is a warning!')

$ python warnings_once.py
/Users/dhellmann/Documents/PyMOTW/in_progress/warnings/warnings_once.py:15: UserWarning: This is a warning!
  warnings.warn('This is a warning!')

Similarly, "module" will suppress repeated messages from the same module, no matter what line number.

Alternate Message Delivery Functions:

Normally warnings are printed to sys.stderr. You can change that behavior by replacing the showwarning() function inside the warnings module. For example, if you wanted warnings to go to a log file instead of stderr, you could replace showwarning() with a function like this:

import warnings
import logging

logging.basicConfig(level=logging.INFO)

def send_warnings_to_log(message, category, filename, lineno, file=None):
    logging.warning(
        '%s:%s: %s:%s' % 
        (filename, lineno, category.__name__, message))
    return

old_showwarning = warnings.showwarning
warnings.showwarning = send_warnings_to_log

warnings.warn('This is a warning message')

So that when warnings.warn() is called, the warnings are emitted with the rest of the log messages.

$ python warnings_showwarning.py
WARNING:root:/Users/dhellmann/Documents/PyMOTW/in_progress/warnings/warnings_showwarning.py:25: UserWarning:This is a warning message

Formatting:

If it is OK for warnings to go to stderr, but you don't like the formatting, you can replace formatwarning() instead.

import warnings

def warning_on_one_line(message, category, filename, lineno):
    return '%s:%s: %s:%s' % (filename, lineno, category.__name__, message)

warnings.warn('This is a warning message, before')
warnings.formatwarning = warning_on_one_line
warnings.warn('This is a warning message, after')

$ python warnings_formatwarning.py
/Users/dhellmann/Documents/PyMOTW/in_progress/warnings/warnings_formatwarning.py:16: UserWarning: This is a warning message, before
  warnings.warn('This is a warning message, before')
/Users/dhellmann/Documents/PyMOTW/in_progress/warnings/warnings_formatwarning.py:18: UserWarning:This is a warning message, after

Stack Level in Warnings:

You'll notice that by default the warning message includes the source line that generated it, when available. It's not all that useful to see the line of code with the actual warning message, though. Instead, you can tell warnings.warn() how far up the stack it has to go to find the line the called the function containing the warning. That way users of a deprecated function see where the function is called, instead of the implementation of the function.

import warnings

def old_function():
    warnings.warn(
        'old_function() is deprecated, use new_function() instead', 
        stacklevel=2)

def caller_of_old_function():
    old_function()
    
caller_of_old_function()

Notice that in this example warnings.warn() needs to go up the stack 2 levels, one for itself and one for old_function().

$ python warnings_warn_stacklevel.py
/Users/dhellmann/Documents/PyMOTW/in_progress/warnings/warnings_warn_stacklevel.py:19: UserWarning: old_function() is deprecated, use new_function() instead
  old_function()

References:

PEP 230 -- Warning Framework
Python Module of the Week Home
Download Sample Code

Technorati Tags:
python, PyMOTW

Static Code Analizers for Python | And Now For Something Completely Different

Old-school developers remember lint, the static code analysis tool for C programs. There are several similar programs available for Python, and they can all help you clean up your act.

This column was originally published by Python Magazine in March of 2008.

Read more

experts and jargon

"real experts use language that the people they’re talking to can understand" - Tim Bray

Via ongoing · Deletionist Morons

How to Write (in a thousand words or less) - Stepcase Lifehack

Dustin Wax gives us 17 concise tips on How to Write (in a thousand words or less) over at lifehack.org. They are a good reminder of some basic dos and don'ts.

code_swarm - Python on Vimeo

code_swarm - Python on Vimeo

This is a really cool visualization of the commit history of the Python core by Michael Ogawa. His site includes movies for other open source projects, too. It's very interesting to see the difference in the results caused by variations in the organizational structure of the projects.


code_swarm - Python from Michael Ogawa on Vimeo.

(Via Barry Warsaw on python-dev.)

CommandLineApp 3.0

Ben Finney provided a patch to CommandLineApp convert the names of the module, method, etc. to be PEP8-compliant. Thanks, Ben!

These changes are obviously backwards incompatible.

Updated: I botched the packaging for 3.0. Release 3.0.1 includes the test script as well.

PyMOTW: platform

Probe the underlying platform's architecture and version information with the platform module.

Module: platform
Purpose: Access system hardware, OS, and interpreter version information.
Python Version: 2.3+

Description:

Although Python is often used as a cross-platform language, it is occasionally necessary to know what sort of system you're running on. Build tools obviously need that information, but you might also know that some libraries or external commands have different interfaces on different operating systems. For example, if you are writing a tool to manage the network configuration of an operating system, you can have a portable representation of network interfaces, aliases, IP addresses, etc. But once you get down to actually editing the configuration files, you need to know more about your host and how it is configured. The platform module gives you some tools for learning about the interpreter, operating system, and hardware platform where your program is running.

Examples:

The example output below was generated on a MacBook Pro running OS X 10.5.2 and a VMware VM running CentOS 4.6. I don't have ready access to Windows, but these functions all work there, too. (If someone wants to run the scripts on Windows and post the output in the comments, I would appreciate it!)

Interpreter:

There are four functions for getting information about the current Python interpreter. python_version() and python_version_tuple() return different forms of the interpreter version with major, minor, and patchlevel components. python_compiler() reports on the compiler used to build the interpreter. And python_build() gives a version string for the build of the interpreter.

import platform

print 'Version      :', platform.python_version()
print 'Version tuple:', platform.python_version_tuple()
print 'Compiler     :', platform.python_compiler()
print 'Build        :', platform.python_build()

OS X:

$ python platform_python.py
Version      : 2.5.1
Version tuple: ['2', '5', '1']
Compiler     : GCC 4.0.1 (Apple Computer, Inc. build 5367)
Build        : ('r251:54869', 'Apr 18 2007 22:08:04')

Linux:

$ python platform_python.py 
Version      : 2.4.4
Version tuple: ['2', '4', '4']
Compiler     : GCC 3.4.6 20060404 (Red Hat 3.4.6-9)
Build        : (1, 'Mar 12 2008 15:09:04')

(It looks like I need to upgrade that system...)

Platform:

A general purpose platform identifier is available via the platform() function. platform() accepts two optional boolean arguments. If aliased is True, the names in the return value are converted from a formal name to their more common form. When terse is true, returns a minimal value with some parts dropped.

import platform

print 'Normal :', platform.platform()
print 'Aliased:', platform.platform(aliased=True)
print 'Terse  :', platform.platform(terse=True)

OS X:

$ python platform_platform.py
Normal : Darwin-9.2.2-i386-32bit
Aliased: Darwin-9.2.2-i386-32bit
Terse  : Darwin-9.2.2

Linux:

$ python platform_platform.py 
Normal : Linux-2.6.9-67.0.4.ELsmp-i686-with-redhat-4.6-Final
Aliased: Linux-2.6.9-67.0.4.ELsmp-i686-with-redhat-4.6-Final
Terse  : Linux-2.6.9-67.0.4.ELsmp-i686-with-glibc2.3

Operating System and Hardware Info:

More detailed information about the operating system and hardware the interpreter is running under can be retrieved as well. uname() returns a tuple containing the system, node, release, version, machine, and processor values. Individual values can be accessed through functions of the same names:

system() returns the operating system name. node() returns the hostname of the server, not fully qualified. release() returns the operating system release number. version() returns the more detailed system version. machine() gives a hardware-type identifier such as 'i386'. processor() returns a real identifier for the processor, or the same value as machine() in many cases.

import platform

print 'uname:', platform.uname()

print
print 'system   :', platform.system()
print 'node     :', platform.node()
print 'release  :', platform.release()
print 'version  :', platform.version()
print 'machine  :', platform.machine()
print 'processor:', platform.processor()

OS X:

$ python platform_os_info.py
uname: ('Darwin', 'farnsworth.local', '9.2.2', 'Darwin Kernel Version 9.2.2: Tue Mar  4 21:17:34 PST 2008; root:xnu-1228.4.31~1/RELEASE_I386', 'i386', 'i386')

system   : Darwin
node     : farnsworth.local
release  : 9.2.2
version  : Darwin Kernel Version 9.2.2: Tue Mar  4 21:17:34 PST 2008; root:xnu-1228.4.31~1/RELEASE_I386
machine  : i386
processor: i386

Linux:

$ python platform_os_info.py 
uname: ('Linux', 'zoidberg', '2.6.9-67.0.4.ELsmp', '#1 SMP Sun Feb 3 07:08:57 EST 2008', 'i686', 'i686')

system   : Linux
node     : zoidberg
release  : 2.6.9-67.0.4.ELsmp
version  : #1 SMP Sun Feb 3 07:08:57 EST 2008
machine  : i686
processor: i686

Executable Architecture:

Individual program architecture information can be probed using the architecture() function. The first argument is the path to an executable program (defaulting to sys.executable, the Python interpreter). The return value is a tuple containing the bit architecture and the linkage format used.

import platform

print 'interpreter:', platform.architecture()
print '/bin/ls    :', platform.architecture('/bin/ls')

OS X:

$ python platform_architecture.py
interpreter: ('32bit', '')
/bin/ls    : ('32bit', '')

Linux:

$ python platform_architecture.py 
interpreter: ('32bit', 'ELF')
/bin/ls    : ('32bit', 'ELF')

References:

Python Module of the Week Home
Download Sample Code

Technorati Tags:
python, PyMOTW

IPython and virtualenv | And Now For Something Completely Different, Feb. 2008

IPython is a feature-rich interactive shell for Python developers. Virtualenv creates isolated development environments so you can test or install packages without introducing conflicts. In this column, I examine how both tools can make your life a little easier.

This column was originally published by Python Magazine in February of 2008.

Read More

looking for good examples of source from large Python projects

I was recently asked for some advice about Python source code that would serve as a good example of organization, style, and documentation (including comments in the code itself). I had a few suggestions for projects to look at, but I wonder what other people think about the question.

Can you point to a project with source that you would hold up as an example of how to set up a medium to large code base? What makes it a good example?

PyMOTW: dircache

The dircache module includes a function for caching directory listings.

Module: dircache
Purpose: Cache directory listings, updating when the modification time of a directory changes.
Python Version: 1.4 and later

Listing Directory Contents:

The main function in the dircache API is listdir(), a wrapper around os.listdir() that caches the results and returns the same list each time it is called with the a path unless the modification date of the named directory changes.

import dircache

path = '.'
first = dircache.listdir(path)
second = dircache.listdir(path)

print 'Contents :', first
print 'Identical:', first is second
print 'Equal    :', first == second

It is important to recognize that the exact same list is returned each time, so it should not be modified in place.

$ python dircache_listdir.py
Contents : ['.svn', '__init__.py', 'dircache_annotate.py', 'dircache_listdir.py', 
'dircache_listdir_file_added.py', 'dircache_reset.py']
Identical: True
Equal    : True

Of course, if the contents of the directory changes it is rescanned.

import dircache
import os

path = '/tmp'
file_to_create