Tuesday, March 24, 2009

virtualenvwrapper updates

The latest version available for download is 1.12. New features include pre and post operation hooks for operations involving virtualenvs (so you can do things like install your favorite common packages every time you make a new virtualenv), navigation functions for working inside the environment, and a host of fixes and optimizations. Check the release log on PyPI for a complete history of the updates.

In the ~2 weeks since I've uploaded virtualenvwrapper to bitbucket, I've received more patches than in all the previous time I maintained the script. Score 1 for DVCS!

[Updated: Changed "bitkeeper" to "bitbucket" - Sorry Jesper!]
Posted by at 8:18 AM 0 comments
Labels: ,

Saturday, March 14, 2009

PyMOTW source now available on BitBucket.org

Earlier this week I uploaded all of the history for PyMOTW to BitBucket.org.

BitBucket uses Mercurial as its DVCS, and after a couple of hiccups with the initial conversion I've had pretty good luck. I think I have the workflow down far managing local changes vs. updates pushed to the central repository.

I still need to work out how (and whether) to use branches. I have a sense that I can use local clones in the way I used to use branches: starting an article that was going to take several sessions, without committing the changes to the trunk. What are the typical practices for hg branching? Do people use them?

Today's PyMOTW post is the first post since the change-over. If you encounter any problems, please report them as an issue via the BitBucket issue tracker.
Posted by at 11:01 AM 4 comments
Labels: , ,

PyMOTW: asynchat

asynchat – Asynchronous protocol handler

Purpose:Asynchronous network communication protocol handler
Python Version:1.5.2 and later

The asynchat module builds on asyncore to make it easier to implement protocols based on passing messages back and forth between server and client. The async_chat class is an asyncore.dispatcher subclass that receives data and looks for a message terminator. Your subclass only needs to specify what to do when data comes in and how to respond once the terminator is found. Outgoing data is queued for transmission via FIFO objects managed by async_chat.

Message Terminators

Incoming messages are broken up based on terminators, controlled for each instance via set_terminator(). There are three possible configurations:

  1. If a string argument is passed to set_terminator(), the message is considered complete when that string appears in the input data.
  2. If a numeric argument is passed, the message is considered complete when that many bytes have been read.
  3. If None is passed, message termination is not managed by async_chat.

The EchoServer example below uses both a simple string terminator and a message length terminator, depending on the context of the incoming data. The HTTP request handler example in the standard library documentation offers another example of how to change the terminator based on the context to differentiate between HTTP headers and the HTTP POST request body.

Server and Handler

To make it easier to understand how asynchat is different from asyncore, the examples here duplicate the functionality of the EchoServer example from the asyncore discussion. The same basic structure is needed: a server object to accept connections, handler objects to deal with communication with each client, and client objects to initiate the conversation.

The EchoServer needed to work with asynchat is basically the same as the one created for the asyncore-based example, with fewer logging calls because they are less interesting this time around:

import asyncore
import logging
import socket

from asynchat_echo_handler import EchoHandler

class EchoServer(asyncore.dispatcher):
    """Receives connections and establishes handlers for each client.
    """
    
    def __init__(self, address):
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.bind(address)
        self.address = self.socket.getsockname()
        self.listen(1)
        return

    def handle_accept(self):
        # Called when a client connects to our socket
        client_info = self.accept()
        EchoHandler(sock=client_info[0])
        # We only want to deal with one client at a time,
        # so close as soon as we set up the handler.
        # Normally you would not do this and the server
        # would run forever or until it received instructions
        # to stop.
        self.handle_close()
        return
    
    def handle_close(self):
        self.close()

The EchoHandler is based on asynchat.async_chat instead of the asyncore.dispatcher this time around. It operates at a slightly higher level of abstraction, so reading and writing are handled automatically. All we need to do is tell the handler:

  • what to do with incoming data (by overriding handle_incoming_data())
  • how to recognize the end of an incoming message (via set_terminator())
  • what to do when a complete message is received (in found_terminator())
  • what data to send (using push())

In the example application, we have 2 operating modes. We are either waiting for a command of the form ECHO length\n, or we are waiting for the data to be echoed. We toggle back and forth between the two modes by setting an instance variable process_data to the method to be invoked when the terminator is found and then changing the terminator as appropriate.

import asynchat
import logging


class EchoHandler(asynchat.async_chat):
    """Handles echoing messages from a single client.
    """
    
    def __init__(self, sock):
        self.received_data = []
        self.logger = logging.getLogger('EchoHandler%s' % str(sock.getsockname()))
        asynchat.async_chat.__init__(self, sock)
        # Start looking for the ECHO command
        self.process_data = self._process_command
        self.set_terminator('\n')
        return

    def collect_incoming_data(self, data):
        """Read an incoming message from the client and put it into our outgoing queue."""
        self.logger.debug('collect_incoming_data() -> (%d)\n"""%s"""', len(data), data)
        self.received_data.append(data)

    def found_terminator(self):
        """The end of a command or message has been seen."""
        self.logger.debug('found_terminator()')
        self.process_data()
    
    def _process_command(self):        
        """We have the full ECHO command"""
        command = ''.join(self.received_data)
        self.logger.debug('_process_command() "%s"', command)
        command_verb, command_arg = command.strip().split(' ')
        expected_data_len = int(command_arg)
        self.set_terminator(expected_data_len)
        self.process_data = self._process_message
        self.received_data = []
    
    def _process_message(self):
        """We have read the entire message to be sent back to the client"""
        to_echo = ''.join(self.received_data)
        self.logger.debug('_process_message() echoing\n"""%s"""', to_echo)
        self.push(to_echo)
        # Disconnect after sending the entire response
        # since we only want to do one thing at a time
        self.close_when_done()

Once the complete command is found, we switch to message-processing mode and wait for the complete set of text to be received. When all of the data is available, we push it onto the outgoing channel and set up the handler to be closed once the data is sent.

Client

The client works in much the same way as the handler. As with the asyncore implementation, the message to be sent is an argument to the client’s constructor. When the socket connection is established, handle_connect() is called so we can send the command and message data.

The command is pushed directly, but a special “producer” class is used for the message text. The producer is polled for data to send out over the network. When the producer returns an empty string, it is assumed to be empty and writing stops.

The client expects just the message data in response, so it sets an integer terminator and collects data in a list until the entire message has been received.

import asynchat
import logging
import socket


class EchoClient(asynchat.async_chat):
    """Sends messages to the server and receives responses.
    """
    
    def __init__(self, host, port, message):
        self.message = message
        self.received_data = []
        self.logger = logging.getLogger('EchoClient')
        asynchat.async_chat.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.logger.debug('connecting to %s', (host, port))
        self.connect((host, port))
        return
        
    def handle_connect(self):
        self.logger.debug('handle_connect()')
        # Send the command
        self.push('ECHO %d\n' % len(self.message))
        # Send the data
        self.push_with_producer(EchoProducer(self.message))
        # We expect the data to come back as-is, 
        # so set a length-based terminator
        self.set_terminator(len(self.message))
    
    def collect_incoming_data(self, data):
        """Read an incoming message from the client and put it into our outgoing queue."""
        self.logger.debug('collect_incoming_data() -> (%d)\n"""%s"""', len(data), data)
        self.received_data.append(data)

    def found_terminator(self):
        self.logger.debug('found_terminator()')
        received_message = ''.join(self.received_data)
        if received_message == self.message:
            self.logger.debug('RECEIVED COPY OF MESSAGE')
        else:
            self.logger.debug('ERROR IN TRANSMISSION')
            self.logger.debug('EXPECTED "%s"', self.message)
            self.logger.debug('RECEIVED "%s"', received_message)
        return

class EchoProducer(asynchat.simple_producer):

    logger = logging.getLogger('EchoProducer')

    def more(self):
        response = asynchat.simple_producer.more(self)
        self.logger.debug('more() -> (%s)\n"""%s"""', len(response), response)
        return response

Putting It All Together

The main program for this example sets up the client and server in the same asyncore main loop.

import asyncore
import logging
import socket

from asynchat_echo_server import EchoServer
from asynchat_echo_client import EchoClient

logging.basicConfig(level=logging.DEBUG,
                    format='%(name)s: %(message)s',
                    )

address = ('localhost', 0) # let the kernel give us a port
server = EchoServer(address)
ip, port = server.address # find out what port we were given

message_data = open('lorem.txt', 'r').read() * 2
client = EchoClient(ip, port, message=message_data)

asyncore.loop()

Normally you would have them in separate processes, of course, but this makes it easier to show the combined output.

$ python asynchat_echo_main.py
EchoClient: connecting to ('127.0.0.1', 49919)
EchoClient: handle_connect()
EchoProducer: more() -> (512)
"""Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra f"""
EchoProducer: more() -> (512)
"""ringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iac"""
EchoHandler('127.0.0.1', 49919): collect_incoming_data() -> (9)
"""ECHO 1474"""
EchoHandler('127.0.0.1', 49919): found_terminator()
EchoHandler('127.0.0.1', 49919): _process_command() "ECHO 1474"
EchoHandler('127.0.0.1', 49919): collect_incoming_data() -> (1024)
"""Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iac"""
EchoProducer: more() -> (450)
"""ulis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"""
EchoHandler('127.0.0.1', 49919): collect_incoming_data() -> (450)
"""ulis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"""
EchoHandler('127.0.0.1', 49919): found_terminator()
EchoHandler('127.0.0.1', 49919): _process_message() echoing
"""Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"""
EchoProducer: more() -> (0)
""""""
EchoClient: collect_incoming_data() -> (512)
"""Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra f"""
EchoClient: collect_incoming_data() -> (512)
"""ringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iac"""
EchoClient: collect_incoming_data() -> (450)
"""ulis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra fringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"""
EchoClient: found_terminator()
EchoClient: RECEIVED COPY OF MESSAGE

See also

asynchat
The standard library documentation for this module.
asyncore
The asyncore module implements an lower-level asynchronous I/O event loop.

PyMOTW Home

Posted by at 10:51 AM 2 comments
Labels: ,

Saturday, March 7, 2009

pictures of the new cichlids

After several years of tame fish like tetras, we've restarted the tank with cichlids. I had both Africans and South Americans many years ago, but we decided to go with Africans this time because we liked the colors and I had the most fun with my Africans before.

IMG_2733.JPG

It took us a while to settle on a theme for their names, but we've finally chosen to go with middle names of past United States Presidents. The red and blue fellow on the left side of this picture is Fitzgerald (Fitz for short). His friend is Delano, who is not only physically larger than most of the others but also provides something of a calming influence on the tank.

IMG_2747.JPG

We have two of these speckled ones, Arnold and Forbes. I'm not sure which this is, and I can't really tell them apart.

IMG_2749.JPG

This blue one is S, the smallest of the bunch and a fairly aggressive character.

IMG_2755.JPG

And last is Knox. He was the most aggressive of the first three we added, but since Delano came along Knox has calmed down a bit. He hid out for a few days under the spider root, but after I rearranged a bit he has started coming out more regularly.

We've had them for a few weeks now, and they are no longer scared of me when I approach the tank. They've learned that an open lid means feeding time (2-3 times a day). And they're starting to splash a bit when they're hungry. Someone (I think Delano) is able to reach the lid when with their tail. The last time I had cichlids they would woke me up in the mornings this way if I slept past feeding time, so it's interesting to see similar behavior in the new batch.

The tank is a 55 gallon with 2 standard backpack filters and sand substrate. The plants are some sort of generic sword left over from the community setup we had before. The cichlids are slowly nibbling away at the stalks, but we have a nice thick undergrowth still. We'll see if they are just thinning it to make moving around easier or if they'll leave anything at all. They don't seem to be consuming what they chop off, as it finds its way into the filter intake and stays there until I clean it up with a net.
Posted by at 4:51 PM 3 comments
Labels:

Thursday, March 5, 2009

virtualenvwrapper moves to bitbucket.org

I've just uploaded release 1.7 of virtualenvwrapper, my set of extensions to Ian Bicking's virtualenv tool. There are no major code changes in this release, just some packaging updates to fix installation via easy_install (forcing unzipped install) and to deal with the move to bitbucket.org.
Posted by at 9:25 AM 0 comments
Labels: ,

Sunday, March 1, 2009

PyMOTW: asyncore

asyncore – Asynchronous I/O handler

Purpose:Asynchronous I/O handler
Python Version:1.5.2 and later

The asyncore module includes tools for working with I/O objects such as sockets so they can be managed asynchronously (instead of, for example, using threads). The main class provided is dispatcher, a wrapper around a socket that provides hooks for handling events like connecting, reading, and writing when invoked from the main loop function, loop().

Clients

To create an asyncore-based client, subclass dispatcher and provide implementations for creating the socket, reading, and writing. Let’s examine this HTTP client, based on the one from the standard library documentation.

import asyncore
import logging
import socket
from cStringIO import StringIO
import urlparse

class HttpClient(asyncore.dispatcher):

    def __init__(self, url):
        self.url = url
        self.logger = logging.getLogger(self.url)
        self.parsed_url = urlparse.urlparse(url)
        asyncore.dispatcher.__init__(self)
        self.write_buffer = 'GET %s HTTP/1.0\r\n\r\n' % self.url
        self.read_buffer = StringIO()
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        address = (self.parsed_url.netloc, 80)
        self.logger.debug('connecting to %s', address)
        self.connect(address)

    def handle_connect(self):
        self.logger.debug('handle_connect()')

    def handle_close(self):
        self.logger.debug('handle_close()')
        self.close()

    def writable(self):
        is_writable = (len(self.write_buffer) > 0)
        if is_writable:
            self.logger.debug('writable() -> %s', is_writable)
        return is_writable
    
    def readable(self):
        self.logger.debug('readable() -> True')
        return True

    def handle_write(self):
        sent = self.send(self.write_buffer)
        self.logger.debug('handle_write() -> "%s"', self.write_buffer[:sent])
        self.write_buffer = self.write_buffer[sent:]

    def handle_read(self):
        data = self.recv(8192)
        self.logger.debug('handle_read() -> %d bytes', len(data))
        self.read_buffer.write(data)

if __name__ == '__main__':
    logging.basicConfig(level=logging.DEBUG,
                        format='%(name)s: %(message)s',
                        )

    clients = [
        HttpClient('http://www.python.org/'),
        HttpClient('http://www.doughellmann.com/PyMOTW/contents.html'),
        ]

    logging.debug('LOOP STARTING')

    asyncore.loop()

    logging.debug('LOOP DONE')

    for c in clients:
        response_body = c.read_buffer.getvalue()
        print c.url, 'got', len(response_body), 'bytes'

First, the socket is created in __init__() using the base class method create_socket(). Alternative implementations of the method may be provided, but in this case we want a TCP/IP socket so the base class version is sufficient.

The handle_connect() hook is present simply to illustrate when it is called. Other types of clients that need to do some sort of hand-shaking or protocol negotiation should do the work in handle_connect().

handle_close() is similarly presented for the purposes of showing when the method is called. The base class version closes the socket correctly, so if you don’t need to do extra cleanup on close you can leave the method out.

The asyncore loop uses writable() and its sibling method readable() to decide what actions to take with each dispatcher. Actual use of poll() or select() on the sockets or file descriptors managed by each dispatcher is handled inside the asyncore code, so you don’t need to do that yourself. Simply indicate whether the dispatcher cares at all about reading or writing. In the case of this HTTP client, writable() returns True as long as there is data from the request to send to the server. readable() always returns True because we want to read all of the data.

Each time through the loop when writable() responds positively, handle_write() is invoked. In this version, the HTTP request string that was built in __init__() is sent to the server and the write buffer is reduced by the amount successfully sent.

Similarly, when readable() responds positively and there is data to read, handle_read() is invoked.

The example below the __main__ test configures logging for debugging then creates two clients to download two separate web pages. Creating the clients registers them in a “map” kept internally by asyncore. The downloading occurs as the loop iterates over the clients. When the client reads 0 bytes from a socket that seems readable, the condition is interpreted as a closed connection and handle_close() is called.

One example of how this client app may run is:

$ python asyncore_http_client.py
http://www.python.org/: connecting to ('www.python.org', 80)
http://www.doughellmann.com/PyMOTW/contents.html: connecting to ('www.doughellmann.com', 80)
root: LOOP STARTING
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_connect()
http://www.doughellmann.com/PyMOTW/contents.html: handle_write() -> "GET http://www.doughellmann.com/PyMOTW/contents.html HTTP/1.0

"
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 3163 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 1448 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 2896 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 2896 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 2896 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 2896 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 1448 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 1448 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 1448 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 1448 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 895 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_close()
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 0 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.python.org/: handle_connect()
http://www.python.org/: handle_write() -> "GET http://www.python.org/ HTTP/1.0

"
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1257 bytes
http://www.python.org/: readable() -> True
http://www.python.org/: handle_close()
http://www.python.org/: handle_read() -> 0 bytes
root: LOOP DONE
http://www.python.org/ got 18009 bytes
http://www.doughellmann.com/PyMOTW/contents.html got 22882 bytes

Servers

The example below illustrates using asyncore on the server by re-implementing the EchoServer from the SocketServer examples. There are three classes: EchoServer receives incoming connections from clients and creates EchoHandler instances to deal with each. The EchoClient is an asyncore dispatcher similar to the HttpClient defined above.

import asyncore
import logging

class EchoServer(asyncore.dispatcher):
    """Receives connections and establishes handlers for each client.
    """
    
    def __init__(self, address):
        self.logger = logging.getLogger('EchoServer')
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.bind(address)
        self.address = self.socket.getsockname()
        self.logger.debug('binding to %s', self.address)
        self.listen(1)
        return

    def handle_accept(self):
        # Called when a client connects to our socket
        client_info = self.accept()
        self.logger.debug('handle_accept() -> %s', client_info[1])
        EchoHandler(sock=client_info[0])
        # We only want to deal with one client at a time,
        # so close as soon as we set up the handler.
        # Normally you would not do this and the server
        # would run forever or until it received instructions
        # to stop.
        self.handle_close()
        return
    
    def handle_close(self):
        self.logger.debug('handle_close()')
        self.close()
        return

class EchoHandler(asyncore.dispatcher):
    """Handles echoing messages from a single client.
    """
    
    def __init__(self, sock, chunk_size=256):
        self.chunk_size = chunk_size
        self.logger = logging.getLogger('EchoHandler%s' % str(sock.getsockname()))
        asyncore.dispatcher.__init__(self, sock=sock)
        self.data_to_write = []
        return
    
    def writable(self):
        """We want to write if we have received data."""
        response = bool(self.data_to_write)
        self.logger.debug('writable() -> %s', response)
        return response
    
    def handle_write(self):
        """Write as much as possible of the most recent message we have received."""
        data = self.data_to_write.pop()
        sent = self.send(data[:self.chunk_size])
        if sent < len(data):
            remaining = data[sent:]
            self.data.to_write.append(remaining)
        self.logger.debug('handle_write() -> (%d) "%s"', sent, data[:sent])
        if not self.writable():
            self.handle_close()

    def handle_read(self):
        """Read an incoming message from the client and put it into our outgoing queue."""
        data = self.recv(self.chunk_size)
        self.logger.debug('handle_read() -> (%d) "%s"', len(data), data)
        self.data_to_write.insert(0, data)
    
    def handle_close(self):
        self.logger.debug('handle_close()')
        self.close()


class EchoClient(asyncore.dispatcher):
    """Sends messages to the server and receives responses.
    """
    
    def __init__(self, host, port, message, chunk_size=512):
        self.message = message
        self.to_send = message
        self.received_data = []
        self.chunk_size = chunk_size
        self.logger = logging.getLogger('EchoClient')
        asyncore.dispatcher.__init__(self)
        self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
        self.logger.debug('connecting to %s', (host, port))
        self.connect((host, port))
        return
        
    def handle_connect(self):
        self.logger.debug('handle_connect()')
    
    def handle_close(self):
        self.logger.debug('handle_close()')
        self.close()
        received_message = ''.join(self.received_data)
        if received_message == self.message:
            self.logger.debug('RECEIVED COPY OF MESSAGE')
        else:
            self.logger.debug('ERROR IN TRANSMISSION')
            self.logger.debug('EXPECTED "%s"', self.message)
            self.logger.debug('RECEIVED "%s"', received_message)
        return
    
    def writable(self):
        self.logger.debug('writable() -> %s', bool(self.to_send))
        return bool(self.to_send)

    def handle_write(self):
        sent = self.send(self.to_send[:self.chunk_size])
        self.logger.debug('handle_write() -> (%d) "%s"', sent, self.to_send[:sent])
        self.to_send = self.to_send[sent:]

    def handle_read(self):
        data = self.recv(self.chunk_size)
        self.logger.debug('handle_read() -> (%d) "%s"', len(data), data)
        self.received_data.append(data)
        

if __name__ == '__main__':
    import socket
    import threading

    logging.basicConfig(level=logging.DEBUG,
                        format='%(name)s: %(message)s',
                        )

    address = ('localhost', 0) # let the kernel give us a port
    server = EchoServer(address)
    ip, port = server.address # find out what port we were given

    client = EchoClient(ip, port, message=open('lorem.txt', 'r').read())

    asyncore.loop()

The EchoServer and EchoHandler are defined in separate classes because they do different things. When EchoServer accepts a connection, a new socket is established. Rather than try to dispatch to individual clients within EchoServer, an EchoHandler is created to take advantage of the socket map maintained by asyncore.

$ python asyncore_echo_server.py
EchoServer: binding to ('127.0.0.1', 52235)
EchoClient: connecting to ('127.0.0.1', 52235)
EchoClient: writable() -> True
EchoServer: handle_accept() -> ('127.0.0.1', 52236)
EchoServer: handle_close()
EchoClient: handle_connect()
EchoClient: handle_write() -> (512) "Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque vel arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra f"
EchoClient: writable() -> True
EchoHandler('127.0.0.1', 52235): writable() -> False
EchoHandler('127.0.0.1', 52235): handle_read() -> (256) "Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque ve"
EchoClient: handle_write() -> (225) "ringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"
EchoClient: writable() -> False
EchoHandler('127.0.0.1', 52235): writable() -> True
EchoHandler('127.0.0.1', 52235): handle_read() -> (256) "l arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra f"
EchoHandler('127.0.0.1', 52235): handle_write() -> (256) "Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque ve"
EchoHandler('127.0.0.1', 52235): writable() -> True
EchoClient: writable() -> False
EchoHandler('127.0.0.1', 52235): writable() -> True
EchoClient: handle_read() -> (256) "Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque ve"
EchoHandler('127.0.0.1', 52235): handle_read() -> (225) "ringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"
EchoHandler('127.0.0.1', 52235): handle_write() -> (256) "l arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra f"
EchoHandler('127.0.0.1', 52235): writable() -> True
EchoClient: writable() -> False
EchoHandler('127.0.0.1', 52235): writable() -> True
EchoClient: handle_read() -> (256) "l arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra f"
EchoHandler('127.0.0.1', 52235): handle_write() -> (225) "ringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"
EchoHandler('127.0.0.1', 52235): writable() -> False
EchoHandler('127.0.0.1', 52235): handle_close()
EchoClient: writable() -> False
EchoClient: handle_read() -> (225) "ringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"
EchoClient: writable() -> False
EchoClient: handle_close()
EchoClient: RECEIVED COPY OF MESSAGE
EchoClient: handle_read() -> (0) ""

In this example the server, handler, and client objects are all being maintained in the same socket map by asyncore in a single process. To separate the server from the client, simply instantiate them from separate scripts and run asyncore.loop() in both. When a dispatcher is closed, it is removed from the map maintained by asyncore and the loop exits when the map is empty.


Working with Other Event Loops

It is sometimes necessary to integrate the asyncore event loop with an event loop from the parent application. For example, a GUI application would not want the UI to block until all asynchronous transfers are handled – that would defeat the purpose of making them asynchronous. To make this sort of integration easy, asyncore.loop() accepts arguments to set a timeout and to limit the number of times the loop is run. We can see their effect on the client by re-using HttpClient from the first example.

import asyncore
import logging

from asyncore_http_client import HttpClient

logging.basicConfig(level=logging.DEBUG,
                    format='%(name)s: %(message)s',
                    )

clients = [
    HttpClient('http://www.doughellmann.com/PyMOTW/contents.html'),
    HttpClient('http://www.python.org/'),
    ]

loop_counter = 0
while asyncore.socket_map:
    loop_counter += 1
    logging.debug('loop_counter=%s', loop_counter)
    asyncore.loop(timeout=1, count=1)

Here we see that the client is only asked to read or data once per call into asyncore.loop(). Instead of our own while loop, we could call asyncore.loop() like this from a GUI toolkit idle handler or other mechanism for doing a small amount of work when the UI is not busy with other event handlers.

$ python asyncore_loop.py
http://www.doughellmann.com/PyMOTW/contents.html: connecting to ('www.doughellmann.com', 80)
http://www.python.org/: connecting to ('www.python.org', 80)
root: loop_counter=1
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: writable() -> True
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_connect()
http://www.doughellmann.com/PyMOTW/contents.html: handle_write() -> "GET http://www.doughellmann.com/PyMOTW/contents.html HTTP/1.0

"
root: loop_counter=2
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 267 bytes
root: loop_counter=3
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 8192 bytes
root: loop_counter=4
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 6288 bytes
root: loop_counter=5
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 1448 bytes
root: loop_counter=6
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 1448 bytes
root: loop_counter=7
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 2896 bytes
root: loop_counter=8
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 2343 bytes
root: loop_counter=9
http://www.doughellmann.com/PyMOTW/contents.html: readable() -> True
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.doughellmann.com/PyMOTW/contents.html: handle_close()
http://www.doughellmann.com/PyMOTW/contents.html: handle_read() -> 0 bytes
root: loop_counter=10
http://www.python.org/: readable() -> True
http://www.python.org/: writable() -> True
http://www.python.org/: handle_connect()
http://www.python.org/: handle_write() -> "GET http://www.python.org/ HTTP/1.0

"
root: loop_counter=11
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=12
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=13
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=14
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=15
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=16
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=17
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=18
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=19
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=20
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=21
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=22
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1396 bytes
root: loop_counter=23
http://www.python.org/: readable() -> True
http://www.python.org/: handle_read() -> 1257 bytes
root: loop_counter=24
http://www.python.org/: readable() -> True
http://www.python.org/: handle_close()
http://www.python.org/: handle_read() -> 0 bytes

Working with Files

Normally you would want to use asyncore with sockets, but there are times when it is useful to read files asynchronously, too (to use files when testing network servers without requiring the network setup, or to read or write large data files in parts). For these situations, asyncore provides the file_dispatcher and file_wrapper classes.

import asyncore
import os

class FileReader(asyncore.file_dispatcher):
    
    def writable(self):
        return False
    
    def handle_read(self):
        data = self.recv(256)
        print 'READ: (%d) "%s"' % (len(data), data)
        
    def handle_expt(self):
        # Ignore events that look like out of band data
        pass
    
    def handle_close(self):
        self.close()

lorem_fd = os.open('lorem.txt', os.O_RDONLY)
reader = FileReader(lorem_fd)
asyncore.loop()

This example was tested under Python 2.5.2, so I am using os.open() to get a file descriptor for the file. For Python 2.6 and later, file_dispatcher automatically converts anything with a fileno() method to a file descriptor.

$ python asyncore_file_dispatcher.py
READ: (256) "Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Donec
egestas, enim et consectetuer ullamcorper, lectus ligula rutrum leo, a
elementum elit tortor eu quam. Duis tincidunt nisi ut ante. Nulla
facilisi. Sed tristique eros eu libero. Pellentesque ve"
READ: (256) "l arcu. Vivamus
purus orci, iaculis ac, suscipit sit amet, pulvinar eu,
lacus. Praesent placerat tortor sed nisl. Nunc blandit diam egestas
dui. Pellentesque habitant morbi tristique senectus et netus et
malesuada fames ac turpis egestas. Aliquam viverra f"
READ: (225) "ringilla
leo. Nulla feugiat augue eleifend nulla. Vivamus mauris. Vivamus sed
mauris in nibh placerat egestas. Suspendisse potenti. Mauris massa. Ut
eget velit auctor tortor blandit sollicitudin. Suspendisse imperdiet
justo.
"
READ: (0) ""

See also

asyncore
The standard library documentation for this module.
asynchat
The asynchat module builds on asyncore to make it easier to create clients
and servers communicate by passing messages back and forth using a set protocol.
SocketServer
The SocketServer module article includes another example of the EchoServer with
threading and forking variants.

PyMOTW Home

Posted by at 4:36 PM 1 comments
Labels: ,
Subscribe to: Posts (Atom)