Python development tools you can't live without
I'm working on a series of columns for Python Magazine in which I will be talking about development tools. The first "episode" appears in the January 2008 issue and covers testing tools like frameworks and runners. I have several more columns plotted out, but want to make sure I cover the topics well and don't miss out on mentioning a new or small tool just because I haven't heard about it myself.
So, I'm looking for feedback about the tools you just can't (or don't want to) live without. Tell me all about your development environment, editor, shell, debugger, IDE, version control system, etc. What libraries (in addition to the standard library) do you use on a regular basis? Is there any one thing that you would identify as being so necessary for your Python work that if it was taken away you'd have to give up and use another language? (And "the interpreter" is already on the list, thanks.)
If you're building a new development tool you think I should look at for the series, post a link in a comment here or tag it with pymagdifferent on del.icio.us and I'll review it. I can't promise that everything will make it into the columns right away, but it will eventually. Unusual or unique entries are more likely to be covered sooner.
It seems like everyone and their brother is building an editor, so if that's your game at least tell me why you think yours is better/different from all the others. There are some problems with text editing that are not easily solved, so if you have a killer feature make sure you point that out. That goes double for templating languages -- highlight your special strengths.
55 comments:
Although you say "the interpreter" is on the list, let me just suggest IPython over the vanilla interpreter if you're talking about the must have tools ...
vim as the editor + plugins (for example, taglist, python.vim)
I surely back the IPython suggestion. It is not only an interpreter, it's almost a complete development environment. With IPython and Your Favorite Editor, you'll get a long way.
There are a lot of well executed IDEs for python: Komodo, eric and wing to name but a few. But in my eyes, python is one of those rare languages where you really don't _need_ all the features of an IDE - you just need a snappy editor (YFE), IPython and pdb.
It's not that I'm a purist - I prefer IDEs for languages such as java or c# - python is just so damn intuitive!
Well, in regards to interactive, environment, I like IPython. On windows, Pythonwin was enough for me as an editor, and I often used its interactive shell there as well. (I especially liked the completion filling in using names from the interactive shell.)
On Linux, I use Eric. I don't really need most of the fancy features (of any IDE for that matter).
Regarding libraries:
Well, on windows installing pywin32 is a must. PIL for imaging. I liked pymedia for sound. pygame and pyopengl for interactive graphical environments. All of the numarray\numeric\numpy family of modules are also very useful.
Regarding version control, it's SVN whenever I can.
As mentioned, IPython is an absolute necessity (readline controls, filename completion, shortcuts like ?, etc.)
PDB is an essential skill for python dev. I admit to going for "print kilroy" style debugging at first, but anything difficult to track down needs a quick PDB invocation.
Emacs + Flymake + pylint let's me see color coded lines in my editor corresponding to syntax errors and stylistic warnings. Python itself won't complain if you overwrite built-ins (str, len, etc) but pylint will flag them for me. Also worth running is PEP8.py to check for PEP8 style compliance.
Finally, I think it's hard to over-praise doctests. I've even had success getting anti-unittesting people to write doctests (it's just checking that your documentation is correct!)
IPython is as crucial.
I love the latest WingIDE. It's debugging features are the best I have seen (and I have tried all the python debuggers). The auto completion is the best I have seen as well. Where other systems get it wrong or cant figure it out, Wing gets it right.
In the past I only broke out Wing for Zope/Plone issues and used Komodo and SPE; but that is no longer the case. The remote debugging is much better than komodo's and is invaluable for developing grid distributed programs. I used to use the print statement or just pdb, but those options are actually more work. For django development, I would just be lost without it.
The Active State Komodo introspective completion feels a little better/faster, but is very often missing, or just plain wrong.
Komodo is faster on Win32, but is plagued by lockups when using svn+ssh (at least on my 4 machines). I do not see the speed issues on linux or Mac.
Vim, IPython and the pylab/scipy combination are the keys to me getting *any* work done
I'll also throw in a vote for IPython, though I would not say I could not live without it. It's just nicer. :)
As for editor, I agree with nonetype. I like SPE, but I don't mind Kate in KDE, and Notepad++ on Windows. I usually just use a text editor over an ide.
For libraries, setuptools in indispensable. Also top on my list are Beautiful Soup, configobj, SQLAlchemy, docutils and pretty much anything written by Ian Bicking and Philip J. Eby. :) Oh and TurboGears! :D
I'm also a big fan of nose, pylint and epydoc (the call graphs and neat!). I usually use all 3 on any significant project.
Lately I've been enjoying the Plugins package as well as pyparsing and PyProtocols.
Hope this helps!
(Oh and this comment was posted from my new OLPC CO laptop.) :)
EMACS is my editor of choice, but as far as totally indispensable tools go, I'm gonna have to cast another vote for IPython. Couldn't live without it.
Sorry to write a 'me-too' entry but gvim/vim; that is plain vanilla gvim, I find to be indispensable in so many ways. It doesn't matter what the language is, or what the data is vim is usually available for the machine, and is my editor of choice. There are vim add-ins to make programming in Python easier but I find gvim + idle/pythonwin/python works just fine for me and leaves me independant of too many plug-ins or gui's that may not work in different environments.
- Paddy.
I haven't used it long enough to say 'can't live w/out' -- but, I'm really liking the pydev (http://pydev.sourceforge.net/) plugin for eclipse. And I use it.
-reed
+1 ipython
+1 (g)vim, compiled with all the bells and whistles enabled.
Also exuberant ctags to use within vim.
I also use zsh for my shell, but mainly for the ability to use "**/*.py" filename expansion to search through subdirectories instead of having to use find.
vim. ctags. id-utils. A version control system (e.g. Subversion or Bazaar). A Linux-based OS (e.g. Ubuntu).
There is a surprisingly high % of vim users out there. Is that a Linux thing? I mean, I know and love vi, too, but is vim really "all that"? I used vi for many years, then xemacs, but lately I've been looking for something a little more modern. I'm working with TextMate right now, and loving the integration with OS X and the easy extension hooks they have in their bundles.
I still use vi and xemacs every day at work, but I think TextMate is going to become my editor for doing my own stuff. Once I'm more comfortable with it, I'll see if it scales up to handle projects the size of what we have at Racemi.
IPython is another tool I'm going to have to check out. Noah and Jeremy have both strongly suggested that I check it out, and I know it will feature prominently in the book they're working on for O'Reilly, but I'm not sure I see the big deal. Maybe it's because I don't usually work interactively with the interpreter. I usually write tests (formal or informal) and then run the script. That way as I change code I can just re-run and not mess around with reloading. So tell me, what am I missing? What's the big deal with IPython?
@lorg: Not being a Windows user any more, I can't comment on PythonWin. I used it ages ago, and remember it being a fairly simplistic editor with an interpreter embedded. Has it expanded/grown over the last few years?
@metapundit.net: Flymake looks pretty useful, I'll have to see about setting that up with xemacs on my Linux dev box. Thanks for the tip!
As far as IDEs go, the killer, absolutely-can't-live-without, feature in my mind is "jump to definition". That's actually one area TextMate is lacking. AFAICT, there isn't a way to jump to the definition of a symbol if it is in another file. Please correct me if you know differently.
Using ctags & vim or etags & emacs hasn't been beaten for reliability. Do the project-based IDEs like PyDev, Komodo, Wing, etc. have good support for this feature? I know many of them seem to concentrate on code-completion, which isn't as a big of a sale for me.
I agree with the general opinion that, though useful, IDEs are not essential for a satisfactory Python programming experience. I use the excellente Scite editor myself, which maybe is the simplest editor there is. I also use Winpdb when debugging seems to be the only way to find a problem.
--Omar
Several items - database connectors, whether its for postgres, oracle or a win32 database via odbc. The other must is the pywin32 default extensions when running in Windows.
Editing - vim, on the unix/linux side. Any text editor that works better than notepad on Windows.
Program Management - LEO. Anything above 40/50 lines goes into LEO. Once the code base goes beyond what I can keep in my head, it allows for me to segment out the code to manageable chunks. Plus, it is easier to focus the documentation at the appropriate details.
SVN for version control, looking at Mercurial or Bazaar for future version control.
gvim plus minibufferexplorer, the stock interpreter, and svn for "my"etc and "my"bin, bunch of scripts and configuration files that I want to keep the same on all accounts / platforms.
And right now, I think it's hard to beat subversion for real multiplatform support (old Unices), a C compiler and some pain pulling the deps and it's good. Cant tell the same thing for git, even if I'm trying to move to it on Linux and OSX. OK hg might be working too.
I'll go with ipython as critical. For writing code, I am still looking for a 'favorite' editor or IDE, so I am glad there have been some suggestions in these responses.
For libraries, PIL and numpy are probably the 3rd party libs I use most.
I have had a good deal of experience introducing people to python. I found that the use of interactive interpreters such IDLE and ipython often leads them to exceedingly bad habits.
For example I have had a student keep years of work, thousands of python files in a single directory (from where IDLE could load them), others getting glued to the up-arrow, key wasting most of their time hunting down the series of previous commands to re-execute a series of steps etc...
Finally I became convinced that these tools do more harm than good. Nowadays I strongly discourage their use, when they want to show me something in IDLE or ipython I send them away and tell them to come back with a working script that they demonstrate with python.
All one really needs is a simple way to execute code as it is being written. So instead of asking them to learn IDLE or the python prompt etc I ask them to get a half decent editor, and learn how to set it up such that it can execute a command (for example python) on the code they are currently editing.
"Give a man IPython skills and you feed him for a day. Teach a man to use an editor and you feed him for a lifetime."
I use ipython, gvim, and mercurial for version control. I've only been doing significant Python development for the last couple of months, although I first learned the language years ago. I plan to add pdb, pylint, and doctest to my arsenal. Thanks to everyone for sharing their experiences. (Even though I work on Windows XP now, I use gvim because I've been a Unix and vi lover since 1984.)
Doug:
Regarding Pythonwin, no I don't think so. Most changes are really minor.
I just liked it because it got the job done. (With good indentation! :)
Longabow:
Well, I still use IPython, but anything more than 4 lines go to a script file. I usually have a directory for my 'generic experimental' scripts, and a directory for projects, both in the pythonpath.
So I think that good habits can be cultivated, while still retaining good use of IPython and such.
eclipse and pydev, especially the "jump to definition" with Ctrl-leftclick is handy. Plus it works on all platforms.
And of course pyflakes, which is a much easier pylint.
iptyhon and
PyScripter IDE(windows only but also said to be running on Linux using wine) - includes excellent code completion, jump to definition etc.
Editor: SPE
Modules: Recently discovered ceODBC and love it for connecting to databases.
Also recently started using SVN/Tortoise SVN on Windows at work and SVN/ZigVersion at home on my Mac. How I've went this long without SVN, I'll never know.
Hey Doug,
For me it's textmate + pylint/pychecker. Also, I use PythonTidy a bit. When I'm working on bigger projects, I switch over to Eclipse+Pydev.
Some of the comments on my Analyzer post here:
http://jessenoller.com/2008/01/16/do-you-use-an-python-code-analyzer/
Might be useful for you too.
Oh, and virtualenv
editors: jedit (heavily tricked-out, primary), vim (aux)
libs: sqlalchemy, pil, docutils, cx_oracle, sqlite, setuptools (as a consumer)
apps: nosetest, epydoc, putty, bzr, hg, ctags
I've been using Komodo / Komodo Edit for initial design/alpha dev. I started off on windows and now dev on linux mainly and some windows so I like cross platform tools very much. Lately, I've been giving SPE a run and it is not bad.
For refactoring I use SciTE and pylint. light and fast :)
ipython for exploratory
nose for testing --all the time.
google and python.org for docs for those modules I use sporadically.
most used modules, MySQLdb and pymssql,
decimal, datetime
query browsers for different db's
subversion.
Hi Longabow,
With your experience of teaching Python, I was wondering what you think of Crunchy. I would so love for it to take off and become next years must have utility for self training.
- Paddy.
I daily use pylint and pydev (paid version) with Eclipse and love both tools.
Editors: PyScripter, Eclipse with PyDev
Libs: Stackless, Win32, cx_Oracle, pyparsing, PyYAML
zc.buildout for managing Python packages, for sure. It's really invaluable to be able to share a development environment configuration with other developers in open source. It makes it really easy to set-up the same set of dependencies and configuration without having to follow through a lengthy INSTALL.txt, a touchy make file or edit and run a clunky bash script.
Doug Hellman asked "There is a surprisingly high % of vim users out there. Is that a Linux thing? I mean, I know and love vi, too, but is vim really "all that"?"
I can't speak for anyone else, but I came to Vim as a long-time Windows user and had never used vi in my life. I hated it for a couple of weeks, now I hate using anything else.
On Windows (when I must), I enjoyed SPE, though now I mostly use (and enjoy) Komodo.
On my real operating system (Linux), though, I often just use vim. It's highly efficient, especially for repetitive changes, and is lightweight enough not to get in my way.
I prefer PyFlakes to pylint and PyChecker because it does not import the code. Some of my code (for Django) requires external setup before it can be imported. Some of my code has side effects when imported. So PyFlakes works better for me. It integrates seamlessly into TextMate.
I use IPython for just a few features:
- persistent, smart history - in a new session I can type "from b" and scroll up to recover my previous imports from blogcosm
- auto-indent
- tab completion
- output history - recover the result of any command
- editor support - edit previous commands in an external editor
- smart paste
- verbose tracebacks
- integration with help
OK that is more features than I thought :-) They are small conveniences but they add up. The standard interpreter seems crippled in comparison.
+1 pydev
Eclipse is an invaluable tool for people like me who have to jump between several different languages.
PyScripter on windows, TextMate on mac. PyScripter is great as an IDE (very lean and fast, jump to definition, debugging, etc), but TextMate is a better editor.
iPython is a must have for me because I spend too much time in the REPL and it has some great features there. Especially when I'm working with matplotlib, numpy/scipy. It also works great as a shell, and I find myself switching to a bash shell far less often after discovering some of the neat shell features.
I'm curious about the large number of vim votes. Is this limited to vim itself (which I like, but don't use as much anymore) or are there python related plugins that push it over the top? If so, I'd love to see a list of must-have plugins.
Having gvim and ipython is pretty much all that i need on windows or linux.
I've started using flymake with Emacs and it's pretty great: http://www.plope.com/Members/chrism/flymake-mode
I'm surprised no one seems to have mentioned nose or py.test. I also use buildutils for a few extensions to setup.py. python setup.py develop isn't a tool, but it's something not everyone knows about (or the trick of "python -c 'import setuptools; execfile("setup.py")' develop"). Some other Setuptools stuff like setup.cfg and tag_svn_revision.
And there's also doctest, ideally with some nose integration (nose doesn't test doctests by default). There's a lot about writing doctests, but not much material about how to easily run those tests.
@everyone - Thanks for the great feedback so far! I've learned about several new tools, and I hope you have all found out about at least one or two you weren't already using.
@omar - Does Winpdb offer features not available in the standard pdb, or is it just a GUI wrapper for it?
@bnf - "better than notepad" isn't hard. :-) LEO sounds intriguing, I'll have to give it a look.
@bimone - What is minibufferexplorer?
The list of "required" libraries is pretty diverse. I guess I shouldn't be surprised.
@longabow - I understand the issues you've encountered. I had a prof (about 15 years ago) who never moved anything out of his inbox because he didn't know how to get to other mailboxes once the messages had been moved. The mode of working you describe is pretty much what I do myself. From what I understand, IPython has some features for saving the current "session" to a script, but I haven't experimented with it enough myself to say for sure.
@Jesse - Several people have mentioned pylint/pychecker. I haven't used a lint tool in a long time, and never with Python. How do they stand up as far as false positives?
PyFlakes sounds popular and useful. I'll have to give that a look, too.
@jeff h - What sort of refactoring aids does SciTE have?
@wheat - Brandon Rhodes gave a good presentation on buildout at a recent PyATL meeting. They recorded it, but I don't know if the video has made it online, yet. It looks useful, but I'm concerned about the dependency of remote repositories. At work, we tend to download packages and create a common local copy with common required software to eliminate that issue. I guess it's a home-grown sort of buildout, since I don't think it existed when we started out.
@dave - "VI is a great place to live, but I wouldn't want to visit there."
@kent - If PyFlakes integrates with TextMate, that's a big plus for it in my book. Is there a special bundle for it? I'm still new to TextMate, but liking it more every day.
@Ian - I'm surprised to see so many votes for linters and so few for testing tools, too. I usually write some tests, some code, and run the tests. I guess that's why a highly interactive operating mode like IPython implies isn't as appealing for me.
Disclaimer: I'm obviously biased, given how I am the original ipython author :)
A couple of comments regarding ipython though.
@longabow: you are certainly free to use whichever tool best fits your working style, but perhaps some notes regarding ipython use may let you find it to be a net gain rather than a hindrance.
IPython is not mean to be an IDE, nor to replace a real editor and writing full-fledged code. My life is spent between an Emacs process on one side of the screen and a terminal with ipython on the other. I edit the code in 'real' scripts in Emacs, and then run it in ipython via
%run myscript.py args...
This is akin to just running it at the system shell as
python myscript.py args...
except that you get much faster cycling if modules have long import times (very important for me), much better exception tracebacks, the ability to fire up the debugger after an exception just by typing %debug, etc. Furthermore, all variables defined in the script are now in the interactive namespace, where they can be further explored, manipulated, etc. This type of workflow happens to be very, very valuable in my field (scientific computing) where I often define entities that I then need to plot, print, compute with, etc.
In summary, all 'real code' lives always in scripts, and the interactive session is the flexible experimentation playground.
Keep in mind that if you ever end up writing in ipython a bit of 'real code', you just type
%hist -n
to get your history printed without line numbers, and you can just copy and paste it back into an editor.
I'm not trying to change your mind, just to offer a perspective on how I, and many others, find ipython to be a very useful tool that *complements, not replaces* a real editor in everyday computing.
@doug: We also use ipython heavily in interactive test flow mode, including doctest development.
In the most recent/svn ipython you can do
%doctest_mode
and ipython flips its prompts to '>>>' ones so you can develop your doctests in a real, running session to then paste them into your code. Furthermore, you can actually paste real doctests *input* (without the output) at the ipython prompt, and it will strip all the leading '>>>' and execute it correctly. This lets you do interactive development of doctests conveniently.
Finally, you can also do (feel free to put this into your init file):
from IPython.dtutils import *
and then type
idoctest()
This function will then let you paste entire doctests (input *and* outptut) at the terminal, and will run them for you with the real doctest runner.
For further questions feel free to stop by the list, but I hope the above will be useful to some.
another +1 for (g)vim
as well as ipython:
what you're missing is:
using your editor during interactive sessions
deep reloads, which allow to reload all your code after editing
the cool syntax coloring, indentation, and ipython magic
as well as built-in shell capabilities
all in addition to the usual benefits of the python interpreter's introspection
I also use pychecker or pylint religiously (though I prefer pychecker, plint is too critical of my "breaking the rules").
And, of course, using svn w/ post commit scripts to compile and run unit tests w/ pyunit
Hi Doug. I'll vote +1 IPython, +1 Wing (on Win and Ubuntu), ssh, svn, rapidSVN (Ubuntu) and TortoiseSVN (Windows), WinSCP along with nosetests, twill (for ShowMeDo web testing), matplotlib, numpy, PIL (for AI contract work).
Want to see screencasts on these topics? We're actively looking for suggestions on which Python tools to screencast in ShowMeDo. Pop into our Group and let us know what you'd like to see demonstrated and we'll do our best (http://groups.google.co.uk/group/showmedo).
Doug - if linking to our IPython, Wing, SPE, svn, PyDev, WinPDB, Crunchy will help your article, please feel very free. All those videos are free to all and are created by members of the Python community.
@fernando - Ok, finally someone explains IPython in a way that makes it sound like it's worth spending time on! I'll definitely check it out.
@ian - Thanks for the pointer, I'll check out the videos.
@Everyone - You've all been very helpful, so I just want to take a quick second to say "Thanks!" and that I really appreciate the feedback and pointers.
I won't live without (g)vim, pydb, ipython like a lot of people here.
But I also like "Bicycle Repair Man"[1] a lot, because it integrates very nicely into vim and is very useful for refactoring stuff like renaming variables/functions.
Another great tool for testing regular expressions is gPyreTest[2]. I use it nearly all the time when I have to do more complicated regular expressions. It has a nice visual representation of matching parts of a regexp.
[1] http://bicyclerepair.sourceforge.net/
[2] http://lucumr.pocoo.org/projects/gpyretest/
Though it is not only for Python projects, I too would like to recommend Mercurial as the revision control system. I have found that I use it much more actively than I have ever used Subversion (or CVS) because it is incredible flexible. That is is written in Python only makes it better.
Apart from that I use Emacs for my Python coding, and I'm currently using Twisted for asynchronous programming. Twisted is really nice once you get it. The unit testing framework that comes with Twisted, Trial, can also be recommended since it makes it relatively easy to test my asynchronous code.
DrPython is another nice IDE although I suspect it is targeted towards those starting out on Python since it seems to be modeled on DrScheme for Scheme which is meant to be more of a learning tool.
@cslife - DrPython looks fairly typical. Does it have any outstanding features that would make you recommend it?
For me, any decent editor with project management feature and pyflakes integration will do.
Once Kate was my preferred choice, but is no longer since KDE developers dropped project management out of Kate saying KDevelop should be used instead.
Now I use PIDA and OpenKomodo for most of my code editing.
Post a Comment