Personal tools
You are here: Home Articles Oct 2004 Pipelines Overview
Document Actions

Pipelines Overview

by paul last modified 2006-01-20 19:06

Some of the goals and issues for the pipeline stuff I've been working on.


The pipeline model I've been working on allows presentation work (templating) to be done outside Zope with off-the-shelf tools, then seamlessly integrated back into Zope.  My current work is heavily tilted towards XML/XSLT-based pipelines, but other technologies can participate as well.  This page captures some of the goals of this approach.


  1. Make a radical improvement in the productivity and learnability of presentation development.
  2. Leverage great tools.
  3. Leverage copious documentation about other technologies.
  4. Improve performance of page rendering.
  5. Improve the productivity of the debugging process.
  6. Speed up the consulting process by allowing UI to be developed against a static model, then replaced at the end by a dynamic model.
  7. Lower some barriers to entry regarding common standards and technologies.


  1. Same pipeline definition and processing works outside Zope (using the <oxygen/> editor, for example) and inside Zope (using libxml2 and libxslt).
  2. Break processing into multiple, focused stages whose inputs and output can be viewed, verified, and debugged.
  3. Might want a tool like oxygen to wire up to live Zope data instead of a static model on disk as XML files.
  4. Aim for a well-scoped target, such as presentation UI only.
  5. Understand the difference between what's currently called "skinning" (with gobs of programmer stuff mixed into rendering) and a "theme", where the customer's Dreamweaver person applies the corporate identity and site structure.
  6. Possibility for "contracts" between stages that can be validated using unit tests.  Helps implement the blame game, so that less technical people downstream in the process know they did or didn't screw up.
  7. Pick technologies that are a good fit for certain task.  XSLT might fit for drawing a tree, SAX for link rewriting and resolution, etc.
  8. Smooth migration path.  Allow the output of existing ZPT to become the input to a pipeline.
  9. Allow a pipeline to impose itself on a site or parts of a site, even if the various templates didn't choose to include the ZPT macros.
  10. Avoid brilliance in the pipeline architecture.  We have too much brilliance already.  Try to stick to immediately obvious ideas that don't require new specifications, architectures, documentation, and surprises.
  11. Leave open the possibility to shift some stage processing to Apache.


  1. URI vs. object.  Editors like oxygen want input documents as files or URLs.  In Zope, we want to cache data as Python objects and certainly don't want to make an HTTP request from Zope out to Zope to retrieve a string of data and make a DOM.  However, we really want the same pipeline definition to work unchanged with a static model (oxygen + XML files on disk) or inside Zope.
  2. libxml2 will never do XSLT2, XPath2, XQuery, etc.  Doesn't look at this stage as if any Python-capable XML engine will implement those standards.
  3. Many of the XML editors on the market are written in Java.  Obviously, our Zope side is Python+libxml2.  This limits choices or increases work for choices such as extension functions that might solve some issues.
  4. Multiple input docs.  libxslt can't pass in nodes as parameters.  Thus, you really only get one input DOM.  From inside, you can point at a URI and get some more nodes, or write an extension function.  But it would be a bit nicer if you could pass in data you already had lying around as part of the transformation input.
  5. If we want to have stages whose output can be validated, that means using schemas (RNG, for example) for XHTML.  That means no null namespaces.  That means that XSLT authors have to put the html prefix in front of every div, p, h1, etc. that they want to match on in XPath expressions.
  6. XSLT has a negative reputation, some of it deserved, some not.
  7. Slippery slope on complexity.  For example, it is tempting to promote template reuse with xsl:include and xsl:import.  This opens the same kind of "where did that come from" reaction that ZPT macros produce.
  8. Might be hard to pinpoint performance gains.  Since this architecture will rely on some tricks like keeping pre-computed DOMs until state changes, it might be hard to do an apples-to-apples comparison to existing template architectures.
  9. With the DOM, static models as XML files, and even RNG schemas, we have a much more obvious way for people to discover the data model.  But with Zope, CMF, Plone, AT, and custom types, there is a mish-mash of Python methods in some state of documentation that are used to grab data.  With this model, someone earlier in the pipeline has done that work for you.