Python Django noob: custom forms, errors, and fieldsets

I recently decided to take a break from my own, somewhat-custom, python, web framework to play around with Django. I've been doing some contracting work, and the most popular web framework for Python seems to be Django. I decided that it would be worth my time to actually build something real using it (other than just a tutorial) to get some deeper familiarity with it (and maybe to lift ideas for pybald).

There are a lot of things to like about Django, I'm saving those up for another post. This post is about how I just burned several hours trying to do a few things that seemed like they should be simple but turned out to be oddly frustrating. Im sure that part of this is just not being Django-expert-enough to have seen the obvious answers but the task at hand seemed simple enough: creating a user registration form for my application. (Yes I know there's a user registration django app, but this seemed like a trivial task that should be easy with a good framework)

Custom Forms and Error Display

I started down the rabbit hole when I decided I didn't really like the default way Django renders forms. There are a couple of options (as_table, as_p, as_ul), but whether displaying as a table, a list, or as <p> tags, the form layout didn't match what I was used to. I know tables are often used for layout, but I really try and keep them for tabular data whenever possible. I also generally like to reserve <p> tags for actual text/content paragraphs and prefer <div> tags for logical divisions in a document. Might be nit-picky, but that seems more semantic to me.

I could have lived with one of the default renderings but I found that I also wasn't crazy about the way Django handles validation error display by default. Normally Django spits out a <ul> list of field errors with a css class of "errorlist" right above the offending field (or at the top of the form for general errors). Since the errors are only tied to the field by proximity, there's no way to call out the offending field directly. With some styling, I could have made the error list clearer, but I still wasn't crazy about this pattern.

When designing forms, I like to provide a small amount of usability by highlighting the error field in some way with an error style. At first I could find no way to do this with the default form rendering. Later I figured out that you can modify the widget tied to the field, but that seemed like putting too much display logic in the controller. It also seemed to require a lot of code repetition, putting this piece of code anywhere a form was instantiated and validated. I thought about creating a custom subclass of the form but that seemed heavy handed and I definitely didn't want to alter Django internals. It also required playing with the two different kinds of field objects that come out of a form (more on that later).

for field in form:
     if field.errors:
        f.fields[field.name].widget.attrs['class'] = "field_error"

Adding error css to the widgets tied to fields. Not very DRY and not very clean in the controller / view (or template) separation

I wanted to keep this logic in the template which meant I needed to define my own form rendering. Easy enough, I just wrote my own generic form using the Django template language. I generally put input fields inside <div> tags. I also add a field_error css class to the divs that contain inputs with errors. That lets me define both special styles to highlight that section of the form and the offending fields directly ( with css like: .field_error input {}), as well as use the error class as a key for special effects (like jQuery fades, etc...).

My initial custom form looked like this:

{# Include the hidden fields in the form #}
{% if form.non_field_errors %}
  {% for err in form.non_field_errors %}
    {{ err }}
  {% endfor %}
{% endif %}
{% for hidden in form.hidden_fields %}
   {{ hidden }}
{% endfor %}
{% for field in form.visible_fields %}
    {{ field.label_tag }}
    {{ field }}
    {% for err in field.errors %}
        {{ err }}
    {% endfor %}
{% endfor %}

Now from the Django documentation, they suggest creating a custom inclusion tag to render forms like this, but I held off on that for now (although it looks fairly easy). Since I'm generally producing one form per page (at the moment) I'm just using the Django include tag and including this form template into my page template where I need a form. So instead of {{ form.as_ul }} I'm using this: {% include base_form.html %} and assuming I'm using a context variable of form in the outer template (or using 'with' to override form in the include as described in the docs).

Getting my own error displays working, with field highlighting I was happy with, was fairly easy. Right now validation errors appear as a series of spans next to the input field, but I'll probably change this later.

Error css applied in the display logic. A simple yellow bg color styles allow me to highlight the error and the field.

Custom Form Validation

Next I started working with validation. Since this is for a custom user registration form, the first thing I need to do is validate that the username being registered is unique. Django has the concept of validation functions that can be passed to form fields. For my own web framework I've been using FormAlchemy, to provide a validation system and Django's system is similar in some respects.

def validate_username_unique(value):
    '''Custom validator for user uniqueness.'''
    if User.objects.filter(username=value).exists():
        raise ValidationError(u'Sorry, someone already has that [...]')

class UserRegisterForm(forms.Form):
    # [... stuff ...]
    username = forms.CharField(validators=[validate_username_unique])
    # [... stuff ...]

Nice and clean, this is how I expect form validation to work.

Next I needed to confirm that the password and password confirm fields match. Validators only take one value input, so how do I validate that the fields match? Turns out Django's validators aren't up to the task so you have to plumb a little deeper into how it validates forms (as a note, Django's docs are very good, but often times the difficulty is figuring out where in the docs something may or may not live).

To validate based on two fields you have to override the clean method on your subclassed form. When you want to flag validation errors on a particular field, you set the _errors['FIELDNAME'] on the form to flag the invalid fields. (remember that the clean method must return the cleaned_data member. That bit me in the rear for a while.)

class UserRegisterForm(forms.Form):
    # [...stuff...]
    password = forms.CharField(widget=forms.PasswordInput)
    password_confirm = forms.CharField(widget=forms.PasswordInput)

    def clean(self):
        '''Required custom validation for the form.'''
        super(forms.Form,self).clean()
        if 'password' in self.cleaned_data and 'password_confirm' in self.cleaned_data:
            if self.cleaned_data['password'] != self.cleaned_data['password_confirm']:
                self._errors['password'] = [u'Passwords must match.']
                self._errors['password_confirm'] = [u'Passwords must match.']
        return self.cleaned_data

validation of two dependent fields: the magic 'clean' method and the _error attribute

So now I have a form that checks that two fields are equal or else it flags a validation error. I had to add validation logic using two different mechanisms which seems a little messy but not too bad.

Passwords must match!

Fieldsets

Lastly I decided I wanted to split my form inputs into fieldsets. Generally this is considered good practice, especially for usability and accesibility. The first option would be to just write out the form by hand, including the fieldsets, but that seemed like it could be brittle and not very django-like. Django also has some nice fieldset functionality for it's admin app, so I assumed this would be trivially easy. Google searches seemed to turn up overly complex solutions for something that seems like it should be 'built in'. There's also a library 'django form-utils' that has fieldsets but I was trying to stick with generic Django forms.

This is where my frustration level really started to rise. I became very annoyed at the philosophical position that Django has taken that Django templates can't run arbitrary python code. There are arguments for why this is/isn't a good idea in a template system, but having come from using Mako for my templating system, this limitation started to drive me a little crazy.

My first impulse was to create a one-off custom form and use display logic in the template to change how things were laid out.

Name
{% for fieldname in ('prefix','first_name','last_name') %}
    {# render the fields for this fieldset #}
{% endfor %}

Bzzzt.

TemplateSyntaxError at /register/user
Could not parse the remainder: '('prefix','first_name','last_name')' from 
'('prefix','first_name','last_name')'

Django doesn't allow you to create tuples or lists inside blocks. The for tag only seems to work on iterators passed into the context. This seemed a little annoying to me since this logic seems ideally suited as display logic

Then I decided to create an iterator that I could pass into the context so I could call out the individual field names.

    fieldset = ({'label':'Name','fields':('prefix','first_name','last_name')},)
    return render_to_response('registration/user_register.html',
                             {'form': f,'fieldset':fieldset}, 
                             context_instance=RequestContext(request))

Then in the template I tried using the names on the fields dictionary.

{% for set in fieldset %}
    {{ set.legend }}
    {% for fieldname in set.fields %}
        {{ form.fields[fieldname] }}
    {% endfor %}
{% endfor %}

Nope: TemplateSyntaxError. Again, the Django template language doesn't like you accessing dictionary values by name in a variable block. Attributes seem OK, key values no. I'm not sure I like this "echoes of python" approach in the template language because it means learning another logic system rather than applying the full expressiveness of Python.

So, no dictionaries, I'll pass the fields themselves in the fieldset as iterable objects.

fieldset = ({'legend':'Name',
             'fields':(f.fields['prefix'],
                       f.fields['first_name'],
                       f.fields['last_name'])},)

{% for set in fieldset %}
    {{ set.legend }}
    {% for field in set.fields %}
        {{ field }}
    {% endfor %}
{% endfor %}

Alright, no TemplateSyntaxErrors! but wait... what the...

It took a little experimentation and some object introspection but the issue here is that the form actually contains two representations of fields, bound and unbound. The code: for field in form.fields returns different objects than: for field in form.

Poking around the Django core I found that the iterator for a form instantiates BoundField objects from it's internal fields and returns those. That's what gets rendered as HTML. The docs do talk a bit about this distinction, but it's mostly in passing and mentioning you have some additional methods on BoundFields.

Ok, so knowing I need to pass in an iterator, and that the iterator must return BoundFields to properly render in the template, I came up with this FieldSet class.

from django.forms.forms import BoundField
class FieldSet(object):
    def __init__(self, form, fields, legend='', cls=None):
        self.form = form
        self.legend = legend
        self.fields = fields
        self.cls = cls

    def __iter__(self):
        for name in self.fields:
            field = self.form.fields[name]
            yield BoundField(self.form, field, name)

So now in my 'view' code I instantiate FieldSets and pass them into a new form template that knows what to do with them.

    fieldsets = (FieldSet(f, ('prefix','first_name','last_name'),
                        legend='Name',
                        cls="form_name_info"),
                FieldSet(f, ('username','email'), 
                        legend="User Info"),
                FieldSet(f, ('password','password_confirm'), 
                        legend="Password") )

    return render_to_response('registration/user_register.html',
                          {'form': f,'fieldsets':fieldsets},
                          context_instance=RequestContext(request))

Then I wrote an alternate form template with fieldsets to include when I want to use these fieldsets:

{# Include the hidden fields in the form #}
{% if form.non_field_errors %}
  {% for err in form.non_field_errors %}
    {{ err }}
  {% endfor %}
{% endif %}
{% for hidden in form.hidden_fields %}
   {{ hidden }}
{% endfor %}
{% for set in fieldsets %}
  {{ set.legend }}
  {% for field in set %}
      {{ field.label_tag }}
      {{ field }}
      {% for err in field.errors %}
          {{ err }}
      {% endfor %}
  {% endfor %}
{% endfor %}

And finally: voila, a form with fieldsets generated on the fly, with custom rendering, custom validators, and styled error fields.

Behold! a, well, rather unremarkable form.

Is this the best way to do it? I don't know, but by the end I was just glad I got at least something to work. This was a first pass so I'm sure I'll modify this over time (for example, it would probably make sense to create some kind of a FieldSet collection object that could render out a default fieldset for fields not tied to a FieldSet object). I found this day of frustrations definitely dampened my Django enthusiasm a bit. I'm sure as I understand and accept more of Django's design (like no python code in templates) it will get easier to work with. My initial impressions of Django still stands, it makes getting up and running very fast and easy, but it can get thorny when you deviate from 'the path' at all.