The ever-helpful naneau has just pointed out on #zftalk that a validator has to exist for all form elements.

So by simply adding this to my $validators:

'*' => array()

I default all elements to use an empty validator. Problem solved!