Overview

A few months back I remember seeing Dr. Nic's Custom Google Forms Post (and source) and thinking that was genius. The takeaway:

If a service doesn't give you the programmer enough control over their content, write a web scraper and build an API from the raw html.

All services should make it top priority to give developers complete control over the system right away. I don't mean "make it so we can do a million things", rather, make it so we can easily access the core data in a scalable fashion. If you do that for us, we'll make you big. Google doesn't care about that though, they're already big :).

Anyway, check out the google form (before) and the styled one using the api (after):

Before After

Online Form Services

There are quite a few web form services out there that I've tried:

  1. Google Forms (live examples)
    • 100% Free
    • Minimal feature set (6 form field types only)
    • Easy to use
    • Lots of (ugly) templates
    • Can only view form responses by going to the Google Spreadsheet. You can tell Google Docs to send you an email whenever someone fills out a form, but you have to go to the actual spreadsheet to view the responses (waste of time).
  2. Wufoo (live examples)
    • 3 forms free (100 entries per month, which is nothing).
    • Pretty expensive otherwise
    • Tons of form field types
    • Very nice interface, very easy to use.
    • Form API
  3. PandaForm
    • 100% free
    • Almost the same feature set as Wufoo, but it's FREE.
    • No limit
    • No API

All of them are great if you just need to get up a form to collect information and the user experience doesn't really matter.

But once user experience matters, good luck. Wufoo and Google both allow you to style the forms with CSS, but unless you're a ninja, you can't remove the copyright info and "powered by X" stuff that they all come with. To top that off, you only have 2 options for viewing the form:

  1. On their website (definitely not an option if you want a seamless UX).
  2. In an iframe. Needless to say, iframes are inherently limited. Good luck applying cascading styles to your form and using javascript to enhance the experience (or for client side validations!). You're stuck with the defaults...

I'd go with Google Forms if you want just a basic setup. It's free, and you don't have to worry about being charged. And while PandaForm looks very cool, we know Google Forms will be around for a while.

Build an API around Google Forms

I spent a late night once a while back seeing how Google Forms were structured. After spending a few weeks with the Surveyor Ruby Gem and seeing how they built forms with haml, I decided you could probably do that with Google Forms. Combine that with Dr. Nic's idea, and you've got a very customizable web form setup.

1. Install Googletastic

One of my clients wanted a completely custom interface to Google Apps to manage all of their documents/manuals/brochures, forms, assets, communication, etc. So I built Googletastic. Here's a Sinatra/Googletastic demo I setup randomly to show of it's features. Doesn't do much justice, but you can fork the source anyway.

Googletastic uses Nokogiri to parse XML from the Google Data API. After spending hours on this, I discovered HTTParty. Would've made my life easier, could've used JSON :). But it's done, and it works.

Install the gem:

1
 
sudo gem install googletastic

2. Create a Google Form

Create a simple Google Form, you can copy one of these templates.

3. Convert the Google Form to JSON!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
 
# setup credentials
Googletastic.keys = {
  :username => "[email protected]",
  :password => "my-password",
  :domain => "my-site.org" # only if applicable
}

# get first form from your account (latest form)
form = Googletastic::Form.first
hash = form.to_hash
#=> [
  {:tag=>"text", :value=>[""], :help=>"", :key=>"email", :required=>false, :id=>"2", :title=>"Email"},
  {:tag=>"text", :value=>[""], :help=>"", :key=>"firstname", :required=>false, :id=>"0", :title=>"First Name"},
  {:tag=>"text", :value=>[""], :help=>"", :key=>"phone", :required=>false, :id=>"3", :title=>"Phone"}
]

Here's how that works:

  1. Grab the list of spreadsheets from your Google Spreadsheets.
  2. Login to your Google Apps account using Mechanize.
  3. For each spreadsheet, find the formkey by parsing random html and javascript after a few redirects (since there's no way to get a formkey from Google other than by manually going to the url or doing this).
  4. Download the form at the url http://spreadsheets.google.com/viewform?hl=en&formkey=#{form_key}#gid=0
  5. Parse the html into the data structure used to build the form.

An paragraph text node from a google form looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
 
<div class="errorbox-good">
  <div class="ss-item ss-paragraph-text">
    <div class="ss-form-entry">
      <label class="ss-q-title" for="entry_1">How could we improve?
      </label>
      <label class="ss-q-help" for="entry_1">
      </label>
      <textarea name="entry.1.single" rows="8" cols="75" class="ss-q-long" id="entry_1"></textarea>
    </div>
  </div>
</div>

So googletastic's just saying, for each <div class='ss-item'>, find the title, help text, description, default values, whether or not it's required, and the id of the entry. We know that id's look like entry_(\d+) and form field names, which are passed as params, look like entry.(\d+).(single|group). From that information, we have the entire form structure.

You could do the same thing to wufoo and pandaform no problem.

4. Build a HAML template around the Google Form API

Here's the first template I made using this new-found api:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
 
%form#ss-form{:action => @post.data["submit_to"], :method => :post}
  - @fields.each do |entry|
    - extra_classes = entry[:required] == true ? "ss-item-required " : ""
    - extra_classes += "ss-#{Googletastic::Form.to_google_tag(entry[:tag])}"
    - single_id = "entry_#{entry[:id]}"
    - single_name = "entry.#{entry[:id]}.single"
    - group_id = "group_#{entry[:id]}"
    - group_name = "entry.#{entry[:id]}.group"
    .errorbox-good
      .ss-item{:class => extra_classes}
        .ss-form-entry
          %label.ss-q-title{:for => single_id}
            = entry[:title]
            - if entry[:required]
              %span.ss-required-asterisk *
          %label.ss-q-help{:for => single_id}= entry[:help]
          - tag = case entry[:tag].to_sym
            - when :textarea
              %textarea.ss-q-long{:id => single_id, :cols => "75", :name => single_name, :rows => "8"}
            - when :text
              %input.ss-q-short{:id => single_id, :name => single_name, :type => "text"}/
            - when :radio
              %ul.ss-choices
                - entry[:value].each_with_index do |value, i|
                  %li.ss-choice-item
                    %input.ss-q-radio{:id => "#{group_id}_#{i}", :name => group_name, :type => :radio, :value => value}/
                    %label.ss-choice-label{:for => "#{group_id}_#{i}"}= value
            - when :checkbox
              %ul.ss-choices
                - entry[:value].each_with_index do |value, i|
                  %li.ss-choice-item
                    - attributes = {:id => "#{group_id}_#{i}", :name => group_name, :type => :checkbox, :value => value}
                    %input.ss-q-checkbox{attributes}/
                    %label{:for => "#{group_id}_#{i}"}= value
            - when :select
              %select{:id => single_id, :name => single_name}
                - entry[:value].each do |value|
                  %option{:value => value}= value
            - when :range
              %table{:border => "0", :cellpadding => "5", :cellspacing => "0"}
                %thead
                  %tr
                    -# need start and end ones
                    %td.ss-scalenumbers
                    - entry[:value].each_with_index do |num, i|
                      %td.ss-scalenumbers
                        %label.ss-scalenumber{:for => "#{group_id}_#{i}"}= num
                    %td.ss-scalenumbers
                %tbody
                  %tr
                    %td.ss-scalerow.ss-leftlabel= entry[:left_label]
                    - entry[:value].each_with_index do |num, i|
                      %td.ss-scalerow
                        %input.ss-q-radio{:id => "#{group_id}_#{i}", :name => group_name, :type => :radio, :value => num}/
                    %td.ss-scalerow.ss-rightlabel= entry[:right_label]
            - when :grid
              %table{:border => "0", :cellpadding => "5", :cellspacing => "0"}
                %thead
                  %tr
                    %td.ss-gridnumbers
                    %td.ss-gridnumbers
                    - entry[:labels].each_with_index do |val, i|
                      %td.ss-gridnumbers
                        %label.ss-gridnumber= val
                    %td.ss-gridnumbers
                %tbody
                  - entry[:value].each_with_index do |value, i|
                    %tr{:class => "ss-gridrow #{(i % 2 == 0) ? "ss-grid-row-even" : "ss-grid-row-odd"}"}
                      %td.ss-gridrow.ss-leftlabel.ss-gridrow-leftlabel= value[:label]
                      %td.ss-gridrow
                      - entry[:labels].each_with_index do |label, j|
                        %td.ss-gridrow
                          %input.ss-q-radio{:id => "group_#{i}_#{j}", :name => "entry.#{i}.group", :type => :radio, :value => label}/
                      %td.ss-gridrow
  %input{:name => "pageNumber", :type => "hidden", :value => "0"}/
  %input{:name => "backupCache", :type => "hidden", :value => ""}/
  .ss-item.ss-navigate
    .ss-form-entry
      %input{:name => :submit, :type => :submit, :value => "Submit"}/

That looks crazy but it's actually really simple: for each form field, build a renderer depending on the field type.

Then you just style the elements and you're golden.

5. Add Validations (and other Javascript)

Now that you have the form and can render it independently of Google, you are free to set up custom validations.

In jQuery, using the jQuery Validate Plugin, you can do something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
 
$(document).ready(function() {
  $("#ss-form").ajaxForm({
    success: function(responseText, statusText, xhr, $form) {
      alert("Thanks for filling it out!\n\nMaybe one day we'll make a chart of the responses.");
    }
  });

  $(".ss-form-entry").each(function() {
    var context = $(this);
    var required = $(".ss-required-asterisk", context).get(0) != null;
    if (required) {
      $("textarea, input, select", context).addClass("required");
    }
  });

  // construct jquery-validate.js validation rules
  var rules = {};      
  $(".ss-form-entry .required").each(function() {
    var element = $(this);
    var name = element.attr("name");
    rules[name] = "required";
  });

  // setup validation
  $("#ss-form").validate({
    rules: rules,
    errorPlacement: function(error, element) {
      var entry = element.parents(".ss-item");
      entry.addClass("errorbox-bad");
      error.insertAfter(entry.find(".ss-q-title"));
    }
  });
});

Some notes:

  • If you're using these forms on a completely static site and don't have server-side access (Github Pages for example), then you can't do server-side validations obviously. This means that if users don't have javascript enabled (which is a tiny fraction nowadays), they could fill out forms incorrectly because they'd bypass javascript validations.
  • So, I recommend validating all required fields in jQuery. Otherwise, if they submit the form and something's wrong, Google will redirect them to the ugly Google Form page for them to correct invalid fields. Let's avoid that :). However, if you have server-side access, you could capture the response, extract the json, and render that using your HAML template. I'm doing that on a site now.

6. Use Google Forms in Your App

To sum up, here's all you need to use Google Forms in this completely custom way:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
 
require 'rubygems'
require 'googletastic'
require 'json'

# setup oauth parameters
Googletastic.keys = {
  :username => "[email protected]",
  :password => "mypass"
}

def parse_form(form)
  title = form.title
  file_name = title.downcase.gsub(/[\s_-]+/, "-").gsub(/[^a-z0-9-]/, "")
  content = ""
  content << "---\n"
  content << "title: #{title}\n"
  content << "---\n\n"
  content << JSON.generate(form.to_hash)
  File.open(file_name, "w+") { |file| file.puts content }
en

# in this example, save form json to jekyll-style post
Googletastic::Form.all.each do |form|
  parse_form(form)
end

# or if you want to skip the mechanize part and know the form key...
Googletastic::Form.new(:form_key => "abc123", :title => "What do you think of the Google Form API")
parse_form(form)

Then just build an ERB/HAML view of it, add some validations, style it, and it looks like you've built a survey system from scratch.

Here's what the final result looks like. And this is the original form.