Simple Live Validations in Rails

The first day of working with NuLayer, I had the task of adding live validations to a signup form, something very lightweight and clean. Although there are a few plugins (see below), hand coding this into a rails app is really simple.

The Scenario: Validate the username is unique as it’s typed into the forms text field

1) Live_validations controller and validation method

Create a controller to store your validation methods and add the appropriate routes.

2) Message partial

Create a partial that will handle the HTML that is returned for the ajax call.

In your form add an observe_field helper (JRails or Prototype required). This helper will observe the field every 0.5 seconds and pass the value of the text_field via params[:username] to the live_validation controller method.

I used accept.png and cross.png icons from Fam Fam.

Done.

JQuery + Rails Plugin Options

For straight javascript there are a few options, such as jQuery Validation and LiveValidations.com; these are great for basic validations (ex. email has a proper format) but they don’t have access to ActiveRecord.

There is also a rails plugin LiveValidations that hooks up to your existing validations and comes with a form builder for the AJAX, but after testing it: it doesn’t work with nested forms and with hundreds of lines of code it seemed like overkill.

WeeklyBuilder: A weekly calendar plugin for Rails

I implemented a scheduling system into an app this month and I needed a calendar with monthly and weekly views.

There are over 80 repo’s on github for calendar + ruby. Calendars in rails are a solved problem, right? But oddly I couldn’t find one with a weekly view. All of them are either date pickers or monthly calendars.

So I decided to build my own and I’m happy to announce my first plugin: WeeklyBuilder [Github | Live Demo]

The builder creates a horizontal scrolling calendar, mapping events for each day of the week. The layout is fluid CSS and the hours can be switched from business hours to 24 hours.

The code was inspired by table_builder by Petrik de Heus which is a clean way to generate monthly calenders.

How to Use WeeklyBuilder

First install by downloading the source from github or via command line:
The weekly calendar builder:
The Next/Previous link helper:
The controller code:
Example events model:

The event model only requires 2 attributes: starts_at:datetime and ends_at:datetime to calculate width and position on the calendar. To simplify this for the user, I only ask the user for one datetime (starts_at) and how long it will take to complete, it then calculates ends_at based on those 2 attributes.

Github: http://github.com/dmix/weekly_builder/
Example: http://scheduler.integratehq.com/
Note: The demo uses table_builder for the monthly view and includes a few extra styles on the weekly. Not yet tested on IE6.

Update (03/06): Added a truncate_width method so that long event names are truncated in proportion to the width of the event, this is passed through the week block: |event, truncate|.

Turning Your Macbook Into a Better Ebook Reader

I was reading The Selfish Gene on my Macbook this weekend and since Kindle isn’t yet available in Canada, I have to work with what I have. Here is how to improve the experience (thanks to Andrew Burke for the heads up):

1. Download Adobe Reader or the lighter weight Skim PDF app
2. Open the PDF file, rotate view counter-clockwise (cmd+shift+- in Reader) and make it full screen (cmd+L)
3. Set mouse button to turn to the next page
4. Now the hard part: Turn Macbook sideways so the screen is vertical and can be held like a book
5. Bonus: Download Flux if you do a lot of reading at night, it adjusts the screens contrast based on the time of day

Macbooks have really wide screens that are awkward for ebooks when horizontal but great when vertical. The pages are much larger and fit better into the screen. Plus having the mouse button next to your hand to change pages is more ergonomic.

The drawbacks are that its heavy if your not resting it on a pillow and the glossy screen isn’t quite as readable as E-ink.

Creating a BlackBerry Version of Your Rails Application (Web-based)

With an application being used by salesmen, the most demanded feature has no doubt been a Blackberry version.

I found a good tutorial on developing for the iPhone but almost nothing on working with Blackberry’s, so I decided to post some notes to help any rails developers creating a Blackberry version of their application.

The goal in creating the mobile version was to:

• Minimize the size of the application
• Reduce features to only those necessary when on the road
• Optimize the UI for the smaller screen

Subdomain

The first step was to set up a subdomain for the site, for example mobile.dmix.ca. I deployed the app using Nginx so I just created a new DNS and vhost following Slicehosts tutorials.

It used to be common practice to automatically redirect mobile phones to the mobile version of the site. But this has recently been deemed bad usability, users should always have the choice. I created an entirely seperate rails app; some people create seperate stylesheets and some variable layout elements depending on the user agent, if thats the path your taking Mobile_fu might help.

Detecting user-agent

Instead of automatically redirecting, I set up a helper to check the user-agent to see if includes the string “BlackBerry”.

#application_helper.rb
def blackberry_user_agent?
  request.env["HTTP_USER_AGENT"] && request.env["HTTP_USER_AGENT"][/(Blackberry)/]
end

#application.html.erb
<% if blackberry_user_agent? -%>
  <div class="message">
  <p>Using a Blackberry?
  <a href="http://mobile.site.ca/">Use the mobile optimized version</a>.</p>
</div>

BlackBerry user agent: “BlackBerry8700/4.1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1”

Phone Simulator

If you want to test the site locally you can download the RIM Smartphone Simulator. Its Windows only so I wasn’t able to run it on OSX (I tried Crossover and can confirm it does not work). If you are running Windows it will probably save you a lot of time not having to redeploy the site to test changes.

The Blackberry Browser

By default the Blackberry Browser is configured to ignore stylesheets, javascript and background colors. To make your life easier as a designer you can turn these on and recommend your users to do the same (configuration is in the options inside the browser). This is much easier to do if an IT department controls your users phones, but to be safe always test with it off.

The HTML/CSS support varies depending on the version of the phone, it has been pretty consistent on any phone released in the last 3 years with decent support. The newer Bold’s have a much better browser (4.7) with almost full support of CSS/HTML and zooming (still about half of the capabilities of Safari Mobile). I knew my users would be using Blackberry Curves, so I only had to test on the browser version 4.6.

RIM has  HTML/CSS reference guides that explain the support each browser versions has for the languages.

Optimizing UI

A few notes on the UI…

• Aim for function over form, these are not iPhone’s
• Fonts are automatically converted to the default typeface and are almost always black unless you specify, it’s probably best to leave it that way for usability.
• Pages formats to one continuous page, so remove any sidebars and fancy layouts you had in place
• I kept the navigation at the top of the page and horizontal, and did not repeat it in the footer.
• Be very strict on limiting the amount of images you use
• Use clear separators between content (such as hr and background colors) to separate different areas
• Remember the small screen and constant scrolling makes it difficult for users to become familiar with the layout so be very consistent in design

Optimizing Javascript/CSS

First ask yourself if you really need Javascript on the mobile version. If you use a framework like Prototype, loading an extra 150kb will eat up your users data plans and if you happen to live in Canada, with the second highest data rates in the world, this is a big issue.
There was one section that I needed Prototype and without it I would have had to rewrite the whole section. So I added a check to make sure it only loads on that one action.

<% if current_controller == 'objectives' and current_action == 'new' -%>
  <%= javascript_include_tag 'prototype', 'effects', :cache => true -%>
<% end -%>

For CSS I recommend going through and deleting all the layout code and everything unused, every KB counts. You should also look into Bundle_fu to minimize the HTTP requests and put everything into one file. I also moved all my assets (JS, css, images) to S3 to improve speeds and so that attachment_fu uploads are accessible by both rails deployments (mobile and full version).

Firebug/YSlow Stats

Here is the result of the tweaks done to the mobile version compared to the full:

I would love to post up screenshots and a demo of the finished product but it will have to wait until the full site goes public.

Hackers Market Research: Screen Scraping + Visualization

After learning how to scrape websites I have been playing around with ways to use the information and came across an interesting application: visualizing keywords. Using Wordle, you can take content scraped from your competitors websites, run it through the visualizer and see the most common words used on the sites.

This could have some useful applications for copywriting, SEO and avoiding the most overused jargon (todays winner: CRM).

The Method

I ran a scRUBYt! script to gather all the text from these ten sales software websites (home and product pages):

Salesforce, Right90, PivitolCRM, Sage, Netsuite, Entellium, Goldmine, Maximizer, Softfront, and Aplicor.

You can see the script and the full Wordle cloud.

How to Scrape Websites in Ruby on Rails using scRUBYt!

After launching Contrastream I’ve always wanted to create a script that would automatically add new albums to the site as they were released (saving me a lot of time). The idea was to scrape a bunch of album titles from review sites and combine it with information from Last.fm.

The task seemed daunting at the time, so I never jumped in and tried it. After a year I finally wrote my first script and realized how wrong I was. Scraping sites with ruby is surprisingly easy.

For my first project I decided to try the simple task of pulling product information from homedepot.com. Here’s how I did it after some trial and error. See the completed script: http://www.pastie.org/267676

The Plan

Click for Full-Size:

• Go to http://homedepot.com
• Search for “Hoover Vacuums”
• Find the First Product in the Results
• Click Product Details Link
• Fetch Name + Description
• Repeat for Each Product in the Results
• Save Data to MySQL

What You Need

This is purely a ruby script but I used Rails to save the data using Mass Assignment + ActiveRecord (MySQL).

1. Firefox + Firebug + XPath Checker
2. Set up basic Rails application framework
3. Install scRUBYt! plugin (sudo gem install scrubyt –include-dependencies )
4. Set up the database.yml file
5. Create homedepot.rb in [rails_root]/scripts folder

Now the fun part begins…

Fetching the site

Add this to the top of the script.

	require '../config/environment'
	require 'rubygems'
	require 'scrubyt'
	Scrubyt.logger = Scrubyt::Logger.new
	product_data = Scrubyt:: Extractor.define do

	  fetch 'http://www.homedepot.com/'

These are the basic scRUBYt includes plus the rails environment so we can use ActiveRecord. After that you need to direct the script to the site your scraping (called fetching).

Searching the site

Now that the page is loaded up, it’s time to search the site.

fill_textfield 'keyword', 'hoover vacuums'
		  submit

I highlighted the search input box in Firefox and viewed source. I found that the input name was “keyword”. The second string is the search query “hoover vacuums”.

Find the First Product in the Results (Create a Pattern)

The search results page shows a list of Hoover products. The next step is to create a pattern that the script will follow. In this case, the pattern is: for each product in the results I want to click and see the product details.

product_row "//div[@class='product']" do
   …
end

From viewing the source I found that each product is wrapped in div class=”product”. Now for each div with a class=”product” the next action will take place.

Click Product Details Link

The link to the product details page is the name of the product. For example the link “Hoover Legacy Pet Rewind Vacuum” goes to the details page. Because the link that will be clicked is unique on each row, I need to use the xpath of the link to create a pattern. (Otherwise I would have used product_link “View Details”.)

    product_link "/form/div[1]/p[1]/a", :generalize => false do

To find the xpath, right click the first link in Firefox and click view XPath. This is the xpath for the link in side the div: “/form/div[1]/p[1]/a”. This shows that the link was wrapped in a form tag, followed by div, follow by a p (all within div class=”product”).

Now scRUBYt has a pattern: for each product in the search results, click the link (located inside a form, div and p tag).

Each product in the results has the same HTML structure so the script can easily find the next product in the results.

Grab Title + Description

Now that the script is in the product details page, it’s time to start scraping some data. I wanted two things from this page, the name and description.

	   product_details do
	     product_record "//p[@class='product-name']" do
	        name "Homelite 20 In. Homelite Corded Electric Mower"
	     end
	     parent "//div[@id='tab-features']" do
	       description "/p[1]"
	     end
	   end

First, from looking at the HTML, I found the name was wrapped in the class “product-name”, so now I can grab the contents. I copied the title from the first product so scRUBYt can create a pattern for the rest of the products. Now the script knows exactly what data to kick back (the text within the “product-name” div).

The description is similar except I used the xpath “/p[1]” to create a pattern instead of copying the whole paragraph of text.

The “name” + “description” preceding the paths above it is the label attached to the data when you export the data later on.

Save to MySQL

The script is pretty much done; it is now searching for hoover vacuums, going to each product page and grabbing the name/description. Now you can save the data to MySQL (or any ActiveRecord DB).

The first step is to name your database columns the same name as what you labeled the data above. In this case I used “name” and “description”. Mass assignment will automatically save the data to the corresponding DB columns.

	product_data_hash = product_data.to_hash

	product_data_hash.each do |item|
	  @product = Product.create(item)
	  @product.save
	end

Now, we save it to the DB by converting the data to a hash. Each item creates a DB entry and mass assignment automatically sorts the data into the columns.

Note: you could also save the data to XML or a basic text file.

The Complete Script

You can view the complete script here: http://www.pastie.org/267654

To run the script, run “ruby homedepot.rb” in a command line.