How My Comic Book Obsession Birthed a New Functional Testing Tool

Feihong Hsu Testing in Python Birds of a Feather

March 12, 2011

I like comics

I like studying Chinese

我喜欢学中文

我喜歡學中文

These are two tastes that go great together!

Especially when you find a real gem like this:

Problem: Chinese comics portals load slowly and have poor usability Solution: Write a web scraping tool to batch download images, so I canbe a proper Apple fanboy and read comics on my iPad

First approach: urllib2 + lxml.html

Verdict: Total fail, couldn't handle JavaScript and cookies

Second approach: XULRunner

Verdict: Asynchronous logic was painful, Python integration would not have been trivial

Third approach: spynner

Verdict: Not bad! But... some sites don't load. Why?

Fourth approach: Make spynner dumber

Verdict: Frickin' awesome

Coincidence: At Leapfrog, we needed a better way to test JavaScriptbehavior on our sites

Result: Leapfrog subsidizes my comic book addiction

Over time, we added some nice stuff to our spynner fork

- Ability to ignore SSL errors- Form manipulation methods- Screen capture (consistently-sized images)- Other stuff that I can't remember

Question: Hey Feihong, where can I get this sexy library?

Answer: Nowhere, I'm too busy reading comics to open source it

Real answer: We're working on open sourcing it, but we ran into some blocks

(Cast sidelong glance at Terry)

However, we are NOT soliciting suggestions for a

name. We have thePERFECT name already.

PunkyBrowster

Some basic usage

from punky import Browster browser = Browster(auto_load_images=True)browser.create_webview(show=True) browser.load('http://www.duckduckgo.com/') browser.fill('input#hfih', 'How do I de-pube-ify waterless urinals?')browser.submit('form#hfh', wait_load=True) for element in browser.all('#r12 > div'): print unicode(element.toPlainText())

Random note: I never want to have the need to see a urologist. But if Ido, I hope he's

wearing a badge like this:

I am done