We just released a preview of our new HTML5 editor application on preview.gliffy.com (check it out!) Just before we rolled it out, I used usertesting.com to quickly get feedback from real users.
usertesting.com has a crowdsourced model where you can ask their network of users to do a number of tasks and provide feedback. They record screencasts and audio while their testers look at your site. You pay on a per-review basis. It comes out to about $30 per user test, but they have bulk discount packages. There is some basic demographic targeting, but I didn't use this on my first test. Here's a screenshot of their test results page:
For our rollout, I defined a test at 6pm on a Friday and I got 4 responses by 8pm! Most of the people seemed to be from Europe (not a surprise given the time I started testing.)
They don't really say on their site, but their testers roughly have a 15 minute budget. One tester went for almost 30 minutes; but he went above-and-beyond. In my first test, I gave each person the option of trying to draw one of three diagrams from samples on our website. I tried to find tougher diagrams to see how the users would do in a time crunch.
Watching the screencasts of each session was really interesting. I really felt for some of our testers as they struggled to find the right tool. At the end of the test, the testers also filled out a survey that I defined.
Overall, I like the format. It gets close to having people come in to test, but without the coordination hassle. The main downsides are the inability to question the user directly after their session and the rough time limit. The latter makes it difficult to get feedback on features that advanced users would find useful.
I'm a believer in Agile, so getting fast feedback really makes up for the downsides. Next time I'll try setting up tests that are easier to do in 15 minutes :-)