Photo by Chloe Fan

Headshot is an application that helps you take pictures of yourself by telling you how to move your camera until your head is in the right spot. This was not an easy task. Writing the face detection was the easy part. Much more challenging was to get the user interface correct. How do you tell people how to move their phones? How often do you tell them? How do you teach people to use the app? The first version of headshot was a good start and got people excited, but it still needed a lot of work.

Over the last few months I’ve been polishing headshot, getting feedback from users and trying to develop the right audio prompts and timing to help you get that perfect shot. Here is a summary of all of the improvements for version 3 of headshot:

• Added multiple person mode Now you can get the perfect shot with two or more people! In multiple person mode Headshot gives instructions by taking the average location of all faces in the image.
• Auto snapshot when perfect physically snapping a photo can cause blurry images. Now headshot will automatically take a photo when your head is in the right place so all you need to do is smile!
• Greatly improved audio feedback Headshot now has a bit more personality: It tells you when it can’t see you, and also immediately tells you once your head is in the right place.
• Fixed bug that caused app to crash on international phones. This bug caused many international users to be unhappy. It should be fixed now!
• New logo! Thanks to my friend Chloe Fan for making the avatar used in this logo. Update: The new logo hasn’t propogated to the Marketplace yet but it should be up soon.

I’m releasing an improved version of headshot today (see this post) and in light of this I wanted to tell everybody how Headshot came to be, and what I’ve learned in developing the app.

I came up with the idea for Headshot somehwere in between my cubicle and the microkitchen at Microsoft Research. The idea was to write an application that helped people take pictures of themselves by using face detection to help them adjust their camera until their head was in the right place. I half expected the idea to work, and thought the idea was novel enough to get me into the finals for a Microsoft Intern mobile app contest that summer. Several of my closest friends told me the app was completely useless because of front-facing cameras, but I continued on, spurred on by faith and a desire to complete what I had started. The end result was a decent app and accompanying video (below) which surprisingly enough won me a brief glimpse of the limelight and a trip to Hawaii.

My contest win and a Computational Photography class spurred me to further develop the app from a prototype into a product. I released Headshot (a paid version and an ad-supported free version) on the marketplace in early 2012.

The app tanked. Twice. It got decent reviews in U.S. markets but was doing very poorly internationally, and nobody was buying a full version. It makes about 9 cents a day right now. Not something I’d consider a success given the novelty of the idea. Why is it doing so badly?

Is it because people don’t find it useful? I don’t think so. My friend told me that a necessary but not sufficient condition for having a successful app has to do with how much time to spent perfecting it, getting the polish just right. That’s what I’ve tried to do this third time around: I’ve spent quite a lot of time perfecting the app, getting feedback from lots of different users about what works and what doesn’t, testing the app on different devices, and trying my best to release a perfect product.

I don’t know if this story has a happy ending, we will see how well the app does. I do know that I created something I’m proud of, and that I’ve learned many lessons from the different phases of the app, here they are, in chronological order.

1. Finish what you start. I never would have won the contest if I had listened to my friends who told me I had a bad idea.
2. Sometimes offshoots of your project are more rewarding than the project itself. I’m very glad I wrote the face detection library, facedetectwp7.codeplex.com to go with this project, in some ways I’m more proud of it than headshot itself.
3. Always make a free version of your app.
4. It is essential to give your app to many people (at least ten) and see how they use it. Have them use all versions (free, paid, trial). Also, make sure you actually listen to their suggestions.
5. Spend 3 times more time on something than you think you should. This is at least how long it takes to get an app polished. Though I admit I’ve yet to see whether this actually makes a difference.

That’s my story. I hope you enjoyed it. If you could download and review headshot, I’d really appreciate it:

This post is for people writing “Metro Style Apps”, or apps built for Windows 8 who are trying to set the source of an image within their WPF project. It should be easy to do this but it is surprisingly tricky and took me 1.5 hours to figure out. Here’s the code:

// Usage
myImage.Source = ImageFromRelativePath(this, "relative_path_to_file_make_sure_build_set_to_content");

public static BitmapImage ImageFromRelativePath(FrameworkElement parent, string path)
{
      var uri = new Uri(parent.BaseUri, path);
      BitmapImage result = new BitmapImage();
      result.UriSource = uri;
      return result;
}

And here’s why it’s such a pain: WinRT applications need the full path to an image, they do not accept relative paths. The full path is confusing, but you can get at it if you’re in a class that extends FrameworkElement (basically any UI element) via the BaseUri property. Totally not obvious and not easy to discover!

I have found myself on 5 or 6 too many red eye flights during my cross country excursions between Pittsburgh and Seattle over the last year or so. Red eyes are great, if you can manage to get enough sleep on the flight and face the next day feeling more or less rested. I have successfully done this 1 out of 6 times. The last time was today. Sure, I probably just got lucky this time, but I do know that I had a couple of tricks that I’ve picked up which I’ve noticed make me feel much better, and which I think are worth sharing. Here are three things you can do to make your red eye trips easier. Feel free to add more tips in the comments:

1. Bring an eye mask and neck pillow, and take off your shoes. This is the only way to go on a red eye. These add so much to your comfort. I try to get a window seat as often as possible so I can lean against the window. Prop your pillow against the window for maximal comfort.
2. Brush your teeth and drink some Emergen-C the next morning. I always bring an (empty) water bottle with me to fill with water + emergen-c. Drinking the emergen-c and then brushing my teeth really feels amazing.
3. Keep napping until the plane stops. Unless you’re in first class, it takes about 5 – 15 minutes to actually get up once the plane has stopped. Don’t waste precious sleep time by waking up early. You’ll have plenty of time.

It’s been almost four months since I released my windows phone face detection library and I’m surprised by two things:

1. Microsoft still hasn’t released a face detection library for windows phone
2. People actually downloaded my library! I’ve gotten almost 270 downloads in 4 months which isn’t stellar but it’s better than I expected.

I’ve released a new version of my face detection library, you can get it at http://facedetectwp7.codeplex.com. There are two big changes there:

1. Major fix so that library works in other countries. I have never had to worry about parsing strings in different languages but apparently ignoring this caused my library to totally not work in other countries (this also caused Headshot to not work at all in other countries which is too bad and explains my poor rating there.).
2. User control for managing camera. I find interfacing with the camera annoying, so I created a user control that you can use to show you a preview of what the camera sees and also lets you programmatically take photos and save them to the camera roll. It’s a useful little gem in the library.

I was really surprised by how long it took me to debug the localization problem. I was even more surprised by how I solved it. I posted a request on twitter for help with my problem, not at all expecting to get a reply and lo and behold, I got a reply! I was pleasantly surprised by the competence of my social network. I guess it helps having both Russian hackers like Vadim, Alexei and Misha, and having amazing researchers like Andy, Ken and Scott all in my social circles. Being able to ask these people anything makes me so much smarter, I always forget how useful just asking people for help is.

So, moral of the story: don’t be afraid to ask questions!

In other news, my blog name is actually relevant now that I do actually climb. Beta is super helpful. I was able to get a v4 that I think would have taken me weeks in just a day because somebody showed me how to do it. Like in climbing, so in coding I guess.

This post is for all my friends and acquiantances that might be wondering what on earth I’m doing at Carnegie Mellon for my PhD. The truth is, I’m doing a variety of things here (I always a variety of side projects to keep me motivated and usually you hear about them in some form or another), however my main research is about probabilistic input. DISCLAIMER: This is NOT a thesis proposal, and there is a chance I might do something entirely different for my PhD thesis, however right now probabilistic input seems like the most likely candidate.

what is probabilistic input?

In a nutshell, the goal of my work is to make user interfaces account for more information when deciding what it is you’re trying to do. I am designing, building and evaluating a new method for modeling and dispatching input that treats user input as uncertain. Modern input systems always assume input is certain, that is it occured exactly as the sensors saw it. When a developer handles an event (say, a mouse event), that event has one x and one y coordinate. This works well for keyboards and mice, but less well for touch, and even more poorly for free-space interactions such as those enabled by the Kinect. After all, your finger is not a point! The stuff I’m working on will allow our input systems to be far more intelligent about interpreting user actions, especially for new input techniques such as touch, voice, and freespace interactions enabled by the kinect. In addition to enabling computers to better understand users, I’m interested in evaluating how we can use this probabilistic approach to design feedback that allows users to better understand how computers are interpreting their actions. For example, what’s the best way for a computer to tell you that it is not sure whether you’re doing a horizontal swipe, or a panning gesture for the kinect? If you think about it, a lot of the interactions you do can be interpreted multiple ways. The challenge of how to communicate this to users to that you understand stuff is ambiguous without being confused or working to hard is a problem I’m trying to solve. Finally, I would like to evaluate how easily developers can adopt this probabilistic approach into real applications, as the ultimate goal of this work is to eventually be adapted into all input handling systems.

what have I done so far?

Most of my work so far has been in designing (and validating through implementation) an architecture for actually dispatching uncertain input. In other words, assume that mouse events now aren’t at a location, but rather have a probability distribution over possible locations. I designed a system that figures out which buttons these new mouse events should go to. This system was published in UIST 2010, you can see the paper here. I then published a refinement of this system (with a few extra bits) that made it much easier for developers to write user controls (buttons, sliders, etc.) for my system. This was published in UIST 2011, you can see the paper here.

what is left?

Right now I’m working on designing better feedback techniques when input is uncertain. After that, I’m going to try to tackle mediation. What’s mediation? It’s basically what shoudl happen when you do something (like a gesture) and the computer can’t decide between two things. So, it asks you what you wanted to do. If it just asked you and had you pick from a list, that would feel unnatural (because it’s a break in your workflow). So, I’m trying to see if there are better ways to mediate between alternate actions. The last piece of my thesis is perhaps the most important and most difficult. It involves evaluating my work on real developers. This is still an unsolved and mostly unexplored area for me, though I know I should be working on it.

what is the best possible outcome for my thesis?

I would be thrilled if at some point in my life I saw mainstream input systems such as those in Microsoft and Apple products turning probabilistic, and if those systems used some of the ideas outlined in previous papers, or papers to come. Given the popularity of natural user interfaces, I think this is a very real possibility, which is quite exciting.

I wrote a port of OpenCV’s face detection library in C# for the Windows Phone. If you just want the link to the library and don’t want to hear my story, it’s http://facedetectwp7.codeplex.com/ If you’re curious, read on:

For the final project of my computational photography class, I developed an app that helps people take pictures of themselves (this was actually a fleshing out of an earlier prototype I’d written while an intern at Microsoft). The app is currently in beta testing and I hope to releas it on the Marketplace before the new year! Post a comment if you’re interested in hearing about the release.

The biggest challenge for this project was the face detection. Why? Because there are no good face detection libraries for the Windows phone. I couldn’t use the popular (and what I later found out was not-as-great-as-it-could-be) OpenCV library because Windows Phones only run on managed code (i.e. no C++). So, I wrote a C# port of the OpenCV viola-jones detector that uses OpenCV’s model files to do face detection. I must thank the folks at http://code.google.com/p/jviolajones/ for their helpful code, I used it as a guide for my port.

The library is available at http://facedetectwp7.codeplex.com/. If you want to build a Windows Phone application that uses face detection, I’d highly recommend looking into it!

Just got back from my favorite conference and wanted to share some of my favorite papers, in case anybody wonders what the bleeding edge of HCI looks like. Here are my favorites:

FingerFlux (http://hci.rwth-aachen.de/fingerflux)
In a nutshell: Electromagnets on a table + magnet on your finger = touchscreen with magnetic force feedback.

Why I liked it: I had this idea when I just got into grad school and was always wondering what having magnetic feedback on a touchscreen would feel like. It was really great to see this idea published, great job Malte et al!

Real-Time Collaborative Coding in a Web IDE (http://groups.csail.mit.edu/uid/collabode/)

In a nutshell: A web-based IDE that allows multiple people to edit the same file safely.
Why I liked it: I liked the idea of this one, because it might provide a better way to do pair programming with David.

SideBySide: Multi-user Interaction with Handheld Projectors (http://www.karlddwillis.com/projects/projection/sidebyside/)

In a nutshell: Play boxing games with your projectors.
Why I liked it: I also had a similar idea (and am trying to publish a paper with Chris and Chloe on it), and think that this is awesome. Great to see it published. Also really well implemented.

No More Bricolage! Methods and Tools to Characterize, Replicate, and Compare Pointing Transfer Functions (http://libpointing.org/)

In a nutshell: Try this: move you finger across a trackpad slowly and see where your mouse goes. Now do the same thing, but move your finger quickly. See how the amount your mouse moves changes? 10,000 lines of code to figure reverse engineer mouse movement on Windows and Mac.
Why I liked it: I think about mouse gain much more than most people, and to me it’s always been a mystery what functions different OSes use.  It’s great to finally see these curves. The fact that they are so different makes me wonder why they haven’t converged on an optimal transfer function yet.

KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth (http://www.youtube.com/watch?v=quGhaggn3cQ)

In a nutshell: Make high resolution 3d models using a Kinect. Objects work best when still.
Why I liked it: I was impressed with the high resolution of the created 3D models.

I made an animation of the evolution of man for my computational photography class. The project assignment was to use image morphing to morph one face into another. You can also use morphing to find an average face, as seen above. It turns out this also works for entire bodies.

Here are some interesting images I generated for my computational photography class this semester. Images are generated using Poisson blending and Mixed Gradient Blending. The idea is to copy in the change in pixel values, not the pixels themselves. You can learn more about the project here (here is my project submission with more detail).

Elephant in New York City.

Name on a Wall

Walking on Water.