SeaState: an in-browser tool for recording usability testing sessions.

If you want to build a successful digital product, it’s a good idea have an effective usability testing strategy. Not only will this help you to understand if what you’re creating meets your users’ needs and if your product is easy to use, it’s also a useful way to discover other opportunities that you may have missed.

During sessions we’ve typically had a team member responsible for taking notes while another facilitates. This isn’t always effective for capturing everything that a user is doing, saying or feeling: it’s easy to miss a subtle moment of frustration or a slight pause indicating possible confusion. On the other hand, a video recording of the session can be a very effective way of capturing everything that’s going on, allowing us to analyse it in our own time and refer back to key moments.

There are a couple of options available already, some paid services such as usertesting.com and Lookback.io and other software such as Camtasia or Quicktime

Our needs were a bit different from the options we reviewed, and non of them could do everything we wanted:

• Record what the user was doing on the screen
• Record what the user and interviewer were saying
• Record the user's facial expressions as they were completing tasks
• Work offline

Enter stage left, SeaState

There's been some interesting advancements in web platform API’s of late and we saw an opportunity to experiment with a number of them whilst building something useful. SeaState is our attempt at just that: a progressive web application allowing the facilitator of a usability testing session to record the subject’s screen, facial expressions and voice, combining these into a single video and saving it to their computer. All without the need to install any software or have a network connection. You can try out SeaState here.

How we built it

Combining the following web platform API’s we were able to piece together the elements we needed.

getDisplayMedia

This API allowed us to request a video (and optionally an audio) stream from the user’s display, allowing us to record what the user is doing on the device during testing.

getUserMedia

Since the audio from displayMedia API is actually from audio playing on the device, we needed to use the getUserMedia API to record the audio from the user’s microphone during the session. In addition, this API lets us request a video stream from the user’s webcam so we can record their facial expressions.

requestPictureInPicture

This method on HTML video elements allowed us to pop the user webcam video stream into a floating window on the user’s screen, so the displayMedia video stream would include the user’s webcam video.

MediaStream & MediaRecorder

We have a video track from the displayMedia stream and an audio track from the getUserMedia stream. So we needed to create a new MediaStream instance, add these tracks to it and pass that to the MediaRecorder to record.

ServiceWorkers & StreamSaver.js

The bit that makes this a truly client-side solution*, streamsaver.js allows you to stream data from the media recorder to a service worker proxy and return it back with the Content-Disposition: attachment header as if the stream was coming from a server and the browser will start saving it to disk.

* there is an initial connection to another host required to set up the service worker as mentioned in the next steps below.

Observations & challenges

While things are working we did have a few bumps along the road, not all of which are completely smoothed out, a summary of which are below.

Launching a picture in picture video from a userMedia stream.

We wanted to launch the picture-in-picture as soon as the user’s webcam stream became available. Unfortunately, the requestPictureInPicture can only be triggered by a user event, i.e a click on a button. And since the request to get the user’s webcam video is asynchronous (you need to ask for permission first), we can’t do this. This created a clumsy user experience. To get around the problem we created a “warm-up video” playing in the background that could be launched when clicking the button to ask for permissions, then swap the video track when permission was granted.

Rather large file sizes

The video is encoded as a webm file, but due to limitations with single-pass encoding in the browser, 10 seconds of video equates to approximately 1mb in file size and so there’s a chance these files could get quite large. We could try experimenting with the bitrate to make some reductions, the alternatives are to introduce a server size option that can re-encode at smaller sizes.

Video not scrubbable/seekable (sometimes!)

WebM videos encoded by the MediaRecorder API have sporadic support for seeking/scrubbing. There’s a ‘wontfix’ bug in Chromium flagged here: 642012 - MediaRecorder: consider producing seekable WebM files. Firefox does have support for it. We do have an option to experiment with using an ebml encoder on our stream to see if we can make it seekable.

Browser support

Not entirely a surprise as this was partly an experiment using new WebAPIs, at the time of writing only Chrome supports all the necessary parts to make this work. Firefox is almost there excepting that picture-in-picture is only supported on Windows.

Next steps

We’ve achieved quite a lot of what we set out achieve with SeaState: we’ve created an in-browser tool that’s easy to access, it works online and it combines all the different facets which are useful to gaining insights from our usability testing sessions. We’ve used it numerous times for our own sessions and we’re finding it useful. However, there are some obvious areas for improvements:

• Better user experience for offline support. A network connection is still required the very first time you record a video since we need to establish the service worker proxy for StreamSaver. We could explore ways of establishing this connection when accessing the application for the first time.
• An exciting area to explore next would be offering some server-side option to allow for better encoding, sharing, annotation and collation of sessions around particular projects.

We’d love to hear from you if you find SeaState useful at all or have any ideas or suggestions for it, just get in touch