Introducing SeaState, a progressive web application for use in usability testing sessions, allowing the facilitator of the session to record the subject’s screen, facial expressions and voice for the duration of the session to a video and save it to their computer, without the need to install any software or have a network connection. You can try it out here: https://seastate.wemakewaves.digital
Why we built it
User testing is the foundation on which successful digital products are built. It helps answer key questions such as: Does what you’re creating meet your user’s needs? Can they use it easily? What other opportunities do users spot in our products that we may have missed? So it’s important we make sure the sessions are as effective as possible.
When we run a user testing session we typically have a member of our team responsible for taking notes while another facilitates. Often, this works fine, more frequently however we’ve seen the value in being able to more vividly share what the user was doing, saying and feeling at certain points of our testing, and a video recording is an effective way of doing that.
In looking to introduce video recording to our sessions we had a few options, some paid subscription services such as usertesting.com and Lookback.io. Other options include software such as Camtasia or Quicktime
We thought our needs were simple but a bit different from some of the options we reviewed, we were looking for something that;
- Could record what the user was doing on the screen
- Record what they were saying
- Perhaps most interestingly, record their facial expressions as they were completing tasks.
- Didn't depend on a network connection as we couldn't always guarantee one during our sessions
How we built it.Creatively combining the following new & experimental web platform API’s we were able to piece together the elements we needed to record a users screen, their audio and webcam and save that video to disk without the need for a network connection.
This API allowed us to request a video (and optionally an audio) stream from the user’s display, allowing us to record what the user is doing on the device during testing.
Since the audio from displayMedia API is actually from audio playing on the device, we needed to use this API to get the audio track from the user’s microphone to record what they were saying during the testing session. In addition, this API lets us request a video stream from the user’s webcam so we can record their facial expressions during the session.
This method on HTML video elements allowed us to pop the user webcam video stream into a floating window on the user’s screen, so the displayMedia video stream would now include the users webcam video.
We have a video track from the displayMedia stream and an audio track from the getUserMedia stream. So we need to create a new MediaStream instance and add these tracks to it and pass that to the MediaRecorder to record.
The bit that makes this a truly client-side only solution*. streamsaver.js allows you to stream data from the media recorder to a service worker proxy and return it back with the Content-Disposition: attachment header as if the stream was coming from a server and the browser will start saving it to disk.
- there is an initial connection to another host required to set up the service worker as mentioned in the next steps below.
Observations & challenges.
While things are working we did have a few bumps along the road and some bumps that are still to be smoothed out, a summary of which are below.
Launching a picture in picture video from a userMedia stream.
We wanted to launch the picture-in-picture as soon as the user webcam stream became available. Unfortunately, the requestPictureInPicture can only be triggered by a user event, i.e a click on a button. And since the request to get the user’s webcam video is asynchronous (you need to ask for permission first), we can’t do this. This created a clumsy user experience. To get around this we created a “warm-up video” playing in the background that could be launched when clicking the button to ask for permissions, then swap the video track when permission was granted.
Rather large file sizes
The video is encoded as a webm file, however, due to limitations with single-pass encoding in the browser 10 seconds of video equates to approximately 1mb in file size. So there’s a chance these files could get quite large. We could try experimenting with the bitrate to make some reductions, the alternatives are to introduce a server size option that can re-encode at smaller sizes.
Video not scrubbable/seekable (sometimes!)
WebM videos encoded by the MediaRecorder API have sporadic support for seeking/scrubbing. There’s a ‘wontfix’ bug in Chromium flagged here: 642012 - MediaRecorder: consider producing seekable WebM files. Firefox does have support for it. We do have an option to experiment with using an ebml encoder on our stream to see if we can make it seekable.
Not entirely a surprise as this was partly an experiment using new WebAPIs, at the time of writing only Chrome supports all the necessary parts to make this work. Firefox is almost there with the picture-in-picture being only supported on Windows currently.
The primary value of SeaState was for quick access, local usability session recording, which it’s pretty close to delivering and we’ve used it numerous times. Some obvious areas for improvements include;
- Better user experience for offline support. A network connection is still required the very first time you record a video since we need to establish the service worker proxy for StreamSaver, we could explore ways of establishing this connection when accessing the application for the first time.
- An exciting area to explore next would be offering some server-side option to allow for better encoding, sharing, annotation and collation of sessions around particular projects.
We’d love to hear from you if you find SeaState useful at all or have any ideas or suggestions for it, just get in touch