Automated user feedback monitoring

Alex Savage
Technical Product Owner

When developing software, it can be easy to feel disconnected from our users. We push out releases and hope that the changes are well received. We never talk to the end user, instead we rely on a customer service layer to forward customer feedback in the form of tickets.

Our customers are constantly giving us feedback on all aspects of iflix. - App store reviews - Social media mentions and messages - Customer service enquiries

Most users leave satisfied

Our customer service filter works, it’s great for getting useful information out of users and squashing those bugs. When the issue is resolved we go on our merry way until the next issue comes up, forgetting about the feedback that once was. Often we never even get to see the issue as it is resolved by customer service and closed on the spot. Most of the user feedback is hidden from the engineers.

Don’t discard old feedback

iflix has a user feedback monitoring system that allows us to see a stream of customer feedback on specific topics. It allows us to track the number of complaints (or compliments) for a focus area over time, to see if we are making a positive impact on the end user’s experience.

We take this feedback and pipe it through Zendesk. Then, by filtering on key words we can collect all the feedback for areas that we care about, and both stream these to a Slack channel and send it through as events into Amplitude.

Feedback to Amplitude pipe Feedback to Amplitude pipe

How does this help?

There are several ways to use this information. Of course you can go and read what the users are saying in the Slack channel, but depending on the quantity of feedback you may not want to do this. An aggregate approach may be more palatable.

Improvement over time

Our first use was to monitor improvements to the user authentication within our app. This includes people complaining that they can’t log in, problems resetting password, and getting the wrong country assigned to them.

Identity issues

Working hard to improve these issues, we reduced the negative feedback in this area from 600 to 450 complaints per week over a two month period.

Real time alerting

We use the rate at which we receive feedback to trigger alerts.

subscription issues

We can see from this chart that the hourly rate at which we were receiving subscription feedback was growing each day. Users are struggling to subscribe to our service. Time to focus more energy on making this experience better and perhaps scale up customer service in this area for a while. Luckily there are no huge spikes to show, but we can set alerting threshold on this data as it is an indicator of an incident. We can also see that there are certain times of the day that users prefer to subscribe or at least like to complain about subscribing.

Summary

Its possible for engineers and others to get additional information from user feedback without getting into the weeds about it. By using historical data we can make use of feedback long after it has been initially dealt with and closed by CS. Our connection with the end user can help drive a quality product. What are other ways we can make the most out of user feedback? Some good first steps that you can take toward aggregating user feedback, is to find some common complaints and group them in a meaningful way, that relates to a technical aspect of your product.