If you are ever asked to measure ROI of usability initiatives, then be careful where you tread and what you promise. Good usability has benefits that can be measured through surveys and interviews, such as increased user satisfaction and trust. However, measuring usability ROI, especially for revenue increases, is challenging. It’s not impossible, mind you, but challenging.
Before jumping into measuring usability ROI, it’s important to be aware of the challenges that might impact the objectivity and credibility of the outcome.
Problem: External Factors
Isolating usability as the single or primary reason for increased revenue can be challenging because external factors might also be attributed to the increase, such as:
- increased marketing & promotion
- improvement of offering
- price
- competing offerings and their value relative to your offering
These factors can make it difficult – or even unwise – to state that better usability was the cause for a revenue increase.
However, leaner times present an opportunity to demonstrate usability can have a healthy ROI. If your organization is spending the same or less on marketing, won’t alter product or service lines or price significantly, then you are in a much better position to claim usability improvements as a primary cause for revenue increases. It’s rare when all of those things line up in your favor, but it does happen.
Also, your offering might be something that has limited competition, such as event registration or something that is offered seasonally. In such cases, the only other factor is likely to be marketing & promotion efforts.
Problem: ROI is not tied to revenue
If the area targeted for usability improvements is not tied to revenue, then you’re facing a much tougher road. For example, the usability improvement might lead to an increase in productivity. In this case, there are significantly more metrics to collect. Also, C-level executives want increased productivity, but want increased revenue (surprise!) much more.
It is still possible to show a good ROI in this case, but it’s better to start with an area that is tied to revenue.
Once key people at your organization see that spending money on usability can have a positive impact on revenue, then making a case for ROI connected to productivity will be that much easier to demonstrate and understand.
In a future post, I’ll detail a project that tied usability improvements and productivity increases together to measure ROI.
For this post, I am sticking to revenue increases as the focus.
Calculating ROI
I would recommend using a simple method for calculating usability ROI. Jakob Nielsen, Craig Tomlin and Human Factors International have similar recommendations, but the simplest formula essentially looks like this:
increase in revenue / total cost of usability effort = ROI
Here’s a breakdown with some example numbers:
|
Labor Costs
Incentives
Total Cost of Usability Effort
|
$3000
$500
========
$4000
|
| Increase in Revenue |
$5000 |
| ROI |
125% |
Example in action
I’ve setup a quick review of usability ROI in action – with screenshots! In this case, I tested a previous event registration form (2008 Form).
2008 Form
As far as methodology, I stuck to a simple qualitative test. There’s plenty of information and discussion on usability testing methodology, such the pros and cons of qualitative and quantitative methods. So, I’ll leave review and discussion of various methodologies for another time.
For now, just remember that – to paraphrase Nielsen – zero testing gives you zero insight. In other words, find a method that works best for your particular cases and use it. If it is efficient, repeatable and reliably discovers the problems, then keep using it.
In the previous form, two issues formed the poor usability refrain, including:
- the separation of the number of tickets needed and the attendees
- the intent of the submit button
In the case of tickets and attendees, users were confused as to why those two items were separated. For example, one user noted that they had already filled in the names of attendees and that should imply how many tickets were needed. (Good idea, but what if George, Jr. or James Eason, M.D. wants to go?). Also, the drop down limited the number of tickets that could be purchased. And what if the number of tickets didn’t match the number of attendees?
Another issue was the submit button, specifically the wording. The form used a third party to collect payment, so it was deemed necessary to create another step.
If the wording had been more clear, then it might have been more usable. For the tickets & attendees issue, some additional error handling or better layout might have helped. However, it’s never as simple as that, right? At the time, the developer put together a form that met requirements. Users could register. The form owners were able to get total ticket sales & attendee names. Case closed.
So, armed with the results, the new design (2009 Form) incorporated two key changes.
2009 Form
First, the form allowed the users to key in the number of tickets and then the attendee fields were dynamically generated. This allowed for the form to respond to the user input and provide a connection between the required information.
Second, the submit button was clear. Also, the payment gateway was implemented differently, so a second step was not necessary. If the payment gateway process had not changed, the wording would have been different! Even so, the single step form was a usability improvement.
The results
Here is the ROI based on the increase over the previous year and the cost of the usability testing.
| Total Cost of Usability Effort |
$780 |
| Increase in Revenue |
$4800 |
| ROI |
615% |
Counter-points
Some might argue that the first iteration of the form simply suffered from an "uninspired design", had a horrible layout or other design flaws. No argument here. However, as Jared Spool at UIE points out, even a simple design can be improved.
To that point, the layout and design would have been updated regardless of usability testing, but, even without a layout improvement, users could still complete the form and register. The takeaway is that the most essential issues were revealed only after watching non-web professionals actually try and complete it. Go figure! On top of that, there was a return on investment in the form of increased ticket sales.
Some might also point to external factors, such as increased marketing or better product offering. However, there was no increase in spending for marketing and the offering was essentially the same as the prior year. External factors still play a part, but they were limited in this case.
Overall, I don’t yet see a way to completely single out usability improvements as the cause for increased ROI, but usability improvements can and do impact revenue. So, for now, it seems best to:
- find areas tied to revenue where usability can be improved.
- be aware of external factors that might also impact revenue and, consequently, affect the validity of usability ROI.
If you’ve got other ideas, think I left something out, or just think I am completely off-base, please drop in a comment below.
Special thanks to Roc Johnson and the teams at Methodist Healthcare and Le Bonheur Children’s Medical Center.
Usability and ROI
If you are ever asked to measure ROI of usability initiatives, then be careful where you tread and what you promise. Good usability has benefits that can be measured through surveys and interviews, such as increased user satisfaction and trust. However, measuring usability ROI, especially for revenue increases, is challenging. It’s not impossible, mind you, but challenging.
Before jumping into measuring usability ROI, it’s important to be aware of the challenges that might impact the objectivity and credibility of the outcome.
Problem: External Factors
Isolating usability as the single or primary reason for increased revenue can be challenging because external factors might also be attributed to the increase, such as:
These factors can make it difficult – or even unwise – to state that better usability was the cause for a revenue increase.
However, leaner times present an opportunity to demonstrate usability can have a healthy ROI. If your organization is spending the same or less on marketing, won’t alter product or service lines or price significantly, then you are in a much better position to claim usability improvements as a primary cause for revenue increases. It’s rare when all of those things line up in your favor, but it does happen.
Also, your offering might be something that has limited competition, such as event registration or something that is offered seasonally. In such cases, the only other factor is likely to be marketing & promotion efforts.
Problem: ROI is not tied to revenue
If the area targeted for usability improvements is not tied to revenue, then you’re facing a much tougher road. For example, the usability improvement might lead to an increase in productivity. In this case, there are significantly more metrics to collect. Also, C-level executives want increased productivity, but want increased revenue (surprise!) much more.
It is still possible to show a good ROI in this case, but it’s better to start with an area that is tied to revenue.
Once key people at your organization see that spending money on usability can have a positive impact on revenue, then making a case for ROI connected to productivity will be that much easier to demonstrate and understand.
In a future post, I’ll detail a project that tied usability improvements and productivity increases together to measure ROI.
For this post, I am sticking to revenue increases as the focus.
Calculating ROI
I would recommend using a simple method for calculating usability ROI. Jakob Nielsen, Craig Tomlin and Human Factors International have similar recommendations, but the simplest formula essentially looks like this:
increase in revenue / total cost of usability effort = ROI
Here’s a breakdown with some example numbers:
Labor Costs
Incentives
Total Cost of Usability Effort
$3000
$500
========
$4000
Example in action
I’ve setup a quick review of usability ROI in action – with screenshots! In this case, I tested a previous event registration form (2008 Form).
2008 Form
As far as methodology, I stuck to a simple qualitative test. There’s plenty of information and discussion on usability testing methodology, such the pros and cons of qualitative and quantitative methods. So, I’ll leave review and discussion of various methodologies for another time.
For now, just remember that – to paraphrase Nielsen – zero testing gives you zero insight. In other words, find a method that works best for your particular cases and use it. If it is efficient, repeatable and reliably discovers the problems, then keep using it.
In the previous form, two issues formed the poor usability refrain, including:
In the case of tickets and attendees, users were confused as to why those two items were separated. For example, one user noted that they had already filled in the names of attendees and that should imply how many tickets were needed. (Good idea, but what if George, Jr. or James Eason, M.D. wants to go?). Also, the drop down limited the number of tickets that could be purchased. And what if the number of tickets didn’t match the number of attendees?
Another issue was the submit button, specifically the wording. The form used a third party to collect payment, so it was deemed necessary to create another step.
If the wording had been more clear, then it might have been more usable. For the tickets & attendees issue, some additional error handling or better layout might have helped. However, it’s never as simple as that, right? At the time, the developer put together a form that met requirements. Users could register. The form owners were able to get total ticket sales & attendee names. Case closed.
So, armed with the results, the new design (2009 Form) incorporated two key changes.
2009 Form
First, the form allowed the users to key in the number of tickets and then the attendee fields were dynamically generated. This allowed for the form to respond to the user input and provide a connection between the required information.
Second, the submit button was clear. Also, the payment gateway was implemented differently, so a second step was not necessary. If the payment gateway process had not changed, the wording would have been different! Even so, the single step form was a usability improvement.
The results
Here is the ROI based on the increase over the previous year and the cost of the usability testing.
Counter-points
Some might argue that the first iteration of the form simply suffered from an "uninspired design", had a horrible layout or other design flaws. No argument here. However, as Jared Spool at UIE points out, even a simple design can be improved.
To that point, the layout and design would have been updated regardless of usability testing, but, even without a layout improvement, users could still complete the form and register. The takeaway is that the most essential issues were revealed only after watching non-web professionals actually try and complete it. Go figure! On top of that, there was a return on investment in the form of increased ticket sales.
Some might also point to external factors, such as increased marketing or better product offering. However, there was no increase in spending for marketing and the offering was essentially the same as the prior year. External factors still play a part, but they were limited in this case.
Overall, I don’t yet see a way to completely single out usability improvements as the cause for increased ROI, but usability improvements can and do impact revenue. So, for now, it seems best to:
If you’ve got other ideas, think I left something out, or just think I am completely off-base, please drop in a comment below.
Special thanks to Roc Johnson and the teams at Methodist Healthcare and Le Bonheur Children’s Medical Center.