#38 How do you measure the return on analytics?

The Measure Pod

#38 How do you measure the return on analytics?

Play Episode Pause Episode

Mute/Unmute Episode Rewind 10 Seconds 1x Fast Forward 30 seconds

00:00 / 21:49

Apple Podcasts RSS Spotify YouTube

RSS Feed

This week Dan and Dara discuss a listener submitted topic on how to measure the return on analytics. After arguing about the flawed premise of the quest, they explore different ways that analytics specialists and analysts can measure a tangible return for their hard work.

The key ways that were discussed to measure the impact (or return if you will…) of analytics is via:

1. Optimising marketing campaign ROAS
2. Improving website/app CVR via testing and CRO programs
3. Product optimisation to reduce churn or improve engagement and thus LTV
4. Time saving via automation and trusting the data
5. Impartiality meaning that you can recommend the best course of action, even if it’s not the most popular
6. Damage control – minimising negative impact when it’s inevitable (i.e. COVID-19)

In other news, Dan hosts and Dara parties!

Check out on LinkedIn:

• Dan – https://bit.ly/3JQKHEb
• Dara – https://bit.ly/3vzV0bO
• Measurelab – https://bit.ly/3Ka513y

Music from Confidential, check out more of their lofi beats on Spotify at https://spoti.fi/3JnEdg6 and on Instagram at https://bit.ly/3u3skWp.

Please leave a rating and review in the places one leaves ratings and reviews. If you want to join Dan and Dara on the podcast and talk about something in the analytics industry you have an opinion about (or just want to suggest a topic for them to chit-chat about), email podcast@measurelab.co.uk or find them on LinkedIn and drop them a message.

Transcript

[00:00:00] Dara: Hello, and thanks for joining us in The Measure pod, a podcast for people in the analytics world. I’m Dara, I’m MD at Measurelab, and I’m joined as always by Dan, an analytics consultant also at Measurelab. Firstly, hi Dan, how are you this fine sunny week?

[00:00:30] Daniel: Yeah, I’m really good thanks, Dara. Definitely noticing the nice weather, and yeah, spending as much time in it as possible. How about you?

[00:00:37] Dara: I agree, I don’t want to use my wind down so early in the episode, so I’m just going to say yes the sun has been great, but we’re not here to talk about that, at least not yet. We are here to talk about serious analytics business. So we’re quite excited this week, you and I, because our topic is actually a listener suggested topic this week. They weren’t brave enough to come on and discuss it with us, but they thought they’d leave it in our capable hands.

[00:01:02] Daniel: Yeah, capable may not be the word I would have chosen, but nevertheless, we’ll give it our best go.

[00:01:06] Dara: We will, so our topic for this week is how do you measure the return on analytics? I can see why they didn’t want to come on and discuss this with us. This is a big question, it’s a tough question.

[00:01:16] Daniel: Yeah it’s a big question as well. It’s almost a case of justify yourselves in the company you work for within probably a 20 minute episode of a conversation between us. So we’re going to try our best, we’re going to keep it relatively top level. We don’t want to drill too far into the detail, but yeah try and cover as much from our perspective of how we do it. It’s part of our job, we have to do this all the time and ultimately figure out how companies operate with analytics teams and some of them don’t right. When some companies are confused about how it becomes profitable for them, or what impact they can have, and other companies seem to invest huge amounts of money in them. So there must be something there and hopefully we can go through some of the reasons that they see value from it.

[00:01:54] Dara: Absolutely, let’s dig in.

[00:01:55] Daniel: For sure, well the first thing to note, it feels a bit like a cop-out right at the top, but the thing about a data team or an analytics team or an analyst full stop, is that ultimately what ends up happening is that the credit for your work gets absorbed by the functions that you’re working with. So let’s say we’re doing marketing analytics and we do a bunch of analysis and optimisations on some media campaigns that credit doesn’t get rewarded to the analysts or the data people that gets credited to the marketers. So the first thing that’s really hard to do when compared to something like a marketing team is that there’s no direct revenue associated to the cost that a analytics agency or headcount would actually be, or what we actually have to think about is what is the impact we’re having for those other teams where they’re seeing a positive impact and trying to, in a way, attribute back to the analytics people, the analysts to figure out if that was actually caused by them.

[00:02:46] Daniel: Analytics is one of those teams that has a foot in every camp. It’s got sort of like a finger in every pie, it’s involved in the marketing team, the product team, you know, the merchandisers, everything else. So when you’re an analyst or a business analyst or an analytics specialist, you’re kind of working across lots of different areas, all trying to help out wherever possible. And generally speaking, we don’t get a lot of credit.

[00:03:05] Dara: There’s a key distinction in there for me, which is that it’s a function that sits underneath or along the side, lots of other areas. And it’s not a marketing discipline, it’s not a marketing channel, it’s not an advertising channel. So the idea of having a very clear ROI, it’s not really relevant because it’s not, analytics isn’t an actual, it’s not a marketing channel in its own right. It’s supporting those activities and trying to look for opportunities to optimise or make better decisions through data rather than actually directly acquiring people into a website or an app or into a store. So, first and foremost, you could almost argue, it’s not quite a valid question. You’re looking to justify or measure the effectiveness of the analytics function rather than expecting to find a direct ROI.

[00:03:51] Daniel: I completely agree, but what if for some reason we need revenue against that line item, if we needed to justify a spend, an investment or even headcount potentially. So how about we start with some of the ways that we can measure a revenue or even associate or attribute some revenue to the analytics team.

[00:04:09] Dara: Yeah, absolutely. And I guess to be clear, there are benefits you can look for and then we can go through those, but to qualify my own point I guess, what I’m saying is it doesn’t quite work in the same way. So what we’re probably not going to do at the end of this and spoiler alert here, we’re not going to come up with a magic formula at the end of this to work out ROI for analytics. It’s more about probably providing some tips around where you can expect to see an impact and how you might look at measuring that impact.

[00:04:35] Daniel: It’s more like context and education, rather than trying to assume it works the same way as something else. It’s just kind of that awareness that this is slightly different and we should think about this slightly differently. Alright, so if we had to, gun to the head and we had to come up with a return on our analytics investment, so a company is paying us £10,000 to do some analytics for them. They need justification that they are going to see more than £10,000 back. Where would you start?

[00:05:01] Dara: My first thought is it obviously depends on where that investment in analytics is going to go because as you and I obviously know analytics isn’t a one-off thing, it’s an ongoing effort. So to kind of prove a return on an analytics project that would depend on what that project was. But let’s say it was trying to justify having an analytics expert in your team, in-house hiring that person. Then you might look at areas like, is there work leading to an increase in return on those marketing channels that we mentioned earlier, that analytics will kind of sit underneath. So if the recommendations, the insights coming from that analytics work lead to more efficient ad campaigns or marketing campaigns, then you could look to apportion some of that, whether it’s an incremental revenue or whether it’s a cost reduction, you could look to portion some of that to the analytics work that might lead to a bit of in-house fighting of course, if you’re trying to divide up the same cost saving between two different teams, but that would be my first thought is you would look at what the result of that analytics work is in terms of whether it’s driving a bigger ROI on marketing channels or whether it’s improving ROAS on your ad campaigns, or if it’s actually cutting dead weight. So if you’re reducing spend on poorly performing campaigns as a result of doing good analysis work based on good trustworthy data that you have because of that analytics function.

[00:06:22] Daniel: Yeah it’s interesting you said about the kind of taking credit for the work and there’ll be some conversations happening there. It’s almost like attribution, that kind of stuff we were talking about last week, wasn’t it?

[00:06:31] Dara: I knew you’d say that.

[00:06:32] Daniel: Yeah, it’s almost like attribution again, let’s assume we’re working on a last analytics person touch attribution model, right and trying to proportion or attribute as much value back to us. And you can say that, you know, by doing this new incrementality study or this attribution model that we built across the marketing channels, you’ve managed to make a change, which has improved your profit or your ROI or your ROAS by X then, you know, one could argue without us being there or this analyst being there, then that wouldn’t have happened. So it’s almost like that what’s the incremental value via whatever methods they choose, whether it be kind of like quality of the data or kind of modelling on top of the data. It almost doesn’t matter at that point, it’s just, without this, you wouldn’t have been able to do this and thus there is an attribution there.

[00:07:14] Daniel: But I completely agree it’s always the first step actually, especially when we talk about Google Analytics quite a lot, especially from our perspective. And that’s because it’s a marketing tool, right? And its primary focus is to measure marketing campaigns and attribute value to marketing channels. Obviously limited by lots of different factors we’ve been through loads of times. Like it’s digital only just has to go through a website not store, all of those things considered. I mean, that’s what we spend a lot of our time doing actually at Measurelab is marketing analytics and looking for those incremental gains across those ad channels.

[00:07:43] Dara: Yeah and just a point about what you mentioned about this is what we do, but the same on everything we’re going to talk about the same would apply whether you’re working with an analytics consultancy or whether you’ve got an in-house team. The same questions are probably going to arise because analytics is newer, relatively speaking. Nobody questions whether you need to have a finance team, if you’re a large organisation, or if you need to have an IT team, they’re all givens. Whereas when it comes to an analytics or data team, certainly for maybe less modern, less progressive companies, they would not necessarily see the justification to have that. So whether it’s working with a consultancy or whether it’s building an in-house team, the same questions probably apply. The second thing then so if we’re saying one area is looking at improvements made to marketing and advertising based on analytics, the natural extension of that is then looking at improvements to the website or the app themselves based on again, on analytics.

[00:08:35] Dara: So if you’re running AB testing or personalisation efforts, then that can be underpinned by good analytics data as well. And even though the people working in that area will be somewhat analytically minded, they have their own job to do so having somebody who’s that step removed to provide the analysis and the reporting based on any of those kind of website optimisation efforts, that’s another area where you can prove further incrementality.

[00:09:00] Daniel: Yeah, for sure. It’s almost like the analytics has to come first, especially when you’re talking about AB testing or CRO campaigns in general. Just because without the data you can’t measure the success of those and the whole purpose of the program in itself is to kind of get these improvements and test them, make sure they’re correct before you roll them out. So it’s almost like the analytics or the data, the quality, their trust in the data opens doors and enables you to do other things, to invest more in marketing or to invest more in CRO. But without that foundation, that solid foundation, you just can’t, you’re kind of blindly guessing, or just going based on gut right? Which is the opposite of what it’s all about really, at least from a data perspective.

[00:09:36] Daniel: I suppose, as an extension of that as well, you talked about the kind of incrementality from marketing and then the CRO side, but if you think of it from a product perspective as well, some things I suppose simple to consider is things like a SaaS product where, you know, a login portal, you log in, you use a product and then you leave right? And it’s just another, in a sense, another app or website, but the analytics there is going to be imperative to understand sort of product optimisations. So yeah, whether it’s product, marketing, optimisation, and sort of CRO optimisation, it doesn’t really matter because the data, the analytics, the sense of doing better, wherever possible, they’re just parameters within a function. I don’t care what I’m optimising for, whether its ad spend or usage in an app, it’s kind of one of the same thing.

[00:10:16] Dara: Yeah and we’re back to that point about analytics sitting underneath various different processes, teams, functions, because analytics itself can mean it could be focused on marketing, could be focused on product, or it could be focused on the customer and customer data. So customer insights and looking to improve, repeat purchase behaviour, looking to do cross-selling, looking to improve lifetime value. So some people might think of analytics as supporting marketing efforts, but actually it could be end to end. So it’s just looking for ways to improve either performance or the understanding of customers or looking for efficiencies in a process somewhere.

[00:10:52] Daniel: It’s all about incrementality at the end of the day, isn’t it? What we’re saying here is not that analytics gets all the value for all of these CRO programs and web optimisations or in product changes. The point is saying that there’s a baseline, without analytics and then once we introduce an analytics function or a team or an analyst that helps open doors in sense of decisions you can make or suggestions in terms of optimisations. The improvement that you see to the bottom line from those can be attributed back to the analyst or the analytics work. So if it turns out that you’ve increased conversion rate from 2% to 2.5%, because of the data that’s being collected. Let’s say that makes a £10,000 a month difference then you could associate that incrementality back to the analytics function. It’s crude, it’s very crude, you know, in terms of attribution, again, it’s just, I have had an impact, I’m going to claim all that revenue for myself. Ultimately, if you needed to kind of say, this is the overall impact or the maximum impact we can attribute to this function, this is how you do that.

[00:11:50] Daniel: You do the same with the change in ROAS, from a marketing perspective, you talk about the CRO conversion rate improvements from the CRO side. And, you know, in terms of the product optimisations or the kind of customer behaviour in terms of repeat purchase and customer loyalty, those can all be quite explicitly measured and ultimately measured back to revenue, or if you don’t deal in revenue, then it could be in terms of number of conversions. It can all be quantified, which is the purpose of this. So we can attribute value whether it’s revenue or good actions, you know, conversions back to an analytics function.

[00:12:23] Dara: Everything we’ve discussed so far has been well, mostly revenue or cost related. Although as you say, potentially you could be looking at signed up users or something like that, where it isn’t necessarily revenue. The other core area where you can look for value is around timesaving. So like a really well-oiled analytics function is going to save a lot of time within the business. Not even just within the function itself, but within the business, because it could be instrumental in automating reports, surfacing the right data to the right people, encouraging the organisation to make better decisions based on the data available. So there can be a huge time saving by having a solid analytics function, either within the business or working with an analytics professional.

[00:13:04] Daniel: Yeah, exactly. Automation is a huge path that if it takes you a day a week to put together the report, then that’s a day a week back, which is a huge saving from an employee or a team of employees. But also there’s stuff that’s maybe less obvious as well is things like trust, improving trust in the data, investing in a product, having people monitor and look at the quality of the data so that whenever someone gives you a number, you don’t second guess it, and think, oh, I need to double check that because I don’t really trust this number. So all of these second guesses, validations, these QA sessions that people might be doing, just because they don’t really trust the data is being saved as well, because in a sense you can put some trust or confidence in that person or that team to maintain this data, maintain the quality. So, yeah, automation for sure but also things like the confidence in the data to make those decisions, but trust is a huge part of that as well. The amount of times you’ll catch yourself doing it, I suppose like if someone sends you something, or you look at a dashboard and you’re like, ah, I should double check that before I do something with it, all of that will be shaved away in terms of the time, which is a huge saving.

[00:14:01] Dara: And that double checking as we know that’s not a quick process because double-checking in that context, usually involves going and doing something really tedious and really manual. And you use up a lot of time, and then you’ve got less confidence in the decision, even if you have double checked it. So if you were to add up all of that time wasted across the business, it would be significant. The other point I was going to add is that when you mentioned trust, the other aspect of that is impartiality, which is something we talk about quite a lot as well. So if you’re trusting the wrong team or the wrong source for that information, then again, you might not be making the right decisions. So having this as a, either a central function within the company, or working with an external provider means that you’re going to have an element of impartiality. So you’re going to have more trust in that data and in the recommendations that are being made as a result of that data, because they’re not coming from somebody who has a horse in the race.

[00:14:54] Daniel: Yeah, for sure. We see this all the time, a really good example is a media agency, right? If you say you’ve got an extra 5 grand to invest, and you go to your PPC media agency, they’ll probably find a way to invest that in PPC for sure. Which might get you a 2x return right. So, which is great, you know, you get a return on that investment, you see a slight bump, but if you’ve got someone central or independently external to kind of take the money and say, okay, well actually let’s put this in affiliate marketing or email or let’s do some radio ads or something, anything like that, but that will get you a 10x return, right. Which is a huge boost compared to the previous recommendation, like you said, is they’ve got a horse in the race they’ve got, I don’t want to say a bias, but it’s like, they are a specialist in that one area. And so if you have someone that’s central, someone that’s unbiased, that’s agnostic when it comes to this kind of stuff, that can make the best decision for the company rather than the best decision for their slice of the pie.

[00:15:47] Daniel: It makes perfect sense when you break it down, but it’s not going to someone that has a horse in the race, as you said, Dara. But having someone that doesn’t, and that independence is really key to having the best recommendations ultimately. I suppose you can think of this as a super fuzzy or loose way of attributing a return for analytics investment. And that is if you’re in a market that I suppose over time goes down or deteriorates or in case of like a membership system, natural churn. So if there’s like a natural churn or a natural decline in terms of a revenue using analytics to maintain a flat revenue structure is actually a increase, it’s actually a profit I suppose for the return on analytics.

[00:16:26] Daniel: So it’s not always about improving compared to last month, last year, last quarter, however your reporting. It’s not always about improving return or decreasing costs year on year, period on period. But it can also be about maintaining in a difficult environment or a new market that you’re opening up in or during COVID where lots of industries took a bit of a hit, but minimizing the impact that something might have in a business is very difficult to measure, and I think it almost might be anecdotal at best because you can’t measure what if the pandemic didn’t hit, what would we have done in that case? But from a statistical perspective, you can do forecasting models or predictive models and it kind of almost forecasts the data you would have got today from the historical data and measure what you’ve done today and use that as a comparison. So it’s very fuzzy, very loose, but it’s just to say that sometimes it’s not always about improving, it’s minimizing damage or damage control from a perspective of getting a return. It’s very hard, if impossible to measure, you definitely can’t AB test damage control, but it’s just something to be mindful of that you could be in a position where you’re working really, really hard, and you’re still seeing a loss year on year by 5%. But without that work, it would have been 50%, there’s just no way of knowing.

[00:17:37] Dara: Very true. I’m going to, as I tend to do, summarise in a very simple way, and I’m going to be a bit controversial, maybe. I’m going to say that, back to the question, how do you measure the return on analytics? Well, I’m going to say that you can’t really, at least not in the same way as you would with a marketing channel with an advertising campaign, because that’s not what it is. It’d be a bit like trying to calculate the return on investment of any other kind of non-marketing function in a business like the IT team. So for me, it’s more about trying to justify it as an essential function, rather than trying to prove a direct return. And for all the reasons we’ve mentioned it kind of crosses over with lots of other areas where it can be, it could play a supporting role in terms of making efficiencies or driving incremental revenue. But coming up with a very clear ROI, I am not convinced it’s possible or practical.

[00:18:26] Daniel: No so you won’t see a profit on the profit and loss sheet against analytics, but what you would, I suppose expect is to see a better profit margin for other investments that you might be making as an organisation. I suppose what you’re saying is you stop trying to prove an ROI for this line item like you wouldn’t for something like the IT team, assume that it’s going to have a wider impact across multiple functions and kind of use the incrementality over there to kind of help justify it I suppose.

[00:18:52] Dara: All right, what have you been doing? Impress us with your exciting life outside of work, Dan, what have you been up to, to wind down?

[00:18:59] Daniel: My exciting life outside of work. Well, it has been quite fun actually, the last weekend just gone, I had got some friends from the United States come over to visit from Seattle. So it’s a friend of mine that grew up where I’m from and he married a girl from America. So they’ve been living over there for the last sort of seven years or so, and occasionally they come back or we go out there. And so we hosted them this weekend and it was really good fun. I haven’t seen them in four or five years, so it’s been really nice to catch up. How about you Dara? What have you been up to?

[00:19:24] Dara: That sounds great by the way. Always nice catching up with old friends. I get a get out of jail free card this week. I don’t have to come up with something interesting I’ve done in my actual life because we had a work social last week, which you know, because you were at. But we do something called First Thursday Club, which is a real world get together on the first Thursday of every month. So last Thursday we met up, some of the team worked from the coworking space in London and then a bunch of us met up and we had a couple of drinks outside it was lovely, lovely weather, which was nice. And then we have pizza followed by cocktails in an insanely loud bar that I think none of us enjoyed. So after that, the few that were left, went and found somewhere a bit quieter for a bit of a chat, and then we all got home safely and sensibly. So it was a good night out.

[00:20:12] Daniel: As you said, I was there, it was great fun. The pizza was awesome, the drinks after were very loud, but I couldn’t tell if that was because of my age or the actual volume of the room. I couldn’t tell, but either way we had to find somewhere quiet.

[00:20:23] Dara: It was definitely, definitely loud in that room. It’s not a place to go for good conversations.

[00:20:29] Daniel: No, for sure, for sure.

[00:20:30] Dara: One final point to add before I do the usual wrap-up. We’re actually at MeasureCamp tomorrow at the time this podcast comes out. So if you’re at MeasureCamp, come and say hello, be nice to see you there. Where can people find out more about you Dan apart from coming and chatting to you at MeasureCamp?

[00:20:46] Daniel: I’ll skip the regular plugs this week and just say, come to MeasureCamp. If this is the first you’re hearing about it, unfortunately you probably can’t get a ticket, but hopefully there’s a couple of you that are listening that will be there tomorrow and that’s how you can find out more about me, come and ask me.

[00:20:59] Dara: Okay, that’s it from us for this week. As always, you can find out more about us and you can find all our previous episodes in our archive over at measurelab.co.uk/podcast. If you want to join us on the podcast, or if you want to just suggest a topic for us to discuss, then you can either reach out to Dan or myself or both of us on LinkedIn, or you can email us at podcast@measurelab.co.uk. Our theme music is from Confidential, links to their Spotify and Instagram are in the show notes.

[00:21:32] Dara: I’ve been Dara joined by Dan. So it’s a bye from me.

[00:21:35] Daniel: And bye from me.

[00:21:35] Dara: See you next time.