#12 Why impartiality is important in marketing analytics (with Mark Rochefort)
This week Dan and Dara have their first guest Mark Rochefort, Technology and Operations Director at Measurelab. They discuss why being impartial is vital in marketing analytics and the skills becoming more important to spot unconscious and conscious biases when working with data.
The article from Google announcing the new Google Analytics 360 is at https://bit.ly/3FIxJHf.
The documentation on the new Google Data Studio navigation updates is at https://bit.ly/3j0bvHf.
Mark talks a bit about the ‘Five Whys’ originally developed by Sakichi Toyoda and used within the Toyota Motor Corporation. Read more about the method at https://bit.ly/3BUtztP.
Dara refers to Measurelab’s company values, which he wrote an article about over at https://bit.ly/3BFplWN for more information.
In other news, Dan gets older, Dara has some new house guests and Mark stays in the woods!
Please leave a rating and review in the places one leaves ratings and reviews. If you want to join Dan and Dara on the podcast and talk about something in the analytics industry you have an opinion about (or just want to suggest a topic for them to chit-chat about), email podcast@measurelab.co.uk or find them on LinkedIn and drop them a message.
Transcript
[00:00:17] Dara: Hello and thanks for joining us in The Measure Pod, a podcast for analytics enthusiasts where each week we dive into some analytics topic, problem, or opinion, and we try to have a little bit of fun along the way. I’m Dara, MD at Measurelab. I’m joined as always by Measurelab’s longest serving analytics consultant, Dan. Hey Dan, what’s new in the analytics world?
[00:00:37] Dan: So this week, two things to talk about. First of all, Google formally announced the new Google Analytics 360 contracts. Or at least they formally announced the fact that Google Analytics 360 is being updated within the context of Google Analytics 4. So this has a lot of, well, this makes me always think about Nintendo when they released the new version of the 3DS. They’re calling it the new Google Analytics 360, they don’t want us to call it GA4 360. There’s a couple of announcements about some of the differences there are. There’s a support page as well if you’re on the beta, which you can sign up for by the way of GA360 for GA4 properties. You can sign up for the beta if you’re already GA360 customer. I’ll link off to the announcement, you can have a read through yourself about a couple of things is just the news around sub properties and roll up properties are going to be a 360 only feature. And there’s some limits that have been upped in the new version. And the second one is Google Data Studio had a bit of a small update, but actually was a big one in terms of everyone’s experience of using the product. They’ve now improved the navigation of Data Studio, and you can now use icons, you can nest pages together and you can really categorize tabs however you want. And it’s just such an, uh, welcome update to a product where we’ve been fudging navigations through hyperlinks and trying to hide all the pages and try and make a prettier version of this. We’re starting to have a play with it, we’re already starting to use it for some of the dashboards we’re creating for our clients and for ourselves. I’d like to see where this goes and how far they take this. I like the fact that we can start using icons and using the nav as intended without having to shy away from it.
[00:02:06] Dara: It takes all the fun out of it Dan. I’ll miss the pain of having to do all those hacks and workarounds to get it to work. I won’t really, very welcome updates. So this week, we are joined by our very first guest. And not just any guest, but the one and only Mark Rochefort. And Dan, I’m going to give you the fun or not so fun task of introducing Mark.
[00:02:30] Dan: Thanks Dara. All Right, Mark started back in the nineties in this industry is an interactive developer and he’s now our Technology and Operations Director at Measurelab. He’s joined us today to talk about why impartiality is important in analytics. But first Mark, I’ve got to ask. What is an interactive developer?
[00:02:48] Mark: Great question Dan. It’s an honour to be here, being your first guest makes me slightly nervous. I’ll do my best and I’m sure you’ll splice my words together and make me sound like the experts at the end of this anyway. So an interactive developer really harks back to the days of CD ROMs, where fancy user interfaces were coming into play. And from there I became a web developer and everything else, but I think the important thing that has remained throughout all of this for me is understanding how people use stuff. And it’s been fascinating to think about that as developer and then later on in life. When we’re collecting data on products or marketing campaigns or whatever else, it’s all about understanding how people are using things and been getting to the end point.
[00:03:32] Dara: So it’s great to have you on the show Mark, that background and the kind of curiosity that you were talking about in terms of understanding how things work and solving problems for clients. That’s relevant to what we’re going to be talking about today. So our topic is why impartiality is important in marketing analytics. And I think you’ve got a lot to say about this
[00:03:51] Mark: Indeed. so it goes without saying that we’re an independent analytics consultancy. So always our angle is going to be that we’re going to help people take that impartial view of their analytics. And I think that’s an important thing for any organization, even if you’re working internally, to remain impartial about the work you’re doing, and that work is either as an analyst or even as a developer, an implementer of tracking or whatever. On the analyst side of things, it’s very easy to get into mindset of marking your own homework. You might be coerced in some way to make a particular channel look better than another because that’s where your focus is. Or if you’re more on the development side of things, you might be caught into a mindset which means because you’ve got a hammer, every problem is a nail.
[00:04:42] Dan: It’s one of my favorite sayings as well is not marking your own homework. And, as you’ve just said, it’s something I’ve not really thought about before, but applying that client side, whether you’re the dev or whether you’re an analyst. You, you have a skew, you built something, you want it to be successful. You might be a product owner, in which case it might be your part of the product and you want it to look good. But the same can be said for marketing agencies. And if they are running maybe one or two of the five or six channels for you as a client, then they also have a skew that they would want to, you know, prove their worth. And this isn’t in any kind of insidious kind of, uh, undertones to this of them having to force you to spend more money and, or whatever that might look like. It’s just they have a way of thinking, and quite rightly so, where they’re trying their hardest to make this work, they want it to be successful. I would love to ask you a question Mark, and to you actually Dara. So, uh, if anyone that doesn’t know, Mark and Dara are my bosses, so I’m on my best behavior today. And they’re the two co-founders of Measurelab, which as Dara said last week, just has its 8th birthday. So you both are eight years into this venture, into this idea of this independent analytics consultancy. So tell us a bit about that. How did you get into that idea of thinking this is an important thing to have and to start something up? Which I’m very grateful for by the way, thank you for my job and I’d like to keep it.
[00:05:53] Mark: What motivated us to start? I think we felt we could do things a little bit differently. We both came from marketing agencies. We’ve come from that marketing world. I’ve got more of a technology background, Dara more on the analysis side of things. But it’s always been very much the marketing agency side, which I guess pulls into the impartial aspect really. I think there’s a lot of cases we saw where we were literally marking our own homework. We therefore saw a need to be creating a company that offered that as a, as a service. And of course he wants to have some fun along the way.
[00:06:27] Dan: How about you Dara?
[00:06:28] Dara: I think the, I mean, obviously. I’m going to echo a lot of what mark said. I think the impartiality for me maybe came more as a, that became like a USP as opposed to the motivating factor. I think we are motivations for setting up Measurelab were initially more about creating an environment that we would want to work in. So I talked before on the show about our values and I think we’ve tried to live those values from day one. So we tried to create a company where we would want to work. Part of that then is the impartiality, because integrity is one of our values and we want to make sure that we are helping our clients to make the right decisions. And we’re working with people at the end of the day. So we form quite close relationships with our contacts, with our clients, and we want to help them do the right thing. So we want to help them make the right decisions. And it’s always been a kind of natural way to do that being impartial, because we don’t have any vested interest in where they’re spending their budgets. So we’re helping them look at that data objectively. What we’re trying to do is enable our clients to do the same thing. So even if it is internally, it’s very difficult to do something and evaluate that thing that you’re doing yourself. If the same person or the same people are doing the thing and evaluating it, there’s a natural conflict between the two. So whether it’s using an external consultancy or whether it’s a team internally, it’s trying to have that objectivity. It’s having somebody who isn’t as invested in the activity itself to have a look at it with fresh eyes and say, actually, what you think this is saying is not quite the case. And you can only really do that when you’re not so tied up in, in the act of doing the thing, whether it’s a marketing campaign, building a website or an app, whatever.
[00:08:11] Dan: To play devil’s advocate for a second, the marketing analytics world is full of tools and quite a lot of those tools are in essence not impartial in any way. The one we use day-to-day in most cases is Google Analytics. So Google analytics is owned by one of the biggest advertising platforms in the world, Google. And they have a large focus towards integrating it into their marketing products and not into other non Google marketing products. I suppose, just go back to the devil’s advocate, how can you remain agnostic or impartial when the tools that you might be using aren’t?
[00:08:43] Mark: I think it’s important to have an impartial mindset whether you are working internally or externally to create for yourself that objectivity. There’s ways to do that with organizational structures, we’ve seen clients create centers of excellence, hubs internally if they’re a large organization. If you’re a smaller organization, that’s obviously harder. But creating a mindset that facilitates this impartiality is important. One trick that we’ve seen used is the five whys, just asking yourself why. It comes from Toyota production systems, which we all know and love as being the originators of Kanban. And the whole Kaizen iterative improvement approaches. But it’s part of their original systems developed, gosh, maybe more than a hundred years ago, to ask why and why and why and why and why, is that five times? Enough so you start to get into the problem. There’s all sorts of frameworks around this there’ve been developed for the point is to keep pulling back, pulling back, pulling back from the question to create that impartiality or objectivity around the problem you’re trying to solve.
[00:09:56] Dan: What I hear then it’s, it’s about the people. So it’s actually not about the tools. And I suppose you can mitigate some of that just by knowing the tool inside and out and knowing where the impartiality and partiality lies so that you are very aware of where you are being fed bias or unbiased data. But then being in an active, impartial mindset and always questioning with those five whys to kind of retain that. So am I right in saying that then it’s about the, kind of the human aspect of it, the ingredient of the human, this isn’t something that you can program into machines. Cause there’s a lots of talk around machine learnings taken over, our jobs are going to be redundant. You know, we’re not going to do be doing this in five, 10 years. Do you see that happening with this kind of stuff? Do you think they’ll never be able to replicate impartiality? Is that something that even when you would trust someone like Google to build? Is interesting, I don’t know.
[00:10:46] Mark: Again, that’s the objective viewpoint to solving problems that isn’t just going to be a black box that you throw stuff at. Yes, things are going to become automated. Yes, things are going to become easier. But there’s going to be more and more things to be solved and things to be done with the back of that, which creates more opportunities, more jobs. The roles of an analyst and developer are converging. There are new job titles being created every minute of the day it seems. I don’t think the term analytics engineer was around when we first started out, it’s now very much the direction things are going. There’s a need for people to be coming to the job with the skills that are wide ranging in scope to give that objectivity and problem solving, and also the impartiality to not be reliant on the machine or the black box that you’re feeding. I guess on that, the things that I see that are interesting there is having a broad knowledge of the tools that are available to solve a problem. More and more data is being collected, it’s all in the cloud now. People are creating pipelines, triggers from this to this to this, things are being automated. But you need people to maintain that. You need people to watch over that. That’s never going to go away. We’ve got all sorts of ever increasingly complex marketing technology stacks. We’ve got data warehousing, we’re now getting reverse ETLs back into other stacks left right and center. Therefore, to remain impartial to those technologies, and see the bigger picture and see how things connect is going to become, or well is increasingly more important.
[00:12:18] Dara: So it’s all about critical thinking so that the technology, everyone will have their preferred software to use their preferred analytics tool, database, whatever. I think at least it’s less about what technology is chosen to do the job, and more about how that’s applied. So it’s a need for critical thinking within the business, whether that’s internally or externally, somebody or some people to look at the problem, or objectively to think, is this really working. Rather than getting stuck in a kind of bias situation where you’re trying to prove your own conclusion. So it’s the role of, we might call them a consultant, but internally it could be an analyst or it could be somebody who’s overseeing the use of data. It could be a Chief Data Officer, whatever, depending on the type of organization. But somebody ultimately needs to be able to look at whether it’s the performance of the business or the campaigns, or whether it’s a problem that’s being looked to be solved, it requires critical thinking. And it’s less about which solution or which technology is used and more about how that’s being used.
[00:13:23] Dan: Those soft skills not so soft anymore right? They’re no longer the secondary or the softer skills.
[00:13:29] Dara: No. And they’re also, um, soft can wrongly imply that they’re they’re easy and they’re not. And this is the kind of changing role I think. If you think about impartiality internally, it’s maybe even more difficult. If you’re an external consultancy and you’re brought in and paid to give your point of view, typically that’s listened to because it’s being paid for. Not always the case, but that’s usually what happens. If you’re somebody internally and that’s your responsibility, it can be very difficult so you need to be quite a, quite a strong negotiator, quite a strong communicator, you’ll probably need to be thick-skinned, because you’ll have to battle against some of these biases. There might be internal pressures to do things a certain way, or to continue investing in a certain area or to increase investments in a certain area. And you have to be able to, um, as much as possible, keep yourself out of that and just stick to the objective view of, of what the data is telling you. And that’s really not an easy thing to do.
[00:14:27] Dan: It’s really interesting when analytics is quite often, in the early stages of a client’s maturity, starts off as a marketing asset rather than an independent team or an independent entity. And so, as a subject matter evolving from the marketing team, and quite often when people come and work with external help is when they don’t have time or the knowledge or the know how will the, all the attention it deserves to really make the most of it. So when we work with a lot of our clients, a lot of them are the marketing teams. It’s really interesting because I can relate in lots of instances where the biases that they might have We mentioned before around the biases that an internal marketer or a developer will have almost gets translate to us in a way. Because we are their asset, our cost, our time is under their budget line item, for example. So it’s really, it’s really hard, uh, really interesting and very important, you know, to still maintain a sense of that, but also know when to back down is maybe the wrong way of saying it, But when to compromise and it’s not about compromising in our ethics or anything like that, if 90% of the work that we’re doing is analyzing the paid search channel, the nature of that means we’re not going to be giving as much attention to other marketing channels. And we’re still being impartial with the analysis, but just doing more analysis on one channel in itself is, is, biased, right? Because we’re going to be finding things in there that we’re not going to be finding elsewhere. Maybe there’s no point in there, but was just really interesting as you’re talking about that, that, that actually there is bias at every level and it can be inherited too. And there’s very few instances I can think, of only a handful of clients in my career at least. Where, and this is generally where they have an analytics team in a way acting like us, their side. So they have their own internal analytics team that acts like an analytics consultancy within the organization. And then we are an extension of that team there. So, I think it’s actually really interesting in terms of the it’s all down to that maturity around data.
[00:16:23] Mark: I think that being aware that that unconscious bias exists is almost all you can do, because by its very nature, it’s unconscious and you can’t necessarily remove it. It’s going to exist in everything we’re doing in fact. But, um, decisions are being made that are based on systems, organizational structure, history, processes that exist around the implementation, the approach, the choice of tools. Just being aware of that and being objective or impartial as best we can is important.
[00:17:02] Dara: And we are seeing that shift, we have seen that shift. So analytics, as we sometimes tend to think of it as web and app analytics. Was a function of the marketing team, the marketing department. And over the years, we have seen that blurring of the lines between marketing data and the broader business data. It is, I think, as you said Dan, it’s about the analytics maturity of the business. If someone’s in a position where they can have a, you know, Mark mentioned analytics center of excellence. Or even if you just got to a data team, a cross functional data team, then you take some of that bias, that risk of bias away because you let the people who are running the marketing, they get on with doing that and then data is seen as a kind of cross-functional thing, rather than just being something related to marketing or sales or, or whatever. Um, and we have seen that over over the years that we’ve been working in this space, where there’s that ever increasing blurring of the lines between web and app analytics and the kind of business intelligence and broader, um, data analytics within a business.
[00:18:13] Mark: Everybody talks about needing a single source of truth. And I know you’ve covered this before in that it’s a view on the truth. The truth is never going to exist, and the data that we are collecting, analyzing for various reasons, we want to avoid that becoming a soup of opinion where people are embroiled in all sorts of views on the why’s and the wherefores of the data and so on. To pull back and remain impartial about that single source of truth, even acknowledging that isn’t going to be true.
[00:18:49] Dan: And those are those unconscious biases, right. And maybe sometimes conscious too. I think it’s all about, from what you were saying, it’s all about just awareness and the ability to spots, biases or unconscious biases or how to handle those. I think there’s a bit there as well, which is about storytelling, right? And probably one of the many job titles that I’ve seen come up and go over the years as well, data storytelling. Which I think is actually tied in here because when you were doing analyses, whatever that happens to be on, there’s a way of telling that story which, just because the numbers aren’t what you expect doesn’t make it a failure. And I think there’s a way of having that ability to zoom out saying, okay, well it didn’t succeed in these two KPIs that were specified, no. But actually there was a longer tail that we can talk about over here. Or maybe there’s opportunities for optimization or, it’s not spinning a story as such, but it’s just identifying all of the avenues that can be beneficial to learn from all the learnings that we can take from something. So it’s not as black and white as this didn’t work, we’re unbiased and it didn’t work. The agency didn’t do well enough on the last campaign they run for you, that was never going to be the case. It’s more around that there’s stuff there that this kind of thing did work. We can see this had a longer tail. Maybe there’s an opportunity to tweak this next time or run an AB test, or split test your audience next time. Identifying these educational elements for something, the way that you provide that analysis back, the way that you, you do the analysis, even. So it’s about again, impartiality for sure, but it’s not always impartial so that we can tell you where things are rubbish, right? That’s not, that’s not the purpose of it. It’s actually being able to tell that story.
[00:20:25] Dara: That’s a really good point and we do need to remind ourselves. This links back a little bit to last week’s topic around fuzzy data. In the industry, we can get a bit carried away with thinking that there is this holy grail of the truth and the data doesn’t lie. And in reality, there’s always going to be an element of subjectivity to it. It’s an interpretation of the data. So as a kind of counter warning, so impartiality is hugely important. Objectivity, critical thinking all the rest of it, nobody would really argue with that. But we shouldn’t take that to the extreme of thinking that we’re talking about facts, because there aren’t really any facts and in marketing or running a business or growing a business. You look at the data as objectively as you can, and that data could be information. It could be opinion, it doesn’t have to be a ones and zeros. You take the information in, you review it as objectively as you can, being as aware as you can be of the biases. And then you make the best decision you can, but you test it. And then you learn from those experiments and you stick with the things that that quote unquote work, and you drop the things that quote unquote don’t work, or you dig into them further and you try and better understand them and then run more experiments. So it’s a subjective interpretation of the of the data really, rather than getting too black and white about it and thinking, well, you know, if I think the data says this, then it must be a fact because there will be lots of contexts that won’t be included in that data
[00:21:49] Dan: We’re steering into the CRO world slightly with this but I’d even say, test the successes equally. Because what’s successful today is not going to be successful next time you run your campaign or launch a new product, for example. And everything’s always in flux. One of my biggest pet peeves, I’m not saying you do this Dara of course, but it’s when I see analyses and it’s like, this worked 10 years ago so we’ve continued to do it since. And this is where we see people stuck in their ways like spending X million pound on direct mail every year because 10 years ago it was perceived to do well. And I think with all this, things change, technology changes as you were talking about Mark, with the technology kind of systemizing and automating lots of things. What doesn’t change is our curiosity and our ability to question or why, why, why, why, why the five whys as you mentioned Mark. And taking that back and saying, well, would it still work today?
[00:22:38] Dara: As humans we’re programmed to look for shortcuts. So If we saw a piece of analysis, if we saw a summary of some data that was looked at before, and there was an outcome. We will naturally try and assume that outcome and still true today, because that’s a quick way. You can just, you can just link back to it, your brain takes a shortcut, you think that must still be true today. And that’s why I think it’s important to get these processes in place, or it is working towards having a different unit or team within a business that’s looking at these things. So something that was true, you’re absolutely right, something that was true yet. Something, it was true when we started recording this podcast, might not be true now. We can’t rely on ourselves as humans to be objective all of the time, because we’re not objective creatures.
[00:23:20] Mark: Well I guess this comes back to the people thing again, isn’t it. We often are dealing with systems and things that we consider to be infallible, when in fact the data; one is not always true and we need to be mindful of that, and also as people, we should be maintaining a sense of impartiality, a sense of objectivity so not to get caught up in, in the view of the data that we may be seeing.
[00:23:54] Dara: Okay, this point in the show Mark, we usually do our little wind down. So we talk about what we’ve been doing outside of work to switch off and recharge. So as our first honoree guest, going to give you the the privilege of going first.
[00:24:11] Mark: Gosh, what a privilege it is. Well, as it happens last weekend, my family and I went and stayed in a hut in the countryside and it was lovely. The weather was lovely, there was no phone reception. The place that only had solar power, so basically devices were out for of us. The kids unplugged, my kids aren’t robots, my kids’ devices were unplugged. And yeah, it was, it was really, really nice. We, we, properly, properly relaxed in the countryside and got away from everything.
[00:24:43] Dara: Very wholesome. What about you, Dan?
[00:24:46] Dan: Well for me, it’s nothing too exciting, but I had a birthday. My wife and I took the day off and spend it together and went out mooching around the seaside. So not super exciting, but a really nice break from work, having the day off just using that to mooch about and just completely switch off. I couldn’t ask for anything better, to be honest. I’m not a huge birthday person, and just spending that time with my wife was, uh, was all I needed. So how about you Dara, what have you been up to in the last couple of days to switch off?
[00:25:17] Dara: Well, people who know me know that I like to take in animals that need a home. I have dogs, I have a cat I’ve had chickens. Um, and this last weekend I took in two little guys, not so little that need a new home. They’re tarantulas. Yeah, most people think it’s a bit weird. I think it’s no different to, anything else. So yeah, two new house guests, Billy and Derek the tarantulas joined us at the weekend on our settling in nicely.
[00:25:50] Dan: The names of very disarming.
[00:25:53] Dara: Don’t be fooled by The names. That’s it for this week. As ever, you can find out more about us over at measurelab.co.uk, or you can get in touch via email at our new email address podcast@measurelab.co.uk or of course you can find us on LinkedIn. And please do get in touch if you’ve got any questions, or better still, if you have an opinion about something. Related to analytics and you want to come on the show to discuss it, please get in touch and let us know. Otherwise that’s it for us, join us next time for more analytics chit-chat. I’ve been Dara, joined by Dan and Mark. So it’s bye for me.
[00:26:31] Dan: Bye from me.
[00:26:32] Mark: And bye from me. It wasn’t as scary as I thought.
[00:26:35] Dara: See you next time.
Daniel Perry-Reed
Subscribe to our newsletter:
Further reading
What is Offline Event Data Import in GA4?
The complexity paradox: why your spreadsheet is complex and BigQuery is simple
