#14 How COVID-19 impacts data analysis (with Jake Birch)
This week Dan and Dara are joined by Jake Birch. Jake is a Lead Analytics Consultant at Measurelab and has come on the pod to talk about how COVID-19 has affected how he has been analysing data for his clients over the last 18+ months.
See the details on the Google Analytics for Firebase updates at https://bit.ly/3EnIAov.
Check out all of the additional ecommerce item-based fields now in the GA4 Data API at https://bit.ly/3jM1q0W.
Find the announcement on data-driven attribution rolling out to all GA4 properties at https://bit.ly/3ErJz7s.
In other news, Dan goes walking, Dara adopts a snake and Jake goes abroad!
Please leave a rating and review in the places one leaves ratings and reviews. If you want to join Dan and Dara on the podcast and talk about something in the analytics industry you have an opinion about (or just want to suggest a topic for them to chit-chat about), email podcast@measurelab.co.uk or find them on LinkedIn and drop them a message.
Transcript
[00:00:17] Dara: Hello, and thanks for joining us in The Measure Pod. A podcast for analytics enthusiasts where each week we dive into an analytics topic, a challenge or an opinion, and we try and have a little bit of fun along the way. I’m Dara, I’m MD at Measurelab. I’m joined as always by Dan, one of our Lead Analytics Consultants. Hey Dan, what’s new in the world of analytics?
[00:00:39] Dan: So a couple of things to report on this week. And the first one is that the analytics section of Firebase is being finally brought in line with that with GA4. So there’s always been a big question what’s the difference between GA for Firebase and GA4, and there is no difference it is one platform. But the UI has always looked different, and so we always thought of them as two things. That’s not the only thing this week. We’ve also got news that GA4 Data API has now got the e-commerce dimensions and metrics added in. So the items within the tracking is now available to use and things like Data Studio which is awesome. And lastly, we’ve been knowing this is coming for a little while now, but we heard more of an announcement specifically around this from GA this week. And that is the data-driven attribution is being rolled out to GA4 properties. I can’t see it everywhere for all my clients just yet, but it is now being rolled out and we’re going to start to see data-driven attribution in the advertising workspace for our GA4 properties, which is going to be exciting.
[00:01:39] Dara: Definitely I’m the most excited about the e-com dimensions and metrics rolling out, but maybe that’s just me.
[00:01:45] Dan: Yeah, it’s about time we get this item level data and the API is just been kind of falling a bit short on that front.
[00:01:52] Dara: Okay, so this week we’re joined by Jacob Birch. So Jake has been in analytics for eight years starting as a web analyst for the Friday Media Group, but now he’s with us at Measurelab. He’s a Lead Analytics Consultant, he’s been here for about a year and a half. So Jake, why have you come on the show? What’s your opinion, or what’s your topic that you want to discuss on here with us today?
[00:02:14] Jake: Hi Dara, thanks for having me, it’s an honor to be on the podcast. I wanted to talk about the lovely subject of COVID and how it’s affected our analysis and reporting over the last 18 months, and how we’ve been using data and what our clients have seen.
[00:02:30] Dara: Yeah this is really interesting, there’s a lot of trends I think we’ve noticed and it’s pretty wide ranging isn’t it. Everything from how people are reporting on performance, right the way through to kind of changes in behaviors for customers.
[00:02:42] Jake: Yeah, and different industries asking different questions, and they see totally different trends as well. But I think that’s the fun of having loads of different clients. We sort of see completely different changes throughout the 18 months.
[00:02:55] Dara: So let’s get into it then Jake. Open you really, what are some of the changes or what are some of the trends that you’ve seen in terms of how COVID has affected the businesses we’re working with in terms of their analytics data?
[00:03:06] Jake: So I think the first thing is keeping a close eye on how things are changing and when they’re changing. One of the first things we did for a few of our clients was build a dashboard and it had a small little glossary of significant dates of all the different changes. So if you remember way back in March 2020, there were quite a few small, significant changes. People didn’t really know what was happening at the time, but they started to roll it out a few different things. So one day the government was saying we really want people to start working from home. And then, I think a week later, there was announcements that the schools are shutting. And then after that there was a complete lockdown. So there was massive changes in behavior for each of those key dates, but it was really important for our clients to notice exactly when the changes were happening. For example, the working from home being encouraged saw a lot of office furniture shoot up and start selling for one of our clients who sells loads of furniture. It’s really interesting to see sort of like changes in trends in specific categories based on those dates and just making sure that those were aligned in a dashboard.
[00:04:15] Dara: Of course the changes weren’t all good or all bad. Some businesses obviously had a benefit in terms of those changes, as you said, with the office furniture. I was one of those people, so I had no desk and no office chair at home pre COVID. So when I worked from home, I would work on the sofa, which is probably not the best thing to do. I needed to urgently get a desk and a chair. Um, then other businesses, obviously it affected them negatively. So you’re right, having those key dates plotted out on a dashboard provides a lot of context around why the data might’ve been pointing in a direction, either up or down at different points in time during the changing restriction.
[00:04:55] Dan: So I have a question, Jake. This must be regional specific, right? Have you seen where you’ve got international retailers that this kind of blurs into lots of different days with lots of different regulations changing? Because I can imagine that this goes beyond just a GA annotation, this isn’t just an annotation in one day, there’s going to be nuance there right?
[00:05:15] Jake: So yeah, when we’re looking at regions that aren’t the UK, say for example APAC which involves Japan, Australia, Taiwan, Singapore. They had very different dates for lockdown, so it was really important to make sure we annotated in a glossary the dates between those countries. And then we saw completely different shifts between those countries at different points. And it’s also worth noting that within these countries as well, there were completely different lockdowns within cities. If you’re looking at things like retailers in specific cities and looking at sales data there, it’s really important to make sure you’re looking at a city or a county level to see how that’s affected data as well. There wasn’t just one set of dates for the whole world, obviously every country is completely different.
[00:06:03] Dan: mentioned Data Studio there and I suppose an open question, but how does that materialize, how do you start reporting on a regional city county level, on a region like APAC and pull all that together. In this example, how does that client look at that data? It feels like quite a micro level reporting where a lot of people are quite used to the macro.
[00:06:23] Jake: When we were looking at these regions that did have different dates with different countries, we did just continue to have this roll up figure. But having the different dates where these countries went into lockdown, doing our analysis behind the scenes that really helps us to tell the story as to why we were seeing different shifts. So for the clients that I had, we didn’t change the reporting view massively. But it just really helps our analysis when we were pushing those insights across.
[00:06:51] Dara: Jake, I’m going to bring up the big headache. So it’s obviously had a massive effect on year on year reporting, which is what most businesses like to use. What’s your experience been of how it’s actually affected those kinds of comparisons?
[00:07:06] Jake: Yeah, so when COVID first hit, it was still really useful to do those year on year analysis. And we were looking at what the year and year increases or decreases were before COVID hit and then comparing that to after COVID and seeing sort of how much of an impact it’s had. So for some of our clients, we saw that the year and year growth suddenly increased for that time of year because of certain products that were selling. But we kind of changed our approach in terms of, we weren’t just looking year on year, but we were monitoring the shorter term changes a lot more. So where I mentioned these dates, we were looking at it more closely in terms of like month on month and period on period. Because everything was changing so quickly and quite drastically, it was really important to look at this shorter term changes and just keeping an eye on how things were changing due to these announcements. And we did start to look back two years as well. When we look at 2021, now we can compare to 2019 and 2018. But I think it’s still important to look at the year and near growth compared to 2020.
[00:08:13] Dan: I suppose. There’s, there’s been a year of disruption in terms of the reporting. So you said about going from classic year on year reporting as a staple, and then the pandemic hits and then we’re looking a bit more reactory and we’re looking at month to month, week on week, even. I suppose, the term that we all kind of have come to hate, which is the new normal, trying to assess the new normal. But now, speaking from the UK at least, the legislation and the lockdowns have come, so far at the time of recording, to an end. And I think now we’re at a point where we can look back. What you’re saying is back two years, almost pretend 2020 didn’t happen, and looking at the year and year now kind of skips a year backwards. Is there a way. Is there a way to kind of handle that year? Rather than almost ignoring it and working around it, is there a way to almost like compare it to, I suppose, at what I’m trying to ask is if you were to kind of like quantify the impact of the pandemic, is there a way to do that?
[00:09:04] Jake: One of the things we did was forecast some of the data from COVID onwards. So we wanted to figure out what we were to expect if COVID didn’t happen. And then we would compare those months against the forecast. So that would help us kind of quantify if we were on track. So we were using forecasts in quite a lot of our clients reporting.
[00:09:26] Dara: What about changes in consumer behavior? Have you done much around trying to understand, maybe the change in buying behavior as a result of the impact that COVID has had on people in terms of maybe their expendable income or even the types of purchases they’re making. Beyond just the initial change of home office furniture and technology may be seeing an uptick due to people working from home.
[00:09:52] Jake: Yeah, so it was really important for us to think about how people have been affected by this pandemic. So one of the things that we thought about was people’s income and how that would affect their affordability. Something we actually looked at was, we took all of our customers from one of our clients and we grouped them into affordability segments. So people that were spending a certain amount within a few months, how that had changed over time through the pandemic. Had certain customers moved from one affordability segment to a level below, maybe perhaps because their income had changed. Or where certain people were spending a lot more now. And Yeah, that really helped the clients to tailor the marketing based on what we’d seen, in terms of changes in affordability and those audience segments.
[00:10:39] Dara: Yeah, because I could imagine it would have an impact on repeat purchase behavior or lifetime value. You know, it’s not just the actual kind of traffic and conversions, it’s that longer term effect.
[00:10:50] Dan: This also touches on like the people shopping as well. I mean, I know for a fact, and this isn’t representative sample, but my folks, they are now shopping online for almost everything. And it was a necessity during lockdown, but now they’ve kind of realized what the hype’s about. But for them to shop online was almost unheard of before lockdown, and now doing groceries online is commonplace in their lives. And this is opening up those doors in terms of not just affordability, but new users and new demographics, I suppose, to be able to tailor that marketing to.
[00:11:25] Jake: Yeah, we’ve definitely been looking at and monitoring the demographic reporting for purchases online, and seeing those shifts in changes is quite different between markets. There’s definitely shifts within industries which is really interesting, and the message there is just to make sure you’re continuing to refresh your audience segment data. Cause it’s really important to keep an eye on that And it’s not going to be the same as it was two years ago. Most likely will be very different than then.
[00:11:56] Dan: And even completely different to what it was last year as well. Right? So there’s that middle, that middle year, that year of lockdown, that kind of throws a spanner in the works and it’s different before it’s different during, and it’s probably going to be different again, after once things have settled down as that’s really good advice Jake, thanks.
[00:12:11] Dara: You mentioned offline that data earlier. Have you got any observations or have you noticed anything interesting around changes in reporting or quality of data, or how data is being used from the real world?
[00:12:26] Jake: Yeah, for one of our clients we do actually look at sales in retailers. So one thing that was really interesting there is we analyze AB test results for all these retailers. So for example, they take half the stores and they’ll do certain promotions and big cardboard cutouts and stuff like that in half of the stores. But then in the other half the stores, they have the control group where they don’t do anything. And then it’s our job to look at that data while those AB tests are running, and work out all the statistical significances and things like that based on the data. But the problem there was that with the countries where they were testing, there were different lockdown rules within different regions within the country, in different cities. So when that started happening, we saw that actually some of our test groups were in the more locked down areas, which we’re only allowing a certain number of people in the stores at one time. But then, some of the control stores were in the less locked down areas where they were allowing loads of people in the store. So it wasn’t really fair to actually compare these tests and control groups cause they had completely different rules and behaviors within those stores. That became really difficult for us to analyze that data.
[00:13:43] Dan: So were you able to analyze the results of those AB tests, or was the outcome that they are not a true split test and you have to redefine the test itself to then analyze? I’m just wondering what, you know, when they asked you to analyze the A’s and the B’s of this experiment, and then you look at it and go it’s apples and oranges, it doesn’t make sense. Um, what happens next?
[00:14:03] Jake: So we weren’t able to quite come up with a confident analysis on that based on how skew the data was. And we just weren’t really confident in terms of coming up with a result because the data wasn’t really fair. We have put forward suggestions in terms of how to segment those that we can then do some fair and confident testing. They haven’t quite been put into place yet, and they’ve kind of put the AB testing on pause because the lockdowns are changing quite a bit in those countries that we’re looking at. And because it takes them about six months to actually plan these tests, by the time the tests come around, the stores could be completely different in terms of what the rules are within those stores.
[00:14:45] Dan: Suppose it makes me think about AB testing in general and how almost lucky the digital side has it, where it doesn’t take six months to roll out. We can give them a slightly unique experiences as people coming to the shop, which is a luxury that the bricks and mortar side doesn’t have. But I’m just wondering now, maybe you got a clear winner from an experiment run just before lockdown to change a layout for a checkout page, for example. They go into lockdown and all the user behavior changes, and all of the experimentations we might have done just before or even during these lockdown periods. I’m just wondering how relevant it is to retest all of the tests that we might’ve just done and got a clear answer on, because we’re now no longer confident that it’s the same outcome because we just so unsure of user behavior and preferences now.
[00:15:30] Jake: Yeah, I think with stuffed like AB testing it’s always important to continually test the same thing. You don’t just test it once and then leave it for 10 years. Um, but yeah, it’s more important than ever now to continually test those because behavior is changing more rapidly now.
[00:15:47] Dara: Okay, so to wrap things up, that’s been really interesting Jake. And I think that’s all real firsthand information about how you’ve seen clients change their reporting, or maybe the focus of their analysis as a result of COVID and the restrictions that came from COVID. I guess COVID is not the only thing that can have this effect though. Things like the financial crash, or even just changes within an industry or, or for particular business itself can have a lot of similar effects. So maybe, tell me if you agree or disagree with this, but the point is all of this should constantly be up for evolution anyway. You can never really get complacent and think you found that perfect formula for reporting. It doesn’t just take COVID to knock your data out of whack, and really whether it’s testing or analyzing, you should be continually evolving that and try to make the decisions in the moment based on the information that you have, which is going to be an average changing picture.
[00:16:42] Jake: Yeah, that’s a really great point. And I think it’s set us up in terms of like processes of things going forward. So where you say industry changes happen, like having those dates down in the dashboard and things like that would really help with those sorts of changes, not just COVID. So there’s lots of things that we can actually learn from this experience that we can use going forward, which is great.
[00:17:06] Dara: Okay, that takes us to the quote unquote ‘fun’ part of the show. And you get the privilege Jake of telling us and our listeners what you’ve been doing outside of work lately to wind down.
[00:17:18] Jake: Great. So this isn’t something I do on the regular, but I actually went to Amsterdam two weeks ago, which is really fun. Went to lots of museums, which was great. And then I got really ill after that for a whole week, but you know, it was fun while it lasted.
[00:17:33] Dara: Yeah, I hear those museums in Amsterdam. They can make people ill.
[00:17:39] Jake: Yeah.
[00:17:39] Dara: Oh, I won’t interrogate you any further. Don’t worry. How about you Dan?
[00:17:42] Dan: So I live in Lewes just outside of Brighton, that’s actually where our Measurelab HQ is located. And it’s actually smack bang in the middle of the south downs, and that’s just a lovely kind of hilly countryside region of the UK that’s just awesome for hikes and walking. And so I’ve just been out both days this weekend for a good couple of hours, saw different friends and eventually made it to a pub and then kind of headed home after a couple of hours hiking. So nothing super eventful, just managed to get out and connect with nature. And by nature, I mean go to a pub. That’s just a couple of miles away and walk back home. Go on Dara, how about you. What have you been doing to wind down?
[00:18:18] Dara: Well, I mentioned on a previous show that I adopted a couple of tarantulas. So my latest adopted animal is a Brooks King Snake, which we took home on Sunday. We actually went to see another snake that we’re adopting, which is a bigger snake, a Carpet Snake. And while we were there having some time with her getting used to handling her, we saw another snake, a slightly smaller one, which we really liked and got on well with. So we thought we’d bring him home in the meantime. So my animal menagerie, if that’s the right word is, is growing. So this is probably going to become a theme on the podcast every week. I can talk about what animal I’ve taken in. But yeah, we’ve got a new snake who’s joined the family, and it’s going to be another snake to come probably in the next couple of weeks as well. So I’ve got my hands full. Okay, that’s us for this week. As ever, you can find out more about us over at measurelab.co.uk. Or you can get in touch with us via email at podcast@measurelab.co.uk, or just look us up on LinkedIn if you have any questions or if you want to suggest a topic. Or better still, if you want to come on the show and actually discuss your topic with us. Otherwise, join us next time for more analytics chit-chat. I’ve been Dara, joined as always by Dan, but also this week by Jake. So a big thank you to Jake for coming on the show. So it’s a bye from me.
[00:19:42] Dan: Bye from me.
[00:19:43] Jake: And bye from me.
[00:19:45] Dara: See you next time.
Daniel Perry-Reed
Subscribe to our newsletter:
Further reading
What is Offline Event Data Import in GA4?
The complexity paradox: why your spreadsheet is complex and BigQuery is simple
