[16.07.24] Part 3: Metric Trees - Advice and best practice

I'm head of data account. We're back for our third installment in our operational clarity's metric tree master class. I've been talking about this one in all the other sessions, so I hope you all got the message to to come with this one. I'm really excited. We're gonna be talking about advice and best practice. I'll introduce our guests in just a moment. I just want to give a bit of kind of admin update. So hopefully you guys have used this Demio tool before. But if not, if you have any questions, please go ahead and drop them in the chat. If you just wanna kind of say hello as well. I'll be keeping an eye on that, throughout the session. So, please just drop questions in at any time. We have some time set aside at the end for, q and a, so some of those might be kind of logged to the end. But, yeah, go ahead and use the chat to to ask whatever you want. Okay. I think we'll go ahead and get started then. So it's time to introduce our guest. We're joined by Calum Ballard, who, is a new analytics director at Omaze. How are you doing, Calum? Yeah. Good. Thanks. Yeah. I'm on on my two weeks between jobs. So, yeah, the maze coming up soon. Yeah. If you saw the the early, early description of this event, for a couple people who might have different positions, so very exciting times. Last joined by Michael Rogers, who is the associate director of product analytics at Bumble. Michael, how are you? Very well. Thank you. Again, yeah, equally, I start Monday, so all change. They're not due to data, I should say. Like, they're not just starting in their data role to, like, So that they have to just different companies. And lastly, by William Mahmood from Mubi. How are you doing well? I'm really good. And, from a reassure Evan, I still work here. K. Yes. Very true. Well, great, guys. I think, we'll go ahead and just dive into the first question that we've got for you. So, how did you first come across metric trees? Maybe Callum will will start with you on this one. Yeah. It was it was some time ago in my my previous life in consulting. So we had a partner, where I worked who, kind of very into this idea of, like, we call them issue trees then. So, like, if you're trying to solve, like, a problem that seemed very high level and, and a little bit hard to approach, what were the things that you could do to kind of break that down into smaller and smaller constituent parts. That kind of translated almost one for one from, like, a general conceptual idea of, like, you know, company has declining sales. How do you break down that problem into something that's more like what we would be describing in this session as, like, a metric tree? So maybe your headline thing rather than is, like, issue, you have declining sales. It's more like, okay. This is my metric of total sales per week, let's say. And, actually, what are the things that you can do to break that down? So in the kinda context of consulting, like, they were super useful for helping to, like, both communicate, like, a problem that a client might be having, but then also, like, motivating, like, the proposed solutions and the proposed advice that you were giving as consultants. Like, you could very easily link that initial problem that they were having with, you know, this is what we say is your potential remedy or potential solution, and, like, this is how you can link the two. Mhmm. But, yeah, I found them useful for both, like, organizing and and structuring our thinking, but they were also really good tools for helping, like, communicate, both the, I suppose, diagnosis of their problems and then the solutions that you wanted to give. So, yeah, they were great in consulting, and they've obviously translated into into an in house role as well. Yeah. Yeah. I think that's, the consulting route is a is a common one, but something you said actually remind me of, something I neglected to say at the start, which is, why I invited you three on specifically in the first place. So just a bit of context. I've spoken to each of you. You guys have each built metric trees in well, some of you, your previous jobs. Well, a lot of your current jobs. And before that, you have a lot of exposure to them. So, yeah, it was good to go get you guys on to, yeah, hear how you know, we're gonna be talking about how you came across them, how you've used them, what's been your approach, just really kind of tangible real life experiences for this. So, yeah, Kyle, thanks for kicking us off with, yeah, the the origin story, I guess, for you on on MetricTrees. Michael, how about how about you? How did you first come across MetricTrees? Very similar. Lack of imagination. I started my career in consulting. And so when when you're prepping for a lot of these, consulting jobs, you do a case study. And one of the big things you get taught is this sort of issue tree technique. So Victor Chang is a really guy great guy. He has, like, a case study interview. So for me, that's where it started. And then you very like, it's very common to use them and build something like that in, like, Excel, Google Sheets, in consulting. And then when I moved into the startup world, I found people were calling these things, like, business equations. There's a lot of what PMs used. What's my business equation? A lot of the FP and A teams I worked to call, like I said, growth models. I was like, oh, these are all the sort of same thing that just keeps showing up again and again. And so, yeah, it's great now that we have a reincarnation, calling it metrics trees, but very familiar with using them throughout my career. Yeah. I think that's a good point. There's a lot of, you guys both touched on my other similar concepts. We covered, last week with Ergist as well, like, how the the origins of metric tree is actually really muddled with all these other, like, really similar concepts. And, Will, how about you? Yeah. So well, somewhat similar at the start of my career as well, but I started out actually at a start up, in a data team. And this was about twelve years ago, so showing my age. It was also done in Excel and to kind of pick up on kind of business trends at the time. It was when sort of social networking was the thing. If you wanted a big valuation, that's what you're focused on. And it was very much the lean analytics movement as well, and that was the context that we kind of started implementing at this time. I was completely new data, just learning SQL. This was done in Excel. Big Tableau was just coming up. AWS was sort of dominant in cloud warehousing. And, and, yeah, it was a really beautiful, striking thing to see. I can I also was struck by the fact that it was really stretching the limits of, Excel, but it was a really great exercise because we were all seeing what we would you know, how what about if the interactions were influencing each other? One could argue that it was also, probably also, there was a big risk of data overload as well, but that's also a function of Excel. So, count is is brilliant because, actually, you can only zoom in and out, and you don't have an issue anymore. And you can kind of, yeah, just zero in and show a lot of information but with with some clarity as well. Yeah. At the start of my career, and, glad to be picking up again. Yes. It must be, interesting to see how it's come back around as, like, in vogue again. Yeah. Very interesting. Great. One of the one of the biggest questions and my most frequent questions that we've been asked, not just in this kind of series of webinars, but just more broadly whenever I post about this, whatever it is, people are always asking where to begin. Do you have to start in a certain part of the business? Do you need to go get approval? How do you just begin this process of of starting a tree? Let's say, I think a lot of people on here agree that it's a really valuable thing, and they just kind of want some some guidance on on where to begin. So, maybe you can kinda give us some insights on where you started, maybe a recent tree that you've done, how that kind of project has kicked off. Maybe Will, because we can start with you. Sure. So I think this is a basic context. So Mubi is, is like a a streaming service for arthouse films. And I think the context for us using these metrics was in the sort of a lot of product analytics work, product data science work. And, essentially, me and the VP of products were sort of, and I'm I'm fairly new to Mubi, actually. So I've been in Mubi now about two, two two years and three months. Way longer than these guys. Yeah. And then, and and, yeah, we were we were actually trying to sort of, yeah, overhaul sort of some of our metrics. So, that was really quite good data maturity at Mubi. Lots of metrics the different teams are focused on, marketing content and and, obviously, within the product organization. But we felt that it was time to move more towards more of engagement North Star metric. So, again, with streaming, you know, you the business is fairly straightforward. It's how long do you need people, are subscribing to your service. The challenge then, though, from our perspective in the product areas, we're now trying to decide what to focus on or what to prioritize, and we need a clear way of understanding how different parts of our product are actually driving value. And it's not always clear how different elements are influencing retention. So, I mean, it was a really, yeah, powerful exercise, board exercise in kind of shifting from a business focused start, like memberships, for instance, to a product one. And for us, it was clear what is the utility of the product. I'm sure we'll talk more about, you know, how to get started and how to flesh these things out. But, yeah, that was sort of the context for us. So it was defining a product focused North Star metric, and and there was a lot of, from the different teams, a lot of, kind of convincing needed to be done. And so show these relationships and, and how actually, these these are the metrics we should be focusing on and convincing them. That was all, you know, really what why we we leaned on using a metric. That's a, yeah, very top down approach. And then you mentioned the the convincing, and, we've got a question coming up on that. But, so maybe I'll hold off and you can tell a bit more about how you how you got people bought into it in a moment. I think that's also a really big part of this. But, yeah, it's interesting to hear. I think that's the most frequent way I hear anyways, that top down approach of, you know, we need to set what star. We need to, this is why we need a a metric tree to to make it work. But, Michael, I think you had a slightly different experience. Is that is that fair? Yeah. Oh, I think you're on mute or I can't hear you. I didn't. Can you guys hear him? No? No. Okay. Callum, maybe we'll go to you. Michael yeah. Callum, we'll go to you, then Michael will come back. That's alright. Yeah. I think our experience or at least my personal experience of of developing them is that, it was very much focused in linking the very high level business metrics. So, like, your total, like, user LTV, for example, which is like this big company goal over here, and thinking about, like, how a specific pillar of the product organization can actually contribute to that. So, I guess, the example, from from my previous role, you know, you had a few different revenue generating features, one of which was, like, this new feature, which was, like, this debit card proposition. So, like, how can you demonstrate, like, some of the impact that my new shade of, like, this debit card could have actually, like, on the top line of LTV? So starting with that big number at the top, like, the different product pillars could, you know, create their own individual trees. But for us, within that bit of the product organization, it was it was very much around, like, hey. Well, how do we contribute to that number knowing that, you know, there are other other areas of business that will also contribute. And, like, how do the metrics that we care about as, like, product squads and looking at kind of the low level things? Like, actually, how does that contribute, to things that, you know, the c suite care about or, like, the commercial team care about? That was that was kind of, like, our experience of, like, going from that quite granular, low level place to to kinda demonstrating how we were having impact overall. Yeah. And that's, I guess, the context as well is because you were kind of sitting within the product team. Right? And you guys were embedded within this product team. So I think it's kind of different than well, you guys are structured as, like, a stand alone data team, and so you are kind of almost advising the the VP of product. Is that just people by context, a different set of So we are sort of I mean, no. We are, somewhat embedded in the product organization. Oh, okay. But we still work in, well, we all get the teams at Movie as well. So, again, it's very much a a sort of a new data team, but we work for for the whole business. We focus on the product side for for various reasons, mostly around that sort of one of the big areas of opportunity to leverage data. We're at the sort of crossroads now with quite a mature product. Where do we go next? So data is really instrumental there. But, also, there's a lot of synergies between, the modeling we might do or the or the data work we would do and then for potential products, we might wanna build in solutions for the other team. So us being there and that kind of part of the of the organization, helps us kind of identify those and and deliver those types of solutions. So, but, yeah, no. We work across the whole business. Okay. Thanks. But our embedded in time. Okay. One of those. Yeah. I'm sorry. Yeah. Michael, how how are we doing? No. Maybe check your, if you go to open settings in the top. Sorry for the live IT support, you might even just check your inputs and stuff there. But they will come oh. Does that work? Yeah. Good. Alright. I I I found it again. Of where, I think the yeah. The focus there was a lot more sort of trying to understand people's impacts or or areas of focus. I think ours was very tactical of just laying out the dynamics of the business. So in terms of where we began, we focused with a dollar number, and my sort of recommendation is start with a very simple thing you can count and kind of decompose it. So whether that's users, dollars, whatever your your thing is. So, yeah, we started with dollars, broke that down into very clear components to understand the dynamics of the business and say, you know, year to year, why are things changing, where to focus, kind of bring that, cohesion across the business. So we found that focusing on being very clear with very precise compositions and, things that ladder up to that dollar number allowed us to go quite deep, quite quickly to understand where and why things are are moving. So that'd be my recommendation, really, at that spot. Yeah. And, essentially, you guys all kind of talked about, yeah, starting at the the highest point, I guess, within your domain that you're talking about and then bringing it down into something, like, really tangible. So I think that helps kind of outline, another question later, like, where where do you stop? I think that's a good kind of outline of both where you should be starting at the high point, but then also kind of looking down where you wanna where you wanna end the tree. On that, I think part of starting is also getting approval, getting buy in. Michael, do you have any kind of advice for people on how to do that? Yeah. So the the things we found that work are when people see this, it's very intuitive. You know, we're we also mentioned it at the beginning of his experience when he first came across them. I think, naturally, this isn't something you need to fight too much for, but people kind of once they see it, go, oh, wow. This is great. Sometimes not people aren't really gonna ask for it. So finding a really common use case where this can just be brought up very naturally and people can understand it. So, you know, people might have weekly trading meetings. They might have a product review. It might be planning. What are those forums? Or just one very clear use case where it's like, this is a place we already talk about metrics, but we talk about five metrics, and we never understand how they link together. This provides a really nice framing for how to do that. So I think once you still start doing that, you've already got a place where people are familiar with the kind of language that you're gonna be talking about, and it will naturally get picked up. And then they'll start asking you to do more, build it out even further, add proxy metrics. It will get very complicated very quickly, but this is, yeah, a great place to start. Yeah. I think that's a good like, where people are used to seeing these numbers anyway, and you're just kind of shifting the the presentation of of them. Yeah. You don't necessarily need to get approval for that. Right? It's just kind of a an enhancement in your. Yeah, budget. Yeah. It'd be obvious. And, Will, you mentioned when you're going through your your story a moment ago about how you needed to get other teams on board. So you kinda had you know, the buy in of, you know, you had a partner with you in the VP of product, but you needed to get these other teams on board with it. How did you approach that? Yeah. So I think what we immediately realized and knew, our businesses, we're all actually influencing each other to a very high degree. I think there are lots of businesses like that, but certainly in the streaming space, sort of when you're building the product, like, you're heavily, beholden to the content that's on the product, the way that content's being marketed, the CRM activities that are going on around, and then and vice versa. The other teams are beholden to sort of the where the product is exposing the content and and servicing it, and how our personalization algorithms are working, all of that. You know? All all of what we do is influence each other, and I think that was one of the big opportunities that we saw with first of all, we need to show these dynamics, like, clearly between the different teams, and so they were essential to be brought on board. So it was about making sure that it was relevant to them all. So showing sort of like, okay. If we're looking at the world from a engagement perspective or utility perspective, in our case, viewing, you know, what needs to be true to drive a viewing, and then we sort of work backwards there. And, actually, thankfully, you could see in the data, we're going down the right path, and you can see, right, actually, problem one. We've got some users that are not visiting. So enough for the central you know, to watch something, you gotta be on the platform. And and then immediately then, you've got the stories that can resonate with CRM and marketing. And, actually, we have further segmentations as well of our users that can connect back with churn and business outcomes. And then on the product side as well, we can go into this kind of consideration funnel of the tree. And we have quite an, I would say, an, a nonconventional tree. We'll touch on that later. But, essentially, it's three trees really that, represent a funnel. And and, again, yeah, that's partly why as well. So this is us kind of trying to show for each team, and make the make the make the tree relevant at a strategic level. But, certainly, on the consideration part, one thing that's been interesting is you need to, when you're on the platform, consider films. So we represented all these different metrics around that aspect, and that's been great because the product teams can say, okay. What things are we doing that we are actually fully uniquely focused on and and influential in? Yeah. And and, yeah, I think from that perspective, it's about making the content really relevant to a bunch of teams, but it should always be done in a way that's actually makes sense. Right? So for us, our activities are all influencing each other, and our goal is to try and get everyone behind the shared golden north star framework. And that that that kind of approach is is really relevant to us. It may not be as relevant to other teams, but for us, that's what we're trying to do. And it's, it looks like it's working quite well. Yeah. I like that. I think I think, what sticks out to me, I guess, is the being empathetic part. I think by saying, you know, this is a problem that you guys face. We all maybe feel like we're not working on the right things, or we're not on the same page. This is something that can help with that. Here's why we're building it. I think it's important to oh, go ahead. Yeah. Yeah. I was gonna say another kind of point as well. It's probably a good kind of just advice as well is you've gotta, yeah, obviously make sure that that you've got content for those different groups that you wanna be engaged by this. But I think I've really loved how with count, you know, they can actually go in and check it regularly. So we we did have some kind of, some monthly reports we did with our with our top level strategic tree. That was kind of key in kind of building kind of interest in it and kind of show people how you interpret this and go through it. But just having this place where we can visit naturally is Mhmm. Key as well. And, certainly, yeah, we have now a mini road map where we're trying to make sure that we're kinda keeping people alive there. And, certainly, this the work is still going on. Like, we're still sort of getting teams fully on board here. There are still, you know, other goals and priorities that specific teams will have when they're sort of reporting about their business impact. So there's gonna be, not necessarily attention, but just how do you actually, kind of sit side by side with that. But, but I think it's working reasonably well. So, yeah, give people relevant content and and and, yeah, use a tool like count so they can go in there and actually use it. Yeah. I think it's, especially talking about, you know, in in your context, bringing all these different teams together. Like you said, it's important to have a place that is visible to everyone. It's really clear how everything's made, like, very transparent, and it's just one one source of truth. I think Ergus said on Tuesday, one of the things he was really, passionate about is the metric tree shouldn't have filters on it. Like, it should all be the same view for everyone. It's not like a typical dashboard that you're trying to create where people can slice and dice and look at things differently. Like, a tree is a tree. It should represent the business. It should be clear. So, yeah, I think having having one place, like you're saying, to to check everything could be really powerful. Calum, I wanted to go to you because I think you had also a different experience getting approval as well in this in this context for Payette maybe. Yeah. I would just say I've had great success adding filters and segmentation to trainees for what it's like. That's really powerful since this anyway, that's a different conversation. Yeah. I guess in terms of, like, getting buy in per se, like, and maybe it's the nature of of the business that I was I was doing this at where, like, there was a relatively high level of data literacy and, like, analysts were embedded across the business already. But, like, my approach was just gonna do it. And then when I showed people, particularly, you know, the there was one conversation that sucks as success was, like, the head of commercial, and he was just like, oh my god. Why are we using anything else? And I think, like, I'm gonna enter my my shilling for count mode, at this point. But the problem, I think, particularly previously, is that, like, there are no good tools otherwise that really unlock the full power of this using, like, locked data. Right? So, like, at a previous job, I tried to use Tableau to build a metric tree, and it was the most horrible experience of my data career. Like, it was very, very funky. The user experience on the other end of it was not great. So, like, having something that exists in that strictly two dimensions that allows you to, like, visualize, okay, this is your metric. These are two metrics that go to that. Let's take this one, and that breaks down into you know what I mean? Like, there's no other tool that you can kinda do that nicely in. So just, like, a combination of, obviously, just theoretically, they are good for, for communicating kind of data, but also, like, having the tool that really unlocks that kind of which you've not necessarily had before. I think, like, that together allows you, if you build a metric tree using count, to go to, like, a stakeholder and say, hey. Look at this. And they probably aren't gonna have seen something like it before, which is obviously, like, is quite powerful. Yeah. I think just I think we think about it anyway. It's just trying to get out of the way. Like, I think it's probably so painful to do it in Tableau because it's not quite made for that kind of thing. So I think it's just about letting, you know, letting you go and let a tree be a tree and let all those, you know, connections be clear and obvious and that kind of thing. But, I really like that just kind of just do it approach and, and let it speak for itself, I think, is is really powerful. Okay. Great. I wanted to jump to a new topic. And when we did our session on Thursday on building a tree live, we had quite a few questions on defining metrics and dependencies and how to think about this and things like, oh, should it be always be MECE? Do things always need to add up to a hundred percent? How important is it to have everything, like, really defined and clear? You know, normalization, these kinds of questions came up quite a bit. And, Michael, maybe we'll start with you on, yeah, your your thoughts on just how to define these metrics. Yeah. I mean, I would say when you're starting, definitely go MECE and definitely go everything adds up to a hundred percent. Your at the level of complexity that you add as soon as you start going away from that is huge. If you are getting stuck decomposing things, and I I Calum has some some good tips, but I've also found that sometimes take dimensions and make the metrics. So, for example, you might have users is a really obvious one, and you can kind of break it down by the growth accounting steps. Right? Like, new, existing, like, retained, churned, that's resurrected, that sort of thing. Like, turning those into metrics makes it much easier, and you can kind of just keep diving deeper. So definitely always go easy and, yeah, split to, non proxy, like, actually being able to decompose. And then also be willing to go backwards. So I think starting top down, but Will mentioned it in some way. The product work is sometimes you know a behavior that works, and you kinda actually need to work backwards and say, how does that behavior and the other things I need to make happen? And then find the kind of in between where it then links to your the metric where you you start at top down. So you go it's not very linear. It takes practice. You'll need to do a bit of pruning, but just stick at it, and stick to the maths being nice and easy. It will work. We pay dividends. Can you you mentioned proxy metrics. Can you say a bit more about what you mean by proxy metrics? Yeah. So, yeah, for example, if you're looking at this will so often happen in the world of product where you don't have a, you know, a very clear, this thing, we move this by two percent. It's gonna, you know, ladder up to this other metric moving ten percent, whatever it is. And so we try to say, oh, well, to get that metric, it would be too expensive. The timeline is too slow. We won't be able to run experiments. So we need to have a layer of abstraction, a metric which doesn't directly translate to that other thing. So this is where we come up with proxy metrics, and we say we think this moving this thing is indicative of some other, metric in the future. And once you start getting down that route, it is very, very scary because those metrics, you don't really know how well that correlation is gonna hold, and it's something you need to monitor quite a lot in terms of, how long does that relationship last? You start getting into is it normally distributed versus kind of power law? You need to consider all these things. And then particularly consumer products, you're often gonna get power laws, so you need to start doing logs of the world or, some normalization, and people get a bit confused. So get there eventually, but just yes. Don't don't start there. Yeah. Don't don't overcomplicate it, I guess. Yeah. Yeah. Definitely. Callum, I think you're kind of in a similar, similar mindset to Michael here in terms of Meecee and that kind of thing. But but anything to add, I think, yeah, yeah, you had a few tips, for this as well. Yeah. MECE is, is kind of worth shooting for. I think the other the advantage of keeping it MECE as well is that you can use the tree in, like, in sizing. So, like, if you know that there's a mathematical formula that links kind of one layer to the next, then you can kind of start making statements like, okay. If we increase this by x percent, then that will have a y percent increase on on the metric above. As as we said earlier, like, when you start getting away from that more mathematical link, like, that kind of exercise becomes a little bit dicier. In terms of, like, actually thinking about how to go, like, from one lens to the next, I think, typically, I will try one of two or three kind of mathematical tricks just to keep it keep it MECE. So an obvious way of doing that is kinda just like a simple addition. So, like, you know, just to take total sales, as your, like, starting point, you could have, okay, total sales made through web, total sales made through store, total sales made through, like, telephone outreach. I'm showing my age there. I'm sorry. But, like, the idea is that you you are accounting for all the different sales channels, and we know that if you sum all that together, you're gonna get the number at the top. So that's one thing that I would have in mind. The other thing in mind that I use a lot is when you're trying to break down a ratio. So to take an example, like, total revenue per customer as an example. So, like, when I'm thinking about breaking down some ratio a over b, a good technique is to introduce, like, a third element c. So, you know, mathematically, a over b equals a over c multiplied by c over b. So just to to give, like, a concrete example of that, so if you had, like, revenue per user, you could introduce, like, total number of sales as your as your third kind of thing. Mhmm. So revenue per customer becomes like, revenue per sale, so the average size of each sale, multiplied by your sales per customer. So you've taken that total higher level metric and broken it down as two separate business kind of problems. Right? So that's you know, is is this number increasing because the average size of each order is increasing, or is it because you have an increase in the total number of orders per customer? So they're kinda like breaking down a higher order problem into, like, two separate business. And it's important that those two things are, like, business relevant. Like, obviously Yeah. It's mathematically correct to add literally anything as you'll see in that, but, obviously, you need to create two things that kind of make sense in the business context. So, so, yeah, those two things are are definitely what I do, particularly for the first few levels. Mhmm. I would I would say I would break that kind of mathematical rule of probably, you know, four or five layers down. An example of this might be, you know, thinking about, like, retention as a metric. We know that there are certain things that drive retention. So, like, you know, you probably want to track stuff like customer satisfaction and MPS in a a metric tree. Like, it's something that the business cares about, but, like, that doesn't mathematically really go anywhere, but you kind of know that it drives attention. So, like, maybe you have, like, a bit of a dotted line linking the two. Those would be an example of of, like, why you might break the mathematical kind of linkage, but trying to keep it mathematical as long as possible, I would say, is is what I would try and do. Yeah. I think it is about, like, holding those two things intention, right, for the business understanding and what's important to the business and then also what's, yeah, mathematically relevant. I think on that tension, I Will, you had quite a different I guess, if if Michael and Kyle are kinda more on the mathematical side, I think the tree that you made is a bit more on the what works for the business? Like, how is this gonna help us more? So maybe you could talk us through your tree. Yeah. Well, it's funny. It sounds less mathematical, but it is actually, I would say, kind of mathematical. It's like you're you're making a formalism for what needs to be tuned to drive an outcome. So, again, partly with kind of an outcome goal. As long as you've picked the right one and you're working backwards, you're then, I think, in a good place. And and it has to be logically consistent, but, in that just just to kind of, you know, maybe maybe use a wider or more general pub's audience. But I think as long as you know your goal, what are you trying to do, and then define metrics. And it'll always be, I think, a single metric you should start with, and then you wanna work back and develop out the the metrics. It's a it's a process to be pathfinding and trying to understand, firstly, what are the metrics that matter? And you'll realize actually that some matter more than others even though they fit logically in that tree and can be related. But it's I've I've ever start with, like, a goal than what is the metric for that. And, again, just yeah. In terms of our one at movie, though, it's a it is a bit different. All we've gotta say North Star Metric, pretty clear for our business. Utility, it's viewing. That's a very clear thing we do. So there's very low risk, I think, with that. And then working backwards through, okay, what needs to be true to drive that? We know that. Then you could say that's a sort of a nice simple formalism. We will need to kind of be or if we're visiting the platform first and foremost, considering films, that's a dynamic we wanna understand more. And then viewing, let's understand that aspect as well, then have appropriate segmentations. And I think provided everything has a logical consistency, it works. But, you're always working, I think, backwards from something. That's, I think, kind of essence of what a tree is. Otherwise, it loses it. But but, yeah, I think start with your goals and just to be true to your kind of moment, you'll you can clearly have a pretty good intuition for when you're varying, of course, or if it's not useful. And on and that's actually the key thing here is just understanding how you're gonna use this. I think that's a really good test here. So, if you're just starting with the top level North Star metrics strategic tree, that's what we call so these are for our strategic metrics, and I think it's important to draw a line under your use case because these can't go on forever. The goal of that strategic tree is for us to know, okay. Actually, we're gonna use this to the wall. Every product, for instance, has to define itself in terms of what job it needs to do strategically. So, you know, when we look at the strategic metrics and an actual product on the product roadmap, probably, man, we have to say, okay. Which part of the the character is gonna be? Is it gonna move consideration to what's your efficiency or or something like that? And then, of course, so you ladder lower. There's lower levels of, like, detail that are relevant to, okay, actually, for this product to be successful, there's probably some other metrics that are probably relevant there. But we don't wanna cloud and confuse people at the strategic level. Yeah. And then you get that kind of you can then maybe there's another tree then for that part of the product that's, you know, really important for viewing for you, and you decompose that, and you can be truthful to that area as long as it's encapsulated. But then you're connected to this higher kind of strategic view. And that actually is already Yeah. With the product space, I would definitely recommend when you're doing that North Star kind of production anywhere. Yeah. Think about that like a strategic tree, and then actually you can create trees for other parts of your product as well. Yeah. I think that's really good advice. I also just wanted to to mention, I think, in your tree, how, you know, you've found out quickly how important, as you said, that the funnel was and even segmentation. And so you I think your tree structure is is kind of unique because you worked out, like, oh, no. This is an important aspect to bring into the tree. So I think I just like that you had the flexibility and the kind of confidence to say, like, yes. This actually makes the most sense to how we understand this. Yeah. Yeah. Go ahead. I'm not saying, like, about strategic log in. Again, just knowing kind of what do I want to do with this tree. Mhmm. And that helpful barometer for kind of where you are with it and if it's working for you. So in addition to just, you know, setting metrics for your other products, so understanding what the strategic goal should be, they should it should also be a guide, I think, to kind of the unknown questions, the big strategic insights. So that's our kind of ultimate test is, are we able to also look at here and say, right. Actually, we should be spending this insight time on trying to understand why that cohort of users is not visiting less. So how do we unpack that, and and can we come up with some recommendations about some opportunities there? So and that's the whole goal of all this is how do you make the data more useful and understand how it relates to each other and the way you should be prioritizing. Because I think it's fine. The data should always be it should simplify decision making. It should make it easier to say kind of what we should focus on. But unless you've got a nice rigorous framework, which, which is what our metric tree, I think, is an exercise in trying to do Mhmm. Then the data can be completely, really confusing and everyone's just running around, and it's actually a muck. But once you do that and you know who I help, you know, what's important, everything then, comes together. Yeah. Awesome. I have I have a question. So Thursday, when we were doing the live session, there was a question. We're building a tree for daily active users. I think one of the we're looking at the conversion rate at some point. There's a question about whether that kind of thing should be normalized because, obviously, it's very subject to noise and, like, if a cohort size is, you know, larger than expected or smaller than that conversion rate's gonna be noisy, might lead to, like, erroneous conclusions. What do you guys think about the approach to that situation? I can take that first, I guess. I think it it depends. That's a really cop out answer, isn't it? It depends. I think it depends, firstly, on, like, how you have presented the tree. Like, if you've got metrics that are legitimately, like, super volatile and super noisy, you might wonder if they are helpful to have in the tree at all. I would imagine that a particularly noisy metric is probably gonna be lower order. It might be quite a long way down the tree anyway. If it is, like, a bit higher up, like, if you've got a total, like, conversion rate across the entire user base that is, like, super noisy, then, obviously, like, that is maybe an interesting finding of the tree itself, and it's worth investigation because you'd probably expect at least on a on a weekly kinda cadence or a monthly cadence depending on the product, like, a a metric like conversion rate wouldn't be too noisy. So it may be revealing something that you didn't know about a particular metric, or it might be revealing something that you didn't know about the product or whatever. So, yeah, that'd be my two cents on it. Yeah. Similar, we we often use our tree to understand why certain metrics have moved. And so being able to really quite quickly point to, oh, it's volume, and then we can say, oh, let's break down volume by, okay, country, let's say. I think that that brings a lot of noise, so I would not want to, you know, re yeah. Just, if you're what's the purpose of the tree? The tree is to kinda, like, understand, why these metrics are moving. And so if volume is quite volatile, that's really good to know and know where where that's happening and why. I think it, yeah, you want it to surface those things. I would say in a really practical effort as well. Like, they are the kind of metrics where in a traditional dashboard, like, you could have a bit of movement, which is just due to volatility or is just due to, like, something a bit random. And in my experience, they're the kind of metrics where, like, a senior person will see it, get scared, and demand an instant explanation about investigation into said metric. And in the old world, like, that's the kind of thing that can basically, like, kill a working day for an analyst because they haven't spent it on that. Obviously, if you've got it in a structured tree like this that is set up in a self serve way, like, what Michael said around, okay, is this an issue with volume? Is this an issue with market? Is this an issue with yada yada yada? Like, it's plainly visible, like, what's going on. So, like, you can use the tool like this just to, like, shortcut a bunch of those steps. Maybe it needs, like, half an hour to look into. But, like, in terms of time saves and, like, maybe allaying the fears that can result in some kind of thrashy kind of decisions, like, they can be a useful tool in that fight, I think. Yeah. I should add really quick. I think most of the points will cover, but I think, yeah, again, the whole point is that you should see the key close neighboring metrics that all interact with each other and be able to immediately understand whether, you know, okay. Right. This increase, was it was it meaningful, or or was it actually anomalous or or or whatever? And I would always say I think this was brought up actually in our pre huddle. But often, actually, things will usually add up to an aggregate metric, almost always just mathematically. And so that's always tricky because someone says, hey. So surely the North Star shouldn't just be gross blah blah blah or aggregate watches. But, actually, then you say and you go down the street, and you say, actually, well, you can have the rate or the frequency or the adoption rate or engagement rate side of it. That is not the full story either. But, so, again, it's all about showing that that in a clear framework. And and a simple thing for us that works, I think, that's used in general is there's the aggregate top level metric, and then it has its derived, you know, frequency and adoption metrics. And and actually, yeah, usually things you can usually cover most bases with that kind of approach. But yeah. No. Too pretty. I guess that's that's kind of the mindset shift. Right? In a in a dashboard, you'd be forced to try to explain so much in in such a small space that you might have to normalize. You might have to do something like that to try to to shortcut or prevent or preempt those questions. But I guess with this, you wanna make all that really visible. What are the trade offs? What is everything leading into this? So you're hopefully you know, you have no need to do those kinds of things, yeah, the same way. Okay. Great. We're on to our last question. So just another call for any questions in the chat. If there's anything, if this isn't the last question that you had, and you have another one, so please go ahead and drop it in the chat. And then, yeah, the last question for you guys, what's one last piece of advice you'd give to someone starting their first metric tree? Tell them, maybe I'll start with you. Yeah. Sure. Yeah. I I kind of had sorry. Another question is one last piece. I've got two things. Oh, no. No. No. No. Yeah. That's fair enough. So, like, I think when constructing them, it's helpful almost kinda like a jigsaw puzzle just to, like, empty all the pieces on the table. So, like, these are the metrics that I kind of know the business cares about, and then try and piece those together in such a way that, like, you have kind of these relationships between the metrics and the tree. That could be useful just, like, help you get started. But it's also useful just to make it as relevant to the business and your stakeholders as possible. It's just like, I know this person cares about conversion, or I know they care about, like, churn or whatever. And it's just like, if you're starting with the mindset that, okay. I'm going to include those things in this dream, it obviously helps make that relevant to your stakeholders and, obviously, buy in becomes easier. And the last point that I'd make is just, like, when you are doing this for the first time, like, it's actually not as easy as we make it sound, I would say. Like, when when I was kinda talking about, you know, this, this partner at this, consultancy where I started, like, he would train us by, like, literally saying these as, like, exercises. Like, I maybe did, like, five before he kind of looked at it on this little sheet of paper that I'd done. He was like, oh, yeah. That's alright. So, like Do you mean the the defining of the metrics itself is is not as easy as it sounds? Or I push part of it? I would say just, like, constructing it, making sure you're starting with the right thing, like, making sure, you know, when you come to break down something into two component parts, like, there is a choice there. Like, sometimes the choice is very obvious, and sometimes it isn't. And, like, you kind of get a bit of intuition after you've done it a few times as to, like, what is likely to be successful. But, yeah, I guess the the summary of that advice is just like, it can take a few goes to get it right, and, like, that's fine. So, yeah, it's it's kind of a bit of practice and a bit of trial and error sometimes, but, yeah, that's what I'd say. Nice. Will, how about you? I say I echo those points. I think for just starting really conceptually, sticky notes, starting with count. But I think starting with sticky notes, you know, starting with outcome, I think we care about your goals. So are you, again, building a strategic view of your metrics? Whatever you're doing, whatever exercise, you'll probably go you're you're trying to bring clarity to your metrics. You're trying to decide what is important and trying to represent that. So, yeah, design it well from the beginning, but iterate with stakeholders and, ideally, with with with a single stakeholder around a kind of clear focused aim. You don't need to have everything in the single metric tree initially, but that might be an ambition that you might wanna have. We certainly actually initially wanted to have in count, like, a complete view of everything in our business, but realized that, there was some technical considerations to the actual loading of the tree. But, actually, we've realized you can actually do a lot more in count. Actually, I might come back to that. But I think focus was important because these things can go on being Mhmm. Logical consistency. Yes. Start with a clear goal. You know, North Star one might be a nice project. Work backwards from that North Star metric because I'm just sure it's the right North Star metric, and think about who your initial stakeholders are and then and be truthful to what you're trying to achieve, which is you are trying to create a tool for either setting metrics and goals in this context for, like, other products or try to identify where you should be investing your insight time, at least if you were doing a a little stuff framework. Those would be. I I mean, like, as well, I know one of the things you told me when we spoke about your tree in the past is when you start building, it was kind of something you did every Friday, and you would just work on every Friday. Creating a ritual around it, I think, is is also something I've seen really effective both in when you're building it, but then also when you go to review it as a team just to know, like, hey. It's Friday. We review this, and that's what we now do our internal metric tree that we've built. We just say, like, every Friday at this time, like, that's what we're doing. We're checking it out. So I think that's also a good thing that you've done that I really like. And that was a good way of also learning count, by the way. Yeah. Just every Friday. Yeah. Michael, how about you? Nothing too original. So, I echo a lot of that. I think, you know, start with an aggregate number like dollars, users. Stick to, you know, mathematical things as long as possible. You'll build the intuition of when something should be a dimension that you can filter on versus when you spread it out into a metric itself, like, you know, new users versus existing blah blah blah. And yet don't have one tree to roll them all. It just, I I mean, you you might be able to get that. Just make it easy for yourself. Don't do it. Yeah. Well, great. Well, I I don't see any questions in the chat. I'm happy to ask you guys one more then if you're if you're up for it. So maybe just in one sentence or or a few kind of bullet points, what impact have you seen metrics have in your organization? Maybe Michael, you're off mute. Maybe I'll start with you. I mean, not to not to push Count's, content too much, but, Count I think, you know, Taylor, you you can probably speak to this better than any of us. You've written this really good post about, like, driving operational clarity, and I think it is it really, really resonates. And when you do something like this, you'll see all the things, like, in that in that post sort of come true. For me, I think that very quickly, it's been, you know, answering why metric has changed, that becomes trivial. Getting alignment on the business of sort of how our business works and what's important even with product teams and finding points of leverage, and then easily understanding the relationship between metrics. So when user volume goes up, conversion rate likely goes down. Mhmm. Those sort of things, that all becomes very, very easy. So Nice. Cal? Yeah. I'd echo based on that, I think. Yeah. It's around, like, just bringing a clarity and understanding to, like, a wider organization about, like, how stuff is interlinked, and how, like, an effect in one place can have an impact elsewhere. Like, just putting that down on very clearly on paper, if you like, is is something that I think is hard to do otherwise. So yeah. There's lots of, like, implications from that as far as, like, you know, planning your activities as a product manager or, like, having an understanding as to what initiative should be focused on if you're, like, a commercially minded person. But, like, the TLDR is that it's it's around, yeah, clarity of of the relationships between metrics. Nice. Agree. Will. Yeah. Yeah. I'd say, actually, again, the main impact being us kinda shifting towards a more engagement based view of of our business and our and certainly in our product and in the product talks. So having a an engagement, but, North Star versus, like, a a business metric in NorthStar, like like, membership sort of revenue. Those are important. But, again, for our purposes in product, we're trying to attribute all the elements of the product to, you know, in our case, like, what you come to do at Mubi, which is you come to watch films. And the more we can understand how different elements of product drive that, I think the more successful will be in, and and, actually, the data will be a more powerful tool. So I think that's been really the main shift. Again, there's still a dialogue going on here. Every you know, our business, like like, all the other businesses here as well are very dynamic businesses. But I think, yeah, that's been the big, big, big shift. And, certainly, I think the big test was, you know, we were shifting and we realized, like, how do we deal with, retention? You know? And and that is very sort of a metric that really I think it's very hard to attribute all the elements that go into retention for a product like, streaming product. And so we were able to kind of take what we've done here with this engagement view and able to really clearly say, like and and identify priorities actually for the different teams. That was, I think, a good, litmus test for kind of this being the right direction. And and, again, yeah, us being focused on actually, the best thing we can probably do as a business for data, like, we just focus on the the consumer utility of what you do on movie, and then, you know, retention and all that stuff, LTV, should be to take care of itself. Yeah. And just to echo, I I think something I find often with product orgs is, with you give them the confidence of, like, the right thing to focus on and that the scoreboard takes care of itself, that freight, it is a quite a big game changer in how you function, and people feel you stop getting the questions like, oh, we're focused on the right things. Well, it just comes back to the impact of did we move the metric we said we wanted, which is actually really, really hard to do. And then we know, okay, once you've moved it, it is gonna have an impact. So it it has a yeah. It makes a big difference. Yeah. I think it's hard to to it's almost something you have to feel. Like, I don't I don't think it's not easy to explain how like, the cognitive load of trying to keep this all in your head all the time. So I think just relieving that stress and then be able to focus on how do I best deliver to this these goals, I think it's, yeah, being able to focus like that is really it's a really, big impact I've seen. Well, cool. We are almost out of time. Thank you all for for coming out. Thanks for everyone who's who's joined in. Thanks to you three in particular. I really appreciate your time. You have such good, wisdom here, good advice. I've there's lots of blog posts in the making with this content, so it will live on in many forms. I hope you're ready for that. And, yeah, up next, Michael did allude to operational clarity. So this is, you know, an idea that underpins Metra Trees as well as some other concepts of just how do you make your business as clear and transparent as possible. And that's not just about MetricTrees. There's some other concepts that go into that as well. So, the last session in this, series is gonna be on Thursday. That's gonna be with our CEO, Oliver Hughes, and he's gonna be talking us through what operational clarity is, what are some other ways that you can apply that, and I think how it can really change how data teams operate, especially if you feel like you're in kind of a service trap or you feel like you're just kind of constantly churning out dashboards. This is a really good way to kind of shift out of that out of that way of working. And that's it. You'll all get a recording link in the next day or so. If you have any questions, feel free to email me or reach out to any of these three as well. And, yeah. Thank you all. Thanks, Tyler. Cheers. Thanks, Tyler. Thanks, guys. Bye. Bye.