What NOT to Do When Framing an Analytic Project

Framing an analytic project is about designing the analysis BEFORE analysis is even done. This conversation should include far more than the analytic team. It needs to include stakeholders, those who will actually use/make decisions based on the analysis.

Below are a couple of examples where the right questions were not asked, the right pre-design was ignored. Then a rule is suggested to counter bad framing:

BAD FRAMING ANECDOTE 1

So a marketing client showed a model their econometrics team developed. This was to be a customer demand model and they wanted to estimate what drives units. The specification of the model was:

Units = f(age, education, income, size of household, consumer confidence, seasonality and lagged units)

The model fit rather well with an adjusted R2 of over 85%. The problem? There was nothing to act on, there was no marketing lever. This was a case of failing to frame the problem. The question was not “Can you estimate demand?” even though that was the questions given. The real question was “Can you show me levers I can pull to influence demand?”

This was a marketing problem NOT an economic problem. What do economists know anyway? They’ve predicted 12 of the last 7 recessions!

An actionable demand model would be:

Units = f(seasonality, # direct mails sent, # emails sent)

Regardless of the fit this is far more useful than the former. This is rule # 1 in framing an analytic project: Frame the Problem with Action-ability in Mind.

BAD FRAMING ANECDOTE 2

A telecomm client had a churn problem. They were losing nearly 25% of their subscribers every year. They had done several analytic exercises but was not able to produce a list of potential 60-day attrition more accurate than 34%. That is, they could produce a list of subscribers most likely to churn but only 34% actually did. That accuracy is not enough to develop a marketing campaign to stop /slow down churn. They framed the question as a probability to churn model using logistic regression.

On thinking about the problem, it seemed the question was not “Who is likely to churn?” but “What causes churn and can marketing do something about it?” That is, frame the question from the customer’s point of view. The job of analytics is always to quantify causality.

So a strategy brainstorm session yielded several causes of churn: high bills, using a lot of data/minutes/texts that contribute to high bills, dropped calls, no relationship with the firm, etc. This yielded a segmentation based on CAUSES of churn. There was one or more segments that were sensitive to each of the four hypothesized causes.

Then a churn model using survival analysis was developed for EACH segment. (My book Marketing Analytics shows the details of developing such a model). Because each segment was distinct based on causes to churn, the independent variables were different for each. To that segment that was sensitive to high bills a promotional offer could be designed to give them a discount. To that segment sensitive to dropped calls whenever a dropped call occurred an apology letter could be created and an offer issued.

This approach generated a list of likely churners that was nearly 80% accurate! The firm had a way to target and action-ability against each target to stop / slow down churn.

This is rule # 2 in framing an analytic project: Frame the Problem from the customer’s POV.