Jeb Stone's Blog – Observation, commentary, and helpful hints at the intersection of Data Science and Social Science

I still think we collectively learned the wrong lesson from the Netflix Prize contest

The collective conclusion from 2006’s Netflix Prize competition was this: You can’t beat massive model ensembles. Ten years on, I’m still convinced the larger lesson was completely overlooked: that sometimes, when modeling human behavior, there are better KPIs than RMSE.

In 2011, I began work on my own SaaS recommendation service (called Selloscope). As part of that work, I evaluated the performance of a Slope-One model (similar at the time to Netflix’s Cinematch algorithm) against a spiffied-up object-to-object collaborative filter. The latter doesn’t produce an error score, so I devised a different goalpost: If I remove random recommendations from each user’s profile, how well does the model fill in those gaps?

Continue reading “I still think we collectively learned the wrong lesson from the Netflix Prize contest”

Why Startup Accelerators and Early Stage VCs Should Provide Centralized Analytics as a Value-Added Service

Note: This is a significantly longer version of an article I recently published on VentureBeat.

“Startups are the sum of the decisions made by the people who run them.” — Uzi Shmilovici, in Data Driven Decisions for Startups.

Uzi’s recent observation that “startups are the sum of decisions made by the people who run them” really resonated with me, and he’s absolutely right — the earlier startups begin collecting data, the better their decisions will be. There’s only one catch: Most startups don’t have any bench strength in Analytics. Continue reading “Why Startup Accelerators and Early Stage VCs Should Provide Centralized Analytics as a Value-Added Service”

Centralized vs Decentralized Analytics: All You Need To Know

Having spent the past dozen years in an Analytics role, both as an individual contributor and as a manager, I’ve had the opportunity to see the analytics function work within centralized, decentralized, and matrixed organizational structures. The relative merits of building a centralized versus decentralized Analytics organization depend largely on exactly what you expect to get out of the Analytics function in your organization, so it’s important to consider the pros and cons — and to structure the Analytics function correctly to meet your organizational goals. Continue reading “Centralized vs Decentralized Analytics: All You Need To Know”

Jeb and Selloscope Featured On Forbes Re:Thinking Innovation Blog!

Just wanted to share the news that my startup Selloscope got some great coverage by Haydn Shaughnessy over on his Re:Thinking Innovation blog at Forbes.com! It was a pleasure to speak with Mr. Shaughnessy, and while I appreciate the feature, I’m particularly looking forward to following Mr. Shaughnessy’s work on innovation moving forward.

These ‘Fail Fast’ Guys Can Pry My Startup From My Cold, Dead Hands

It’s a fact of startup life that you win some and you lose some. Mostly you lose some. I’ve tried my hand a few times at bootstrapping a startup. I have one now that’s just ramping up, but I have to admit that along the way there have been a couple of times I just let a domain expire so I could move on.

Speaking of “moving on,” there’s a hugely popular yet dangerously bad idea out there that’s captivating more and more decision-makers: this idea of “Fail Fast.” I thought Mark Suster had put this one to bed early last year, but amazingly it continues to gain momentum. Here’s why I’m not buying it.

There are four things you absolutely have to know and do if there’s any possibility you’ll be involved in bootstrapping a startup:

1) Define and build your minimum viable product.
2) Aggressively keep your run rate as low as possible.
3) Iterate to find your product/market fit.
4) Don’t die.

If you live in a world where the flavor of the week is the most important thing — and I get it, I’ve worked there myself — then failing fast is a great strategy. Maximize your “successes” and get out while you’re on top.

But if I’ve done my marketing homework, the monthly bills are manageable, and I’m iterating toward market adoption, then as far as I’m concerned those “fail fast” guys can pry my startup from my cold, dead hands.

Enter the ( SEO ) Matrix

There are two very commonly known things about search engine optimization (SEO).

1) Great, targeted content drives PageRank.
2) Google’s page rank algorithm is recursive.

Given how common this knowledge is, it’s surprising that so many Web sites haven’t put 1 + 1 together: Your pages earn a PageRank based on content and links, and then they lend that PageRank power to your other pages based on how you link to them.

In effect, your internal link structure may be causing your most powerful, highest-ranked pages to spread their PageRank weight across your entire site, instead of to the specific campaigns or initiatives you mean to support with those individual pages.

The takeaway: Instead of thinking of your SEO efforts as a collection of channels (FaceBook, Twitter, YouTube, and the obligatory blog) meant to drive content — think of SEO as a structured set of Content Tiers, each of which are built and interlinked to support your marketing and SEO objectives.

Enter: the SEO Matrix.

Continue reading “Enter the ( SEO ) Matrix”

The Organizational Roles of Reporting vs. Analysis: Determining Your Mix

When establishing, expanding, or staffing an analytics and/or reporting function within your organization, there are a few key considerations that will strongly impact the types of services the group can provide, how the group provides value to the firm, what skillsets should be considered for each role, and what kinds of tools will be required in order to adequately support your Analytics function. Getting this right isn’t difficult, provided there’s an honest assessment of what this role’s major responsibilities will be. A small amount of effort in planning and staffing ensures that your organization gets the timely operational information it needs to make day-to-day tactical decisions while leaving you positioned to deliver strategic insights and shape the Analytics function into a revenue-producing powerhouse.

Continue reading “The Organizational Roles of Reporting vs. Analysis: Determining Your Mix”

SEM for Longer Time-to-Convert Businesses: How to Stop the Bleeding

I’m always surprised that you don’t see more direct marketers hired into Web analytics roles — they’re expert in a number of skills such as predictive modeling, experimental design, and statistical analysis that really turn the Web analytics role into a strategic source of business revenue. Along those lines, I wanted to give an example of how a tried-and-true direct marketing technique — using regression modeling to predict how likely each user is to convert — can allow media buyers and SEM managers to significantly reduce the amount of time it takes to determine which campaigns are winners and campaigns are losers.

Companies rely on pay-per-click and online ad buys to drive growth. While buying clicks is easy, ensuring that individual campaigns are profitable – before significant losses have been incurred – presents a host of challenges to media buyers. Continue reading “SEM for Longer Time-to-Convert Businesses: How to Stop the Bleeding”

How Non-Compete Agreements Really Work

Part of the ritual of starting a new job is signing all the paperwork: the acceptance letter, I-9, W-4, insurance forms, direct deposit.

Oh, and the non-compete and confidentiality agreements.

When we accept an offer for a new job, signing a non-compete agreement as a condition of employment seems like a footnote at the end of a long chapter… something that surely we don’t need to actually read. Non-competes are ubiquitous — and, we assume, mostly harmless. However, these agreements are so common (and so one-sided) that I wanted to spend a few minutes describing how they really work in practice. Continue reading “How Non-Compete Agreements Really Work”