Keep internal tests out of activation metrics without erasing useful data
Separate internal tests from customer activation with explicit account classifications, consistent denominators and reversible reports, using a worked example.
In this article
Keep internal tests out of customer activation reports by classifying the relevant accounts and applying the same exclusion to the numerator and denominator. Preserve a way to inspect the excluded activity. Test traffic can be useful for debugging even when it does not belong in a customer adoption measure.
Start with a reporting definition before changing event collection. A reversible report filter lets you inspect the effect and correct mistakes. A permanent collection or processing exclusion can remove information you later need.
The goal is an interpretable customer cohort. Removing awkward-looking accounts because they reduce the rate is not a valid classification rule.
Define which activity does not represent a customer
Use a small set of explicit categories that match the product’s operating reality. Internal quality-assurance workspaces, automated smoke tests and sales demonstration accounts may all be outside a customer activation cohort, but they have different reasons for existing.
Record the reason, who owns the classification and when it became effective. Prefer a reliable account or environment attribute over a guess based only on a name. A personal email address can belong to an employee testing the product; a company-domain address can belong to a real customer organization.
Decide the unit before filtering. If activation is measured per workspace, removing one employee’s events may leave the employee-created test workspace in the denominator. If activation is per person, excluding an entire company can remove legitimate users unnecessarily.
For an account-level measure, distinguish the account’s purpose from the actor performing an event. A support employee helping a real customer does not automatically turn that customer’s workspace into a test account. Whether assisted activation belongs in the main measure is a separate definition to document.
A worked example: the number can move in either direction
The following cohort is entirely synthetic. It represents 20 newly created accounts, each observed for the full seven-day activation window. Activation means completing the same defined first-value event. These are illustrative counts, not product analytics or a benchmark.
| Account classification | Accounts | Activated within the window | Main customer report |
|---|---|---|---|
| Confirmed customer | 12 | 6 | Include |
| Internal quality assurance | 4 | 4 | Exclude |
| Sales demonstration | 2 | 2 | Exclude |
| Purpose not yet confirmed | 2 | 0 | Include pending investigation |
| All accounts | 20 | 12 | Retain as the comparison view |
The unfiltered rate is 12 divided by 20, or 60%. Removing the six confirmed internal and demonstration accounts also removes their six activations. The customer report is therefore 6 divided by 14, approximately 42.9%.
The decrease is expected in this example: internal testers completed the task more consistently than customers. With a different test cohort, the effect could go the other way. Neither direction proves that the underlying product improved or worsened.
A common mistake is to subtract tests only from the denominator. That produces 12 divided by 14, approximately 85.7%, while retaining activations from accounts the report claims to exclude. The rate is internally inconsistent.
Dropping the two unknown accounts would produce 6 divided by 12, or 50%. There is no justification for that exclusion in this example. Unknown purpose is a review queue, not evidence that an account is internal. Keep the main calculation conservative until classification is established.
Choose a report filter before changing collection
Google Analytics’ data-filter documentation makes a consequential distinction: data filters affect incoming events, do not rewrite historical data and can permanently prevent excluded data from being available in Analytics or BigQuery. It points to report filters when the objective is to hide data in particular reports.
For an initial cleanup, retain the original view and create an explicitly named customer activation view. Compare the included and excluded cohorts before changing a shared default. This preserves the ability to notice a mistaken rule, such as a contractor domain that also appears among customers.
There may be good reasons to separate development environments or prevent unwanted data from being collected. Make that a deliberate collection decision with an owner and verified scope. Do not use permanent exclusion merely because the dashboard looks untidy.
A reporting filter also does not replace retention and privacy rules. Keep data only where there is an appropriate purpose and policy. The point is to avoid accidental loss during metric cleanup, not to retain every event indefinitely.
Verify the filter on the reports people use
PostHog’s internal-user guide describes project-specific rules and an insight toggle that applies them. Defining a filter does not enable it. The documented default for new insights also leaves existing insights unchanged, and SQL insights need their own query exclusion.
That creates a practical review task: inspect the headline activation report, its supporting funnel and any exported query used in weekly reporting. A correct project setting does not establish that every existing analysis uses it.
For the synthetic cohort, all three views should agree on the same 14 eligible accounts and six activations when the customer filter is applied. A difference should lead to investigation of the unit, date window, identity mapping or filter scope.
Keep changes to shared reports reviewable. PostHog also warns that its bulk insight operation has no automatic undo that restores every prior setting. Save the relevant previous definitions before making widespread changes. Reversible filtering and reversible configuration changes are related but distinct.
Leave an audit trail for classification changes
A classification can be wrong or can change. A demonstration workspace may later become a genuine customer workspace. An employee may create a production account for personal use. Decide whether the report uses classification at account creation, at event time or at analysis time.
That choice changes historical results. A current-property filter may restate an earlier cohort when the property is edited. If the team needs a reproducible historical view, preserve the rule and effective classification needed to reconstruct it.
When a rule changes, record the reason and show the effect on the recent reporting period. Label a revised series as revised rather than presenting the discontinuity as a product gain. Keep the original definition accessible to the people interpreting the trend.
Keep excluded activity useful
Internal failures can reveal broken events, setup regressions and confusing recovery paths. Maintain a diagnostic view for the test accounts so that removing them from customer activation does not remove them from attention.
Pair the quantitative cleanup with first-time user research when you need to understand why customers stop. A cleaner denominator makes the metric easier to interpret; it does not explain the behavior by itself.
The reporting handoff should state the unit, activation event, observation window, excluded categories and treatment of unknown accounts. Anyone reading the number should be able to reproduce the cohort without relying on the person who built the chart.
