How to approach A/B testing and controlled launch of experimental features for product-led growth
What is A/B testing and how is it useful for product growth?
A/B testing is a product methodology for doing research on feature adoption, from small buttons to large-scale features. We do it to get insight on how each variant performs and which variant has the highest success ratio of feature adoption.
There are different ways to get the user’s attention on a new feature in the product, for example an in-product tour or a nudge. Based on the user’s interaction with these nudges, the A/B app will suggest which feature or experiment has a higher success ratio. These insights are mostly tracked using Heap or other third-party analytics services; on occasion, in-house trackers are built.
Heap is the industry best, and its wide range of insights help us get even better user attention on product details.
How is our A/B service different from traditional A/B services?
Here’s how we use A/B services in Freshworks products:
The isolated data approach
Moving feature configs/rules outside the product as a service.
We have several combinations of properties based on which users will view a particular feature. Some of the common and generic properties that apply to almost all products are:
- Email: generic or business
- Region: U.S. or non-U.S.
- Account state
- Trial
- Active
- Free
- Suspended
- Monthly recurring revenue (MRR) range
- Plan subscribed to
- A/B required
- When A/B required check is enabled, it randomizes the truth based on even/odd account IDs.
- Targeted IDs
The rule will contain all these parameters plus a feature key that uniquely identifies the feature.
Key takeaways while moving the configs outside the product
- We don’t have to wait for the deployment cycle
- Code independence
- Lighter app
- No live sync required
- No data is shared externally
1. We don’t have to wait for the deployment cycle
Usually, we show a feature based on some conditions of the above mentioned properties. When we need to change these conditions, we have to wait and change only in the deployment cycle. The flexibility is missing.
In our method, we manage the configs outside the codebase, and it can be updated using a specific GUI. Now, it is easier for the product to make changes directly to the rules, and this is reflected swiftly in the product.
2. Code independence
Now that the config app has a UI on its own, the product team can get access with levels to control the availability of the feature (features that are not tied to the plan).
This will improve the speed of our experiments, and we can track them in Heap to take actions quickly based on the insights. This helps strengthen and speed up our decision-making.
3. Lighter app
The app is very light since it is only holding configurations and keys. These will create edit and modifiable access, as per the roles.
To be in sync, a dev also will get access to this platform since they will use this key from the app to code conditions.
4. No sync required
In the traditional approach, account information or account attributes are attached to these feature keys, this requires a sync between the feature app and the product instance. For example when a customer upgrades from a plan A to plan B, account attributes have changed, hence this will cause a reliability issue with the feature app as the app holds plan info as plan A. Hence a sync is required in the traditional approach.
Our app does not require sync because our app provides the applicable features based on the parameter sent to the app, also the changes in account and subscription properties happen periodically hence the sync isn’t required in our case
The product holds parameters like account ID, MRR, plan, and account state. At the point of deciding which features are applicable for the account, a call is made and kept in the store. Once in every cycle, it is fetched again when required. Hence, the app does not require sync.
5. No data is shared externally
Sometimes we have policies that prevent user data being shared outside the product or account. So we have two approaches here:
- Send account information and get the applicable features
- Get the configs and store in the app database, then figure out applicable features later
The first option can be used if there are no rules pertaining to data being shared externally.
The second option can be used when data needs to stay within the app. The config will be fetched and the applicable features will be arrived upon.
A basic architecture of the app
There are three major entities that contribute to this process.
- The config app or the A/B testing framework
- Frontend service that interacts with features based on the keys present
- The backend app that will provide the keys from the A/B app
We’ve covered the backend A/B service/app. Let’s see the frontend service.
When does the config data get fetched for each account?
Service is a concept that is present in Ember.js and is being leveraged for our scenarios here. A service is a singleton that carries a set of values globally across the app.
Our service is part of the frontend app. It will be called during the app initialization, then it fetches the information from the config app and gets the applicable features.
There is a method as part of the same service mentioned above to check if a given feature example “support_feature” is present in the list of applicable features for this account. This results in a Boolean value that decides whether to show the feature in the product and how to display it, thereby enabling the app to act differently based on the response from the config app.
The product team can change the config data so the app will pull data for every five route changes or any other desired period, so the data is in sync with the config app.
Scalability
The sync can be scaled up to a push notification level where an event can be triggered to the backend app. This invokes a change in the frontend app to get the changes from the config app. There are several ways to get this synced up on a live basis. This is a scope that is dependent on each product’s scenarios and how likely the features are to be changed.
Handling other features and use cases
Since we have several different parameters based on which feature is displayed, we have a lot of scope to do a controlled experiment.
- Controlled launch
- Beta launches
- For specific accounts or for other params
- Kill switch
- Disabling features when an error occurs
Future scope
There can be several new params added, which can support new scenarios.
- Scheduled release
- Automated release
- Where the feature experiment has a time limit—for example, a year-in-review feature
- Sentiment-based features
Another great feature is to track and report the experiment features to be automated. If tracking is enabled, the performance of these flows should be gathered from the Heap app.
This can be later translated into insights, and those can be automated as well.
