You can apply the data filters used in the Table Manager directly to your transfers, making your transfers faster, more efficient, and more cost-effective.
With transfer-level filtering, you can:
- Increase efficiency: Transfer smaller, more relevant datasets, which speeds up transfer times.
- Reduce costs: Lower your data storage costs by not moving or storing unnecessary data in your destination (like Snowflake, Google BigQuery, or Amazon S3).
- Get cleaner data: Your data arrives in your destination pre-filtered and ready for analysis, with no need for extra cleaning steps.
Note that transfer-level filtering is off by default for all new and existing transfers. It won't change any of your current data pipelines unless you choose to enable it.
There are two main ways to use the transfer-level filtering feature: when setting up a new transfer or when editing an existing one. In both cases, the first step is to ensure you have filters set up in the Table Manager.
Instructions
Step 1: Set up your filters in Table Manager
Before you can apply filters in a transfer, you must first create them for the specific table you want to filter.
- Log in to the Supermetrics Hub.
- In the sidebar, click Storage → Table manager.
- Select the table you want to filter and click the table to edit it.
- Go to the Filter section.
- Add the filter conditions you need (such as, Campaign status EQUALS Active).
- Click Save.
Now this table is ready to have its filters applied in any data transfer.
Step 2: Apply filters in data transfer
You can now head over to your transfer configuration to apply the filters you just set up.
Scenario A: Creating a new transfer
- On the Supermetrics Hub, go to Storage → Transfers.
- Click New transfer.
- Proceed through the setup process. In the Configuration step, scroll down to the "Tables" section.
- Find the table you created filters for in Step 1. Click the chevron icon (>) on the right-hand side of the table row to expand its details.
- In the Review fields tab, scroll down to the Filters section.
- Click the toggle next to Enable for this transfer. You'll see a confirmation message: "Data is being filtered."
- Once you complete the transfer setup and click Save, you'll see a checkmark icon in the Filtered column next to the table name on the transfer summary page. This indicates the filter is active.
Scenario B: Adding filters to an existing transfer
- On the Supermetrics Hub, go to Storage → Transfers.
- Find the transfer you want to modify.
- Click to the 3-dot icon and select Edit transfer.
- Go to the Configuration step and scroll down to your list of tables.
- Find the table you created filters for in Step 1. Click the chevron icon (>) on the right-hand side of the table row to expand its details.
- In the Review fields tab, scroll down to the Filters section.
- Click the toggle next to Enable for this transfer. You'll see a confirmation message: "Data is being filtered."
- Click Save to update the transfer. The filters will be applied on the next scheduled run.
What to expect when filtering your transfers
Here are some key things to know about how transfer-level filtering behaves.
How filters affect existing transfers and historical data
When you enable a filter on an existing transfer, it will apply to the data fetched on all future runs. The filter does not automatically go back and change the historical data that is already in your destination.
The behavior depends on your transfer's date range settings. For example, if your transfer is configured to fetch the last 30 days of data every day, the next time it runs, it will pull the most recent 30 days with the filter applied. This new, filtered data will replace the corresponding 30-day block of data in your destination. However, any data from 31 or more days ago will remain untouched and unfiltered.
Applying filters to your historical data (backfilling)
If you want to apply your new filter rules to the older data already in your destination, you'll need to perform a "backfill." A backfill simply means telling the transfer to re-run for a specific historical period.
To do this, you can manually trigger a new run for the transfer and set a custom historical date range that you want to re-fetch with the filters applied.
Will my transferred data always be smaller?
Usually, yes. The main benefit of filtering is to reduce the amount of data you transfer. However, the actual impact depends entirely on your data and the rules you set.
For example, if you create a filter for Campaign Status = Active but 99% of your campaigns are already active, you may not see a significant change in the number of rows or the size of the transferred data. The filter is working correctly, but it's only removing a very small slice of your data.
Filters are off by default
If a table has a filter defined in the Table Manager, it won't be automatically applied to any transfers using that table. You must manually enable the "Enable for this transfer" toggle for each specific table within each transfer's settings.
How multiple filter conditions work (AND/OR logic)
The filters you set in the Table Manager will respect the logic you define:
- AND Logic: If you have two conditions joined by AND (such as Status = Active AND Country = USA), only rows that meet both conditions will be included in the transfer.
- OR Logic: If you have two conditions joined by OR (such as Status = Active OR Status = Paused), rows that meet either one of the conditions will be included.
Filtering blended data sources
If you're using a blended data source, applying a filter to a field from just one of the tables will affect the entire joined row. If a row's data from the filtered table doesn't meet the condition, the entire combined row will be removed from the transfer.
Using private or inaccessible custom fields in filters
If a transfer is set up by User A with a filter on a custom field that is private to them, and then User B runs that same transfer, the filter on the private field can't be applied. The transfer will run successfully, but that specific private filter condition will be skipped. A warning message will appear in the transfer logs to notify you of this.
What happens if I remove filters from a table?
If you edit a table in the Table Manager and remove its filters, any transfer that was using those filters will no longer apply them. The "Filtered" checkmark for that table in the transfer configuration will disappear automatically, and on the next run, the data will be transferred without those filters.