Once you've set up a data transfer to your destination, you can monitor its status on the Supermetrics Hub.
This guide applies to all data warehouse destinations except BigQuery (marketplace). See separate instructions for monitoring BigQuery (marketplace) transfers.
Instructions
- Log in to the Hub.
- In the sidebar, go to Storage → Transfers.
- You can check out your last 7 transfer runs right away from the status icons. For more details and older transfer runs, click the status icons or select View details from the 3-dot menu.
Transfer views and statuses
Transfer view
The transfer view shows all the transfer runs that are scheduled for one transfer. By default, the view shows all transfer runs created in the past 30 days. You can go further back in time using the dropdown menu in the top right corner.
The table in the transfer view has the following columns:
- The Transfer run created date refers to the date when the transfer was scheduled to run.
- The Duration column tells you how long the transfer ran in total, excluding time spent in the queue.
- The Rows column tells you the number of rows in the data fetched by the transfer run.
- The Date of data column tells you which days data that transfer run fetched.
- The Run type column can have two values:
- Recurring for scheduled transfers that are run at predetermined intervals
- Backfill for historical backfills that are run on demand
- The Status column tells you what the status of the transfer run is.
For all date-partitioned transfers, Supermetrics creates a separate transfer run for each date of data.
For example, if you've scheduled a daily transfer with a refresh window of 7 days, you'll see 7 transfer runs with the same Transfer run created date, but a different Date of data. The Run type will be Recurring for each transfer run.
You can sort the table by Transfer run created date and Date of data. To find transfers that need your attention, use the filter toggle at the top of the table to only show transfers that have had issues in the selected time frame.
Transfer run statuses
The monitoring view shows transfer runs as they are created. Understanding the different stages of your transfer runs helps you monitor the pipeline.
Scheduled: In this phase, the transfer run is created in the system and waits in the queue until it's run. The transfer runs are created based on the schedule you have defined when setting up the transfer or at the time of setting a backfill. For example, when you create a daily transfer and set it to run at 2 a.m., the transfer run will be created after 2 a.m. every day. The time spent in the queue depends on the amount and size of other transfers scheduled to be run during that hour.
Running: During this phase, the transfer run is actively fetching data from the source. It can involve multiple data source requests, and the transfer remains in the running state until all requests are completed.
Paused: The transfer run is paused.
From the Running state, the transfer can move to 4 possible states:
Success: This means all data source requests were successful, and the transfer run was able to write data to the destination.
Error: The transfer run failed to fetch data. Any unrecoverable errors in data source requests result in a failed transfer.
Cancelled: The transfer run was cancelled, and it didn't write data to the destination.
Scheduled: Supermetrics has robust retry policies in place to recover from most data source API errors and other transfer failures. In such cases, the transfer run may return to the "Scheduled" state, awaiting a retry at a later time.
Transfer run details
To diagnose and resolve any issues with failed transfers, you can check the transfer run details. Clicking on any transfer run reveals more information about what caused the error.
Within the transfer run details, you'll find a list of all the jobs and their respective statuses. The view will show a separate job for each table, and a separate job for writing the data to the destination. You can see how long each job took and the number of rows of data returned.
For failed jobs, the Message column shows the specific error details related to the job.
Troubleshooting
If an error has occurred in the transfer, you'll see a red status indicator for that transfer. Find the error message and Run ID in the transfer details section. Follow the possible instructions in the error message, or contact our support for further assistance.