How to use Parquet output format for data lake destinations

Parquet output format makes it easy to set up data pipelines for data lakes. Parquet is more efficient than CSV for storing and querying the data, and it makes processing the data easy as it contains metadata such as the data types of each field.

You can select the Parquet output format for cloud storage destinations when setting up a destination on the Supermetrics Hub.

Instructions

1. Log in to the Supermetrics Hub.
2. In the sidebar, go to Storage → Storage destinations.
3. Click Add storage, or if you have already set up some destinations, click New storage.
4. Fill in the necessary details as you would normally do. Take a look at our prerequisite and configuration guides for various data warehouse destinations.
5. Set a unique upload path for the data, avoiding conflicts with existing destinations.
6. In the Output format dropdown, select Parquet as the output format for the destination. This setting is only visible for cloud storage destinations, not data warehouses.
7. Click Save to apply the changes.
8. Create a transfer to your data lake destination. The standard steps for creating transfers apply, so as usual, specify the source, destination, and any additional transfer settings required.