tecton.declarative.DatetimePartitionColumn

class tecton.declarative.DatetimePartitionColumn(column_name, datepart, zero_padded)

Helper class to tell Tecton how underlying flat files are date/time partitioned for Hive/Glue data sources. This can translate into a significant performance increase.

You will generally include an object of this class in the datetime_partition_columns option in a HiveConfig object.

Methods

__init__

Instantiates a new DatetimePartitionColumn configuration object.

__init__(column_name, datepart, zero_padded)

Instantiates a new DatetimePartitionColumn configuration object.

Parameters
  • column_name – The name of the column in the Glue/Hive schema that corresponds to the underlying date/time partition folder. Note that if you do not explicitly specify a name in your partition folders, Glue will name the column of the form partition_0.

  • datepart – The part of the date that this column specifies. Can be one of “year”, “month”, “day”, “hour”, or the full “date”.

  • zero_padded – Whether the datepart has a leading zero if less than two digits. This must be set to True if datepart = date.

Example definition:

Assume you have an S3 bucket with parquet files stored in the following structure: s3://mybucket/2022/05/04/<multiple parquet files> , where 2022 is the year, 05 is the month, and 04 is the day of the month. In this scenario, you could use the following definition:

datetime_partition_columns = [
    DatetimePartitionColumn(column_name="partition_0", datepart="year", zero_padded=True),
    DatetimePartitionColumn(column_name="partition_1", datepart="month", zero_padded=True),
    DatetimePartitionColumn(column_name="partition_2", datepart="day", zero_padded=True),
]
Returns

DatetimePartitionColumn instantiation