tecton.declarative.DatetimePartitionColumn¶
-
class
tecton.declarative.
DatetimePartitionColumn
(column_name, datepart, zero_padded)¶ Helper class to tell Tecton how underlying flat files are date/time partitioned for Hive/Glue data sources. This can translate into a significant performance increase.
You will generally include an object of this class in the datetime_partition_columns option in a HiveConfig object.
Methods
Instantiates a new DatetimePartitionColumn configuration object.
-
__init__
(column_name, datepart, zero_padded)¶ Instantiates a new DatetimePartitionColumn configuration object.
- Parameters
column_name – The name of the column in the Glue/Hive schema that corresponds to the underlying date/time partition folder. Note that if you do not explicitly specify a name in your partition folders, Glue will name the column of the form
partition_0
.datepart – The part of the date that this column specifies. Can be one of “year”, “month”, “day”, “hour”, or the full “date”.
zero_padded – Whether the
datepart
has a leading zero if less than two digits. This must be set to True ifdatepart
=date
.
Example definition:
Assume you have an S3 bucket with parquet files stored in the following structure:
s3://mybucket/2022/05/04/<multiple parquet files>
, where2022
is the year,05
is the month, and04
is the day of the month. In this scenario, you could use the following definition:datetime_partition_columns = [ DatetimePartitionColumn(column_name="partition_0", datepart="year", zero_padded=True), DatetimePartitionColumn(column_name="partition_1", datepart="month", zero_padded=True), DatetimePartitionColumn(column_name="partition_2", datepart="day", zero_padded=True), ]
- Returns
DatetimePartitionColumn instantiation
-