Time processing of Python Pandas advanced tutorial
- 2021-12-05 06:42:28
- OfStack
date_range and bdate_range
origin
Formatting
Period DateOffset As index Slicing and exact matching
Operation of time series Shifting
Frequency conversion
Resampling resampling Summarize
Brief introduction
Time should be a data type often used in data processing. In addition to datetime64 and timedelta64 in Numpy, pandas also integrates the functions of other python libraries such as scikits. timeseries.
Time classification
There are four types of time in pandas:
Date times: Date and time, with time zone. Similar to datetime. datetime in the standard library. Time deltas: Absolute duration, similar to datetime. timedelta in the standard library. Time spans: The time span defined by the point in time and its associated frequency. Date offsets: Calendar-based time is similar to dateutil. relativedelta. relativedelta.Let's use a table to represent:
类型 | 标量class | 数组class | pandas数据类型 | 主要创建方法 |
---|---|---|---|---|
Date times | Timestamp | DatetimeIndex | datetime64[ns] or datetime64[ns, tz] | to_datetime or date_range |
Time deltas | Timedelta | TimedeltaIndex | timedelta64[ns] | to_timedelta or timedelta_range |
Time spans | Period | PeriodIndex | period[freq] | Period or period_range |
Date offsets | DateOffset | None | None | DateOffset |
Look at an example of use:
In [19]: pd.Series(range(3), index=pd.date_range("2000", freq="D", periods=3))
Out[19]:
2000-01-01 0
2000-01-02 1
2000-01-03 2
Freq: D, dtype: int64
Look at the null value of the above data type under 1:
In [24]: pd.Timestamp(pd.NaT)
Out[24]: NaT
In [25]: pd.Timedelta(pd.NaT)
Out[25]: NaT
In [26]: pd.Period(pd.NaT)
Out[26]: NaT
# Equality acts as np.nan would
In [27]: pd.NaT == pd.NaT
Out[27]: False
Timestamp
Timestamp is the most basic time type, so we can create it like this:
In [28]: pd.Timestamp(datetime.datetime(2012, 5, 1))
Out[28]: Timestamp('2012-05-01 00:00:00')
In [29]: pd.Timestamp("2012-05-01")
Out[29]: Timestamp('2012-05-01 00:00:00')
In [30]: pd.Timestamp(2012, 5, 1)
Out[30]: Timestamp('2012-05-01 00:00:00')
DatetimeIndex
Timestamp is automatically converted to DatetimeIndex as index:
In [33]: dates = [
....: pd.Timestamp("2012-05-01"),
....: pd.Timestamp("2012-05-02"),
....: pd.Timestamp("2012-05-03"),
....: ]
....:
In [34]: ts = pd.Series(np.random.randn(3), dates)
In [35]: type(ts.index)
Out[35]: pandas.core.indexes.datetimes.DatetimeIndex
In [36]: ts.index
Out[36]: DatetimeIndex(['2012-05-01', '2012-05-02', '2012-05-03'], dtype='datetime64[ns]', freq=None)
In [37]: ts
Out[37]:
2012-05-01 0.469112
2012-05-02 -0.282863
2012-05-03 -1.509059
dtype: float64
date_range and bdate_range
You can also use date_range to create DatetimeIndex:
In [74]: start = datetime.datetime(2011, 1, 1)
In [75]: end = datetime.datetime(2012, 1, 1)
In [76]: index = pd.date_range(start, end)
In [77]: index
Out[77]:
DatetimeIndex(['2011-01-01', '2011-01-02', '2011-01-03', '2011-01-04',
'2011-01-05', '2011-01-06', '2011-01-07', '2011-01-08',
'2011-01-09', '2011-01-10',
...
'2011-12-23', '2011-12-24', '2011-12-25', '2011-12-26',
'2011-12-27', '2011-12-28', '2011-12-29', '2011-12-30',
'2011-12-31', '2012-01-01'],
dtype='datetime64[ns]', length=366, freq='D')
date_range is the calendar range, bdate_range is the workday range:
In [78]: index = pd.bdate_range(start, end)
In [79]: index
Out[79]:
DatetimeIndex(['2011-01-03', '2011-01-04', '2011-01-05', '2011-01-06',
'2011-01-07', '2011-01-10', '2011-01-11', '2011-01-12',
'2011-01-13', '2011-01-14',
...
'2011-12-19', '2011-12-20', '2011-12-21', '2011-12-22',
'2011-12-23', '2011-12-26', '2011-12-27', '2011-12-28',
'2011-12-29', '2011-12-30'],
dtype='datetime64[ns]', length=260, freq='B')
Both methods can take start, end, and periods parameters.
In [84]: pd.bdate_range(end=end, periods=20)
In [83]: pd.date_range(start, end, freq="W")
In [86]: pd.date_range("2018-01-01", "2018-01-05", periods=5)
origin
Using the origin parameter, you can modify the starting point of DatetimeIndex:
In [67]: pd.to_datetime([1, 2, 3], unit="D", origin=pd.Timestamp("1960-01-01"))
Out[67]: DatetimeIndex(['1960-01-02', '1960-01-03', '1960-01-04'], dtype='datetime64[ns]', freq=None)
By default, origin = 'unix', which means the starting point is 000:00, 1970-01-01.
In [68]: pd.to_datetime([1, 2, 3], unit="D")
Out[68]: DatetimeIndex(['1970-01-02', '1970-01-03', '1970-01-04'], dtype='datetime64[ns]', freq=None)
Formatting
The time can be formatted using the format parameter:
In [51]: pd.to_datetime("2010/11/12", format="%Y/%m/%d")
Out[51]: Timestamp('2010-11-12 00:00:00')
In [52]: pd.to_datetime("12-11-2010 00:00", format="%d-%m-%Y %H:%M")
Out[52]: Timestamp('2010-11-12 00:00:00')
Period
Period represents a time span, which is usually used from freq1:
In [24]: pd.Timestamp(pd.NaT)
Out[24]: NaT
In [25]: pd.Timedelta(pd.NaT)
Out[25]: NaT
In [26]: pd.Period(pd.NaT)
Out[26]: NaT
# Equality acts as np.nan would
In [27]: pd.NaT == pd.NaT
Out[27]: False
0
Period can be operated directly:
In [345]: p = pd.Period("2012", freq="A-DEC")
In [346]: p + 1
Out[346]: Period('2013', 'A-DEC')
In [347]: p - 3
Out[347]: Period('2009', 'A-DEC')
In [348]: p = pd.Period("2012-01", freq="2M")
In [349]: p + 2
Out[349]: Period('2012-05', '2M')
In [350]: p - 1
Out[350]: Period('2011-11', '2M')
Note that Period can only be arithmetic if it has the same freq. Including offsets and timedelta
In [352]: p = pd.Period("2014-07-01 09:00", freq="H")
In [353]: p + pd.offsets.Hour(2)
Out[353]: Period('2014-07-01 11:00', 'H')
In [354]: p + datetime.timedelta(minutes=120)
Out[354]: Period('2014-07-01 11:00', 'H')
In [355]: p + np.timedelta64(7200, "s")
Out[355]: Period('2014-07-01 11:00', 'H')
Period as index can be automatically converted to PeriodIndex:
In [24]: pd.Timestamp(pd.NaT)
Out[24]: NaT
In [25]: pd.Timedelta(pd.NaT)
Out[25]: NaT
In [26]: pd.Period(pd.NaT)
Out[26]: NaT
# Equality acts as np.nan would
In [27]: pd.NaT == pd.NaT
Out[27]: False
3
PeriodIndex can be created using the pd.period_range method:
In [24]: pd.Timestamp(pd.NaT)
Out[24]: NaT
In [25]: pd.Timedelta(pd.NaT)
Out[25]: NaT
In [26]: pd.Period(pd.NaT)
Out[26]: NaT
# Equality acts as np.nan would
In [27]: pd.NaT == pd.NaT
Out[27]: False
4
It can also be created directly through PeriodIndex:
In [24]: pd.Timestamp(pd.NaT)
Out[24]: NaT
In [25]: pd.Timedelta(pd.NaT)
Out[25]: NaT
In [26]: pd.Period(pd.NaT)
Out[26]: NaT
# Equality acts as np.nan would
In [27]: pd.NaT == pd.NaT
Out[27]: False
5
DateOffset
DateOffset represents a frequency object. It is similar to Timedelta in that it represents 1 duration, but has special calendar rules. For example, in Timedelta1, a day must be 24 hours, while in DateOffset, a day may have 23, 24 or 25 hours depending on daylight saving time.
In [24]: pd.Timestamp(pd.NaT)
Out[24]: NaT
In [25]: pd.Timedelta(pd.NaT)
Out[25]: NaT
In [26]: pd.Period(pd.NaT)
Out[26]: NaT
# Equality acts as np.nan would
In [27]: pd.NaT == pd.NaT
Out[27]: False
6
The DateOffsets and Frequency operations are turned off first. Look at the available Date Offset and its associated Frequency under 1:
Date Offset | Frequency String | 描述 |
---|---|---|
DateOffset | None | 通用的offset 类 |
BDay or BusinessDay | 'B' | 工作日 |
CDay or CustomBusinessDay | 'C' | 自定义的工作日 |
Week | 'W' | 1周 |
WeekOfMonth | 'WOM' | 每个月的第几周的第几天 |
LastWeekOfMonth | 'LWOM' | 每个月最后1周的第几天 |
MonthEnd | 'M' | 日历月末 |
MonthBegin | 'MS' | 日历月初 |
BMonthEnd or BusinessMonthEnd | 'BM' | 营业月底 |
BMonthBegin or BusinessMonthBegin | 'BMS' | 营业月初 |
CBMonthEnd or CustomBusinessMonthEnd | 'CBM' | 自定义营业月底 |
CBMonthBegin or CustomBusinessMonthBegin | 'CBMS' | 自定义营业月初 |
SemiMonthEnd | 'SM' | 日历月末的第15天 |
SemiMonthBegin | 'SMS' | 日历月初的第15天 |
QuarterEnd | 'Q' | 日历季末 |
QuarterBegin | 'QS' | 日历季初 |
BQuarterEnd | 'BQ | 工作季末 |
BQuarterBegin | 'BQS' | 工作季初 |
FY5253Quarter | 'REQ' | 零售季( 52-53 week) |
YearEnd | 'A' | 日历年末 |
YearBegin | 'AS' or 'BYS' | 日历年初 |
BYearEnd | 'BA' | 营业年末 |
BYearBegin | 'BAS' | 营业年初 |
FY5253 | 'RE' | 零售年 (aka 52-53 week) |
Easter | None | 复活节假期 |
BusinessHour | 'BH' | business hour |
CustomBusinessHour | 'CBH' | custom business hour |
Day | 'D' | 1天的绝对时间 |
Hour | 'H' | 1小时 |
Minute | 'T' or 'min' | 1分钟 |
Second | 'S' | 1秒钟 |
Milli | 'L' or 'ms' | 1微妙 |
Micro | 'U' or 'us' | 1毫秒 |
Nano | 'N' | 1纳秒 |
DateOffset also has two methods, rollforward () and rollback (), to move time:
In [153]: ts = pd.Timestamp("2018-01-06 00:00:00")
In [154]: ts.day_name()
Out[154]: 'Saturday'
# BusinessHour's valid offset dates are Monday through Friday
In [155]: offset = pd.offsets.BusinessHour(start="09:00")
# Bring the date to the closest offset date (Monday)
In [156]: offset.rollforward(ts)
Out[156]: Timestamp('2018-01-08 09:00:00')
# Date is brought to the closest offset date first and then the hour is added
In [157]: ts + offset
Out[157]: Timestamp('2018-01-08 10:00:00')
The above operation automatically saves information such as hours and minutes. If you want to set it to 00:00:00, you can call the normalize () method:
In [158]: ts = pd.Timestamp("2014-01-01 09:00")
In [159]: day = pd.offsets.Day()
In [160]: day.apply(ts)
Out[160]: Timestamp('2014-01-02 09:00:00')
In [161]: day.apply(ts).normalize()
Out[161]: Timestamp('2014-01-02 00:00:00')
In [162]: ts = pd.Timestamp("2014-01-01 22:00")
In [163]: hour = pd.offsets.Hour()
In [164]: hour.apply(ts)
Out[164]: Timestamp('2014-01-01 23:00:00')
In [165]: hour.apply(ts).normalize()
Out[165]: Timestamp('2014-01-01 00:00:00')
In [166]: hour.apply(pd.Timestamp("2014-01-01 23:30")).normalize()
Out[166]: Timestamp('2014-01-02 00:00:00')
As index
Time can be used as an index, and as an index, there will be a few convenient features.
You can use time directly to get the corresponding data:
In [24]: pd.Timestamp(pd.NaT)
Out[24]: NaT
In [25]: pd.Timedelta(pd.NaT)
Out[25]: NaT
In [26]: pd.Period(pd.NaT)
Out[26]: NaT
# Equality acts as np.nan would
In [27]: pd.NaT == pd.NaT
Out[27]: False
9
Get data for the whole year:
In [102]: ts["2011"]
Out[102]:
2011-01-31 0.119209
2011-02-28 -1.044236
2011-03-31 -0.861849
2011-04-29 -2.104569
2011-05-31 -0.494929
2011-06-30 1.071804
2011-07-29 0.721555
2011-08-31 -0.706771
2011-09-30 -1.039575
2011-10-31 0.271860
2011-11-30 -0.424972
2011-12-30 0.567020
Freq: BM, dtype: float64
Get data for a month:
In [103]: ts["2011-6"]
Out[103]:
2011-06-30 1.071804
Freq: BM, dtype: float64
DF can accept time as a parameter of loc:
In [105]: dft
Out[105]:
A
2013-01-01 00:00:00 0.276232
2013-01-01 00:01:00 -1.087401
2013-01-01 00:02:00 -0.673690
2013-01-01 00:03:00 0.113648
2013-01-01 00:04:00 -1.478427
... ...
2013-03-11 10:35:00 -0.747967
2013-03-11 10:36:00 -0.034523
2013-03-11 10:37:00 -0.201754
2013-03-11 10:38:00 -1.509067
2013-03-11 10:39:00 -1.693043
[100000 rows x 1 columns]
In [106]: dft.loc["2013"]
Out[106]:
A
2013-01-01 00:00:00 0.276232
2013-01-01 00:01:00 -1.087401
2013-01-01 00:02:00 -0.673690
2013-01-01 00:03:00 0.113648
2013-01-01 00:04:00 -1.478427
... ...
2013-03-11 10:35:00 -0.747967
2013-03-11 10:36:00 -0.034523
2013-03-11 10:37:00 -0.201754
2013-03-11 10:38:00 -1.509067
2013-03-11 10:39:00 -1.693043
[100000 rows x 1 columns]
Time slice:
In [107]: dft["2013-1":"2013-2"]
Out[107]:
A
2013-01-01 00:00:00 0.276232
2013-01-01 00:01:00 -1.087401
2013-01-01 00:02:00 -0.673690
2013-01-01 00:03:00 0.113648
2013-01-01 00:04:00 -1.478427
... ...
2013-02-28 23:55:00 0.850929
2013-02-28 23:56:00 0.976712
2013-02-28 23:57:00 -2.693884
2013-02-28 23:58:00 -1.575535
2013-02-28 23:59:00 -1.573517
[84960 rows x 1 columns]
Slicing and exact matching
Consider the following 1 precision Series object:
In [120]: series_minute = pd.Series(
.....: [1, 2, 3],
.....: pd.DatetimeIndex(
.....: ["2011-12-31 23:59:00", "2012-01-01 00:00:00", "2012-01-01 00:02:00"]
.....: ),
.....: )
.....:
In [121]: series_minute.index.resolution
Out[121]: 'minute'
If the time accuracy is less than minutes, an Series object is returned:
In [122]: series_minute["2011-12-31 23"]
Out[122]:
2011-12-31 23:59:00 1
dtype: int64
If the time accuracy is greater than minutes, a constant is returned:
In [123]: series_minute["2011-12-31 23:59"]
Out[123]: 1
In [124]: series_minute["2011-12-31 23:59:00"]
Out[124]: 1
Similarly, if the precision is seconds, 1 object will be returned if it is less than seconds, and a constant value will be returned if it is equal to seconds.
Operation of time series
Shifting
The shift method allows time series to move accordingly:
In [275]: ts = pd.Series(range(len(rng)), index=rng)
In [276]: ts = ts[:5]
In [277]: ts.shift(1)
Out[277]:
2012-01-01 NaN
2012-01-02 0.0
2012-01-03 1.0
Freq: D, dtype: float64
By specifying freq, you can set the mode of shift:
In [278]: ts.shift(5, freq="D")
Out[278]:
2012-01-06 0
2012-01-07 1
2012-01-08 2
Freq: D, dtype: int64
In [279]: ts.shift(5, freq=pd.offsets.BDay())
Out[279]:
2012-01-06 0
2012-01-09 1
2012-01-10 2
dtype: int64
In [280]: ts.shift(5, freq="BM")
Out[280]:
2012-05-31 0
2012-05-31 1
2012-05-31 2
dtype: int64
Frequency conversion
Time series can be converted in frequency by calling the method of asfreq:
In [281]: dr = pd.date_range("1/1/2010", periods=3, freq=3 * pd.offsets.BDay())
In [282]: ts = pd.Series(np.random.randn(3), index=dr)
In [283]: ts
Out[283]:
2010-01-01 1.494522
2010-01-06 -0.778425
2010-01-11 -0.253355
Freq: 3B, dtype: float64
In [284]: ts.asfreq(pd.offsets.BDay())
Out[284]:
2010-01-01 1.494522
2010-01-04 NaN
2010-01-05 NaN
2010-01-06 -0.778425
2010-01-07 NaN
2010-01-08 NaN
2010-01-11 -0.253355
Freq: B, dtype: float64
asfreq can also specify the padding method after the modification frequency:
In [285]: ts.asfreq(pd.offsets.BDay(), method="pad")
Out[285]:
2010-01-01 1.494522
2010-01-04 1.494522
2010-01-05 1.494522
2010-01-06 -0.778425
2010-01-07 -0.778425
2010-01-08 -0.778425
2010-01-11 -0.253355
Freq: B, dtype: float64
Resampling resampling
The given time series can be resampled by calling the resample method:
In [286]: rng = pd.date_range("1/1/2012", periods=100, freq="S")
In [287]: ts = pd.Series(np.random.randint(0, 500, len(rng)), index=rng)
In [288]: ts.resample("5Min").sum()
Out[288]:
2012-01-01 25103
Freq: 5T, dtype: int64
resample can accept various statistical methods, such as sum, mean, std, sem, max, min, median, first, last and ohlc.
In [289]: ts.resample("5Min").mean()
Out[289]:
2012-01-01 251.03
Freq: 5T, dtype: float64
In [290]: ts.resample("5Min").ohlc()
Out[290]:
open high low close
2012-01-01 308 460 9 205
In [291]: ts.resample("5Min").max()
Out[291]:
2012-01-01 460
Freq: 5T, dtype: int64