User guide
This page describes how to use the TSFrames package for timeseries data handling.
Installation
julia> using Pkg
julia> Pkg.add(url="https://github.com/xKDR/TSFrames.jl")Constructing TSFrame objects
After installing TSFrames you need to load the package in Julia environment. Then, create a basic TSFrame object.
julia> using TSFrames;julia> ts = TSFrame(1:10)10×1 TSFrame with Int64 Index Index x1 Int64 Int64 ────────────── 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10julia> ts.coredata10×2 DataFrame Row │ Index x1 │ Int64 Int64 ─────┼────────────── 1 │ 1 1 2 │ 2 2 3 │ 3 3 4 │ 4 4 5 │ 5 5 6 │ 6 6 7 │ 7 7 8 │ 8 8 9 │ 9 9 10 │ 10 10
The basic TSFrame constructor takes in a Vector of any type and automatically generates an index out of it (the Index column).
There are many ways to construct a TSFrame object. For real world applications you would want to read in a CSV file or download a dataset as a DataFrame and then operate on it. You can easily convert a DataFrame to a TSFrame object.
julia> using CSV, DataFrames, TSFrames, Datesjulia> dates = Date(2007, 1, 1):Day(1):Date(2008, 03, 06)Date("2007-01-01"):Dates.Day(1):Date("2008-03-06")julia> ts = TSFrame(DataFrame(Index=dates, value=10*rand(431)))431×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-01 5.3065 2007-01-02 4.86443 2007-01-03 4.53017 2007-01-04 9.19559 2007-01-05 9.04449 2007-01-06 5.07151 2007-01-07 4.79894 2007-01-08 6.23251 ⋮ ⋮ 2008-02-29 7.94481 2008-03-01 8.14029 2008-03-02 5.38964 2008-03-03 4.58327 2008-03-04 1.84667 2008-03-05 8.73657 2008-03-06 5.85365 416 rows omitted
In the above example you generate a random DataFrame and convert it into a TSFrame object ts. The top line of the ts object tells you the number of rows (431 here) and the number of columns (1) along with the Type of Index (Dates.Date in the above example).
You can also fetch the number of rows and columns by using nr(ts), nc(ts), and size(ts) methods. Respectively, they fetch the number of rows, columns, and a Tuple of row and column numbers. A length(::TSFrame) method is also provided for convenience which returns the number of rows of it's argument.
julia> nr(ts)431julia> nc(ts)1julia> size(ts)(431, 1)julia> length(ts)431
Names of data columns can be fetched using the names(ts) method which returns a Vector{String} object. The Index column can be fetched as an object of Vector type by using the index(ts) method, it can also be fetched directly using the underlying coredata property of TSFrame: ts.coredata[!, :Index].
julia> names(ts)1-element Vector{String}: "value"julia> index(ts)431-element Vector{Date}: 2007-01-01 2007-01-02 2007-01-03 2007-01-04 2007-01-05 2007-01-06 2007-01-07 2007-01-08 2007-01-09 2007-01-10 ⋮ 2008-02-27 2008-02-28 2008-02-29 2008-03-01 2008-03-02 2008-03-03 2008-03-04 2008-03-05 2008-03-06
Another simpler way to read a CSV is to pass TSFrame as a sink to the CSV.read function.
julia> ts = CSV.File(filename, TSFrame)Indexing and subsetting
One of the primary features of a timeseries package is to provide ways to index or subset a dataset using convenient interfaces. TSFrames makes it easier to index a TSFrame object by providing multiple intuitive getindex methods which work by just using the regular square parentheses([ ]).
julia> ts[1] # first row1×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-01 5.3065julia> ts[[3, 5], [1]] # third & fifth row, and first column2×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-03 4.53017 2007-01-05 9.04449julia> ts[1:10, 1] # first 10 rows and the first column as a vector10-element Vector{Float64}: 5.306503881966636 4.864426404474837 4.5301691220437785 9.195586027173078 9.044490183548072 5.07151239582835 4.798939210287169 6.232506101745746 9.227479907560694 2.185338126254596julia> ts[1, [:value]] # using the column name1×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-01 5.3065
Apart from integer-based row indexing and integer, name based column indexing, TSFrames provides special subsetting methods for date and time types defined inside the Dates module.
julia> ts[Date(2007, 1, 10)] # on January 10, 20071×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-10 2.18534julia> ts[[Date(2007, 1, 10), Date(2007, 1, 11)]] # January 10, 112×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-10 2.18534 2007-01-11 4.08667julia> ts[Year(2007), Month(1)] # entire January 200731×1 TSFrame with Date Index Index value Date Float64 ────────────────────── 2007-01-01 5.3065 2007-01-02 4.86443 2007-01-03 4.53017 2007-01-04 9.19559 2007-01-05 9.04449 2007-01-06 5.07151 2007-01-07 4.79894 2007-01-08 6.23251 ⋮ ⋮ 2007-01-25 6.67892 2007-01-26 3.03758 2007-01-27 0.487692 2007-01-28 1.91311 2007-01-29 4.13692 2007-01-30 6.15376 2007-01-31 0.408408 16 rows omittedjulia> ts[Year(2007), Quarter(2)]91×1 TSFrame with Date Index Index value Date Float64 ────────────────────── 2007-04-01 7.32093 2007-04-02 2.51794 2007-04-03 4.72252 2007-04-04 3.25793 2007-04-05 0.886405 2007-04-06 5.36322 2007-04-07 4.91766 2007-04-08 8.67172 ⋮ ⋮ 2007-06-24 8.83008 2007-06-25 9.93744 2007-06-26 0.173108 2007-06-27 4.85811 2007-06-28 3.53503 2007-06-29 6.98281 2007-06-30 7.75674 76 rows omitted
Finally, one can also use the dot notation to get a column as a vector.
julia> ts.value # get the value column as a vector431-element Vector{Float64}: 5.306503881966636 4.864426404474837 4.5301691220437785 9.195586027173078 9.044490183548072 5.07151239582835 4.798939210287169 6.232506101745746 9.227479907560694 2.185338126254596 ⋮ 7.075164117349065 4.728188430246146 7.94480957824288 8.140292546407652 5.389644167327803 4.583272703461838 1.8466728235358842 8.736570442453303 5.853654791150219
Summary statistics
The describe() method prints summary statistics of the TSFrame object. The output is a DataFrame which includes the number of missing values, data types of columns along with computed statistical values.
julia> TSFrames.describe(ts)2×7 DataFrame Row │ variable mean min median max nmissing eltype ⋯ │ Symbol Union… Any Any Any Int64 DataTy ⋯ ─────┼────────────────────────────────────────────────────────────────────────── 1 │ Index 2007-01-01 2007-08-04 2008-03-06 0 Date ⋯ 2 │ value 5.10116 0.00418655 5.07151 9.97559 0 Float6 1 column omitted
Plotting
A TSFrame object can be plotted using the plot() function of the Plots package. The plotting functionality is provided by RecipesBase package so all the flexibility and functionality of the Plots package is available for users.
using Plots
plot(ts, size=(600,400); legend=false)Applying a function over a period
The apply method allows you to aggregate the TSFrame object over a period type (Dates.Period(@ref)) and return the output of applying the function on each period. For example, to convert frequency of daily timeseries to monthly you may use first(), last(), or Statistics.mean() functions and the period as Dates.Month.
julia> using Statisticsjulia> ts_monthly = apply(ts, Month(1), last) # convert to monthly series using the last value for each month15×1 TSFrame with Date Index Index value_last Date Float64 ──────────────────────── 2007-01-01 0.408408 2007-02-01 2.46895 2007-03-01 0.185137 2007-04-01 7.2581 2007-05-01 2.13574 2007-06-01 7.75674 2007-07-01 1.61569 2007-08-01 8.47846 2007-09-01 6.25824 2007-10-01 1.1529 2007-11-01 5.41158 2007-12-01 7.55194 2008-01-01 7.40325 2008-02-01 7.94481 2008-03-01 5.85365julia> ts_weekly = apply(ts, Week(1), Statistics.std) # compute weekly standard deviation62×1 TSFrame with Date Index Index value_std Date Float64 ─────────────────────── 2007-01-01 2.06648 2007-01-08 2.624 2007-01-15 3.12472 2007-01-22 2.657 2007-01-29 2.25522 2007-02-05 3.02554 2007-02-12 3.6383 2007-02-19 1.98456 ⋮ ⋮ 2008-01-21 3.03061 2008-01-28 3.00429 2008-02-04 3.23744 2008-02-11 1.67818 2008-02-18 2.99524 2008-02-25 1.59849 2008-03-03 2.86051 47 rows omittedjulia> apply(ts, Week(1), Statistics.std, last) # same as above but index contains the last date of the week62×1 TSFrame with Date Index Index value_std Date Float64 ─────────────────────── 2007-01-07 2.06648 2007-01-14 2.624 2007-01-21 3.12472 2007-01-28 2.657 2007-02-04 2.25522 2007-02-11 3.02554 2007-02-18 3.6383 2007-02-25 1.98456 ⋮ ⋮ 2008-01-27 3.03061 2008-02-03 3.00429 2008-02-10 3.23744 2008-02-17 1.67818 2008-02-24 2.99524 2008-03-02 1.59849 2008-03-06 2.86051 47 rows omittedjulia> apply(ts, Week(1), Statistics.std, last, renamecols=false) # do not rename column62×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-07 2.06648 2007-01-14 2.624 2007-01-21 3.12472 2007-01-28 2.657 2007-02-04 2.25522 2007-02-11 3.02554 2007-02-18 3.6383 2007-02-25 1.98456 ⋮ ⋮ 2008-01-27 3.03061 2008-02-03 3.00429 2008-02-10 3.23744 2008-02-17 1.67818 2008-02-24 2.99524 2008-03-02 1.59849 2008-03-06 2.86051 47 rows omitted
Joins: Row and column binding with other objects
TSFrames provides methods to join two TSFrame objects by columns: join (alias: cbind) or by rows: vcat (alias: rbind). Both the methods provide some basic intelligence while doing the merge.
join merges two datasets based on the Index values of both objects. Depending on the join strategy employed the final object may only contain index values only from the left object (using jointype=:JoinLeft), the right object (using jointype=:JoinRight), intersection of both objects (using jointype=:JoinBoth), or a union of both objects (jointype=:JoinAll) while inserting missing values where index values are missing from any of the other object.
julia> dates = collect(Date(2007,1,1):Day(1):Date(2007,1,30));julia> ts2 = TSFrame(rand(length(dates)), dates)30×1 TSFrame with Date Index Index x1 Date Float64 ─────────────────────── 2007-01-01 0.311868 2007-01-02 0.176265 2007-01-03 0.800232 2007-01-04 0.937782 2007-01-05 0.840768 2007-01-06 0.506254 2007-01-07 0.0832845 2007-01-08 0.430459 ⋮ ⋮ 2007-01-24 0.808323 2007-01-25 0.0163639 2007-01-26 0.56034 2007-01-27 0.0155923 2007-01-28 0.0596187 2007-01-29 0.0229536 2007-01-30 0.818898 15 rows omittedjulia> join(ts, ts2; jointype=:JoinAll) # cbind/join on Index column431×2 TSFrame with Date Index Index value x1 Date Float64? Float64? ─────────────────────────────────────── 2007-01-01 5.3065 0.311868 2007-01-02 4.86443 0.176265 2007-01-03 4.53017 0.800232 2007-01-04 9.19559 0.937782 2007-01-05 9.04449 0.840768 2007-01-06 5.07151 0.506254 2007-01-07 4.79894 0.0832845 2007-01-08 6.23251 0.430459 ⋮ ⋮ ⋮ 2008-02-29 7.94481 missing 2008-03-01 8.14029 missing 2008-03-02 5.38964 missing 2008-03-03 4.58327 missing 2008-03-04 1.84667 missing 2008-03-05 8.73657 missing 2008-03-06 5.85365 missing 416 rows omitted
vcat also works similarly but merges two datasets by rows. This method also uses certain strategies provided via colmerge argument to check for certain conditions before doing the merge, throwing an error if the conditions are not satisfied.
colmerge can be passed setequal which merges only if both objects have same column names, orderequal which merges only if both objects have same column names and columns are in the same order, intersect merges only the columns which are common to both objects, and union which merges even if the columns differ between the two objects, the resulting object has the columns filled with missing, if necessary.
For vcat, if the values of Index are same in the two objects then all the index values along with values in other columns are kept in the resulting object. So, a vcat operation may result in duplicate Index values and the results from other operations may differ or even throw unknown errors.
julia> dates = collect(Date(2008,4,1):Day(1):Date(2008,4,30));julia> ts3 = TSFrame(DataFrame(values=rand(length(dates)), Index=dates))30×1 TSFrame with Date Index Index values Date Float64 ─────────────────────── 2008-04-01 0.713126 2008-04-02 0.643149 2008-04-03 0.977934 2008-04-04 0.326262 2008-04-05 0.291171 2008-04-06 0.671356 2008-04-07 0.113427 2008-04-08 0.0564313 ⋮ ⋮ 2008-04-24 0.303711 2008-04-25 0.56764 2008-04-26 0.168782 2008-04-27 0.776584 2008-04-28 0.180594 2008-04-29 0.483835 2008-04-30 0.806175 15 rows omittedjulia> vcat(ts, ts3) # do the merge461×2 TSFrame with Date Index Index value values Date Float64? Float64? ─────────────────────────────────────────── 2007-01-01 5.3065 missing 2007-01-02 4.86443 missing 2007-01-03 4.53017 missing 2007-01-04 9.19559 missing 2007-01-05 9.04449 missing 2007-01-06 5.07151 missing 2007-01-07 4.79894 missing 2007-01-08 6.23251 missing ⋮ ⋮ ⋮ 2008-04-24 missing 0.303711 2008-04-25 missing 0.56764 2008-04-26 missing 0.168782 2008-04-27 missing 0.776584 2008-04-28 missing 0.180594 2008-04-29 missing 0.483835 2008-04-30 missing 0.806175 446 rows omitted
Rolling window operations
The rollapply applies a function over a fixed-size rolling window on the dataset. In the example below, we compute the 10-day average of dataset values on a rolling basis.
julia> rollapply(ts, mean, 10)422×1 TSFrame with Date Index Index rolling_value_mean Date Float64 ──────────────────────────────── 2007-01-10 6.0457 2007-01-11 5.92371 2007-01-12 5.80695 2007-01-13 6.04399 2007-01-14 5.98292 2007-01-15 5.11491 2007-01-16 5.54584 2007-01-17 5.68635 ⋮ ⋮ 2008-02-29 5.90517 2008-03-01 6.09506 2008-03-02 5.96457 2008-03-03 6.09159 2008-03-04 5.31394 2008-03-05 6.17454 2008-03-06 6.26765 407 rows omitted
Computing rolling difference and percent change
Similar to apply and rollapply there are specific methods to compute rolling differences and percent changes of a TSFrame object. The diff method computes mathematical difference of values in adjacent rows, inserting missing in the first row. pctchange computes the percentage change between adjacent rows.
julia> diff(ts)431×1 TSFrame with Date Index Index value Date Float64? ──────────────────────────── 2007-01-01 missing 2007-01-02 -0.442077 2007-01-03 -0.334257 2007-01-04 4.66542 2007-01-05 -0.151096 2007-01-06 -3.97298 2007-01-07 -0.272573 2007-01-08 1.43357 ⋮ ⋮ 2008-02-29 3.21662 2008-03-01 0.195483 2008-03-02 -2.75065 2008-03-03 -0.806371 2008-03-04 -2.7366 2008-03-05 6.8899 2008-03-06 -2.88292 416 rows omittedjulia> pctchange(ts)431×1 TSFrame with Date Index Index value Date Float64? ───────────────────────────── 2007-01-01 missing 2007-01-02 -0.0833086 2007-01-03 -0.0687146 2007-01-04 1.02985 2007-01-05 -0.0164313 2007-01-06 -0.439271 2007-01-07 -0.0537459 2007-01-08 0.298726 ⋮ ⋮ 2008-02-29 0.680307 2008-03-01 0.0246051 2008-03-02 -0.337905 2008-03-03 -0.149615 2008-03-04 -0.597084 2008-03-05 3.73098 2008-03-06 -0.329983 416 rows omitted
Computing log of data values
julia> log.(ts)431×1 TSFrame with Date Index Index value_log Date Float64 ─────────────────────── 2007-01-01 1.66893 2007-01-02 1.58195 2007-01-03 1.51076 2007-01-04 2.21872 2007-01-05 2.20216 2007-01-06 1.62364 2007-01-07 1.56839 2007-01-08 1.82978 ⋮ ⋮ 2008-02-29 2.07252 2008-03-01 2.09683 2008-03-02 1.68448 2008-03-03 1.52241 2008-03-04 0.613386 2008-03-05 2.16752 2008-03-06 1.76707 416 rows omitted
Creating lagged/leading series
lag() and lead() provide ways to lag or lead a series respectively by a fixed value, inserting missing where required.
julia> lag(ts, 2)431×1 TSFrame with Date Index Index value Date Float64? ──────────────────────────── 2007-01-01 missing 2007-01-02 missing 2007-01-03 5.3065 2007-01-04 4.86443 2007-01-05 4.53017 2007-01-06 9.19559 2007-01-07 9.04449 2007-01-08 5.07151 ⋮ ⋮ 2008-02-29 7.07516 2008-03-01 4.72819 2008-03-02 7.94481 2008-03-03 8.14029 2008-03-04 5.38964 2008-03-05 4.58327 2008-03-06 1.84667 416 rows omittedjulia> lead(ts, 2)431×1 TSFrame with Date Index Index value Date Float64? ─────────────────────────── 2007-01-01 4.53017 2007-01-02 9.19559 2007-01-03 9.04449 2007-01-04 5.07151 2007-01-05 4.79894 2007-01-06 6.23251 2007-01-07 9.22748 2007-01-08 2.18534 ⋮ ⋮ 2008-02-29 5.38964 2008-03-01 4.58327 2008-03-02 1.84667 2008-03-03 8.73657 2008-03-04 5.85365 2008-03-05 missing 2008-03-06 missing 416 rows omitted
Converting to Matrix and DataFrame
You can easily convert a TSFrame object into a Matrix or fetch the DataFrame for doing operations which are outside of the TSFrames scope.
julia> ts[:, 1] # convert column 1 to a vector of floats431-element Vector{Float64}: 5.306503881966636 4.864426404474837 4.5301691220437785 9.195586027173078 9.044490183548072 5.07151239582835 4.798939210287169 6.232506101745746 9.227479907560694 2.185338126254596 ⋮ 7.075164117349065 4.728188430246146 7.94480957824288 8.140292546407652 5.389644167327803 4.583272703461838 1.8466728235358842 8.736570442453303 5.853654791150219julia> Matrix(ts) # convert entire TSFrame into a Matrix431×1 Matrix{Float64}: 5.306503881966636 4.864426404474837 4.5301691220437785 9.195586027173078 9.044490183548072 5.07151239582835 4.798939210287169 6.232506101745746 9.227479907560694 2.185338126254596 ⋮ 7.075164117349065 4.728188430246146 7.94480957824288 8.140292546407652 5.389644167327803 4.583272703461838 1.8466728235358842 8.736570442453303 5.853654791150219julia> select(ts.coredata, :Index, :value, DataFrames.nrow) # use the underlying DataFrame for other operations431×3 DataFrame Row │ Index value nrow │ Date Float64 Int64 ─────┼──────────────────────────── 1 │ 2007-01-01 5.3065 431 2 │ 2007-01-02 4.86443 431 3 │ 2007-01-03 4.53017 431 4 │ 2007-01-04 9.19559 431 5 │ 2007-01-05 9.04449 431 6 │ 2007-01-06 5.07151 431 7 │ 2007-01-07 4.79894 431 8 │ 2007-01-08 6.23251 431 ⋮ │ ⋮ ⋮ ⋮ 425 │ 2008-02-29 7.94481 431 426 │ 2008-03-01 8.14029 431 427 │ 2008-03-02 5.38964 431 428 │ 2008-03-03 4.58327 431 429 │ 2008-03-04 1.84667 431 430 │ 2008-03-05 8.73657 431 431 │ 2008-03-06 5.85365 431 416 rows omitted
Writing TSFrame into a CSV file
Writing a TSFrame object into a CSV file can be done easily by using the underlying coredata property. This DataFrame can be passed to the CSV.write method for writing into a file.
julia> CSV.write("/tmp/demo_ts.csv", ts)"/tmp/demo_ts.csv"
Broadcasting
Broadcasting can be used on a TSFrame object to apply a function to a subset of it's columns.
julia> using TSFrames, DataFrames;
julia> ts = TSFrame(DataFrame(Index = [1, 2, 3, 4, 5], A = [10.1, 12.4, 42.4, 24.1, 242.5], B = [2, 4, 6, 8, 10]))
(5 x 2) TSFrame with Int64 Index
Index A B
Int64 Float64 Int64
───────────────────────
1 10.1 2
2 12.4 4
3 42.4 6
4 24.1 8
5 242.5 10
julia> sin_A = sin.(ts[:, [:A]]) # get sin of column A
(5 x 1) TSFrame with Int64 Index
Index A_sin
Int64 Float64
──────────────────
1 -0.625071
2 -0.165604
3 -0.999934
4 -0.858707
5 -0.562466
julia> log_ts = log.(ts) # take log of all columns
(5 x 2) TSFrame with Int64 Index
Index A_log B_log
Int64 Float64 Float64
──────────────────────────
1 2.31254 0.693147
2 2.5177 1.38629
3 3.74715 1.79176
4 3.18221 2.07944
5 5.491 2.30259
julia> log_ts = log.(ts[:, [:A, :B]]) # can specify multiple columns
(5 x 2) TSFrame with Int64 Index
Index A_log B_log
Int64 Float64 Float64
──────────────────────────
1 2.31254 0.693147
2 2.5177 1.38629
3 3.74715 1.79176
4 3.18221 2.07944
5 5.491 2.30259
Tables.jl Integration
TSFrame objects are Tables.jl compatible. This integration enables easy conversion between the TSFrame format and other formats which are Tables.jl compatible.
As an example, first consider the following code which converts a TSFrame object into a DataFrame, a TimeArray and a CSV file respectively.
julia> using TSFrames, TimeSeries, Dates, DataFrames, CSV;
julia> dates = Date(2018, 1, 1):Day(1):Date(2018, 12, 31)
Date("2018-01-01"):Day(1):Date("2018-12-31")
julia> ts = TSFrame(DataFrame(Index = dates, x1 = 1:365));
# conversion to DataFrames
julia> df = DataFrame(ts);
# conversion to TimeArray
julia> timeArray = TimeArray(ts, timestamp = :Index);
# writing to CSV
julia> CSV.write("ts.csv", ts);
Next, here is some code which converts a DataFrame, a TimeArray and a CSV file to a TSFrame object.
julia> using TSFrames, DataFrames, CSV, TimeSeries, Dates;
# converting DataFrame to TSFrame
julia> ts = TSFrame(DataFrame(Index=1:10, x1=1:10));
# converting from TimeArray to TSFrame
julia> dates = Date(2018, 1, 1):Day(1):Date(2018, 12, 31)
Date("2018-01-01"):Day(1):Date("2018-12-31")
julia> ta = TimeArray(dates, rand(length(dates)));
julia> ts = TSFrame(ta);
# converting from CSV to TSFrame
julia> CSV.read("ts.csv", TSFrame);This discussion warrants a note about how we've implemented the Tables.jl interfaces. Since TSFrame objects are nothing but a wrapper around a DataFrame, our implementations of these interfaces just call DataFrames.jl's implementations. Moreover, while constructing TSFrame objects out of other Tables.jl compatible types, our constructor first converts the input table to a DataFrame, and then converts the DataFrame to a TSFrame object.