User guide
This page describes how to use the TSFrames package for timeseries data handling.
Installation
julia> using Pkg
julia> Pkg.add(url="https://github.com/xKDR/TSFrames.jl")Constructing TSFrame objects
After installing TSFrames you need to load the package in Julia environment. Then, create a basic TSFrame object.
julia> using TSFrames;julia> ts = TSFrame(1:10)10×1 TSFrame with Int64 Index Index x1 Int64 Int64 ────────────── 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10julia> ts.coredata10×2 DataFrame Row │ Index x1 │ Int64 Int64 ─────┼────────────── 1 │ 1 1 2 │ 2 2 3 │ 3 3 4 │ 4 4 5 │ 5 5 6 │ 6 6 7 │ 7 7 8 │ 8 8 9 │ 9 9 10 │ 10 10
The basic TSFrame constructor takes in a Vector of any type and automatically generates an index out of it (the Index column).
There are many ways to construct a TSFrame object. For real world applications you would want to read in a CSV file or download a dataset as a DataFrame and then operate on it. You can easily convert a DataFrame to a TSFrame object.
julia> using CSV, DataFrames, TSFrames, Datesjulia> dates = Date(2007, 1, 1):Day(1):Date(2008, 03, 06)Date("2007-01-01"):Dates.Day(1):Date("2008-03-06")julia> ts = TSFrame(DataFrame(Index=dates, value=10*rand(431)))431×1 TSFrame with Date Index Index value Date Float64 ────────────────────── 2007-01-01 8.86565 2007-01-02 5.77457 2007-01-03 1.9952 2007-01-04 2.84001 2007-01-05 1.13547 2007-01-06 4.096 2007-01-07 8.18981 2007-01-08 5.27544 ⋮ ⋮ 2008-02-29 4.69285 2008-03-01 5.06278 2008-03-02 1.63004 2008-03-03 7.98468 2008-03-04 9.51486 2008-03-05 0.952593 2008-03-06 1.66704 416 rows omitted
In the above example you generate a random DataFrame and convert it into a TSFrame object ts. The top line of the ts object tells you the number of rows (431 here) and the number of columns (1) along with the Type of Index (Dates.Date in the above example).
You can also fetch the number of rows and columns by using nr(ts), nc(ts), and size(ts) methods. Respectively, they fetch the number of rows, columns, and a Tuple of row and column numbers. A length(::TSFrame) method is also provided for convenience which returns the number of rows of it's argument.
julia> nr(ts)431julia> nc(ts)1julia> size(ts)(431, 1)julia> length(ts)431
Names of data columns can be fetched using the names(ts) method which returns a Vector{String} object. The Index column can be fetched as an object of Vector type by using the index(ts) method, it can also be fetched directly using the underlying coredata property of TSFrame: ts.coredata[!, :Index].
julia> names(ts)1-element Vector{String}: "value"julia> index(ts)431-element Vector{Date}: 2007-01-01 2007-01-02 2007-01-03 2007-01-04 2007-01-05 2007-01-06 2007-01-07 2007-01-08 2007-01-09 2007-01-10 ⋮ 2008-02-27 2008-02-28 2008-02-29 2008-03-01 2008-03-02 2008-03-03 2008-03-04 2008-03-05 2008-03-06
Another simpler way to read a CSV is to pass TSFrame as a sink to the CSV.read function.
julia> ts = CSV.File(filename, TSFrame)Indexing and subsetting
One of the primary features of a timeseries package is to provide ways to index or subset a dataset using convenient interfaces. TSFrames makes it easier to index a TSFrame object by providing multiple intuitive getindex methods which work by just using the regular square parentheses([ ]).
julia> ts[1] # first row1×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-01 8.86565julia> ts[[3, 5], [1]] # third & fifth row, and first column2×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-03 1.9952 2007-01-05 1.13547julia> ts[1:10, 1] # first 10 rows and the first column as a vector10-element Vector{Float64}: 8.865646843081969 5.774571919717318 1.9952046244436006 2.8400084462204775 1.135472705569004 4.096003059998256 8.189811881987815 5.275436387659934 1.7840059352672033 4.839626395560415julia> ts[1, [:value]] # using the column name1×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-01 8.86565
Apart from integer-based row indexing and integer, name based column indexing, TSFrames provides special subsetting methods for date and time types defined inside the Dates module.
julia> ts[Date(2007, 1, 10)] # on January 10, 20071×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-10 4.83963julia> ts[[Date(2007, 1, 10), Date(2007, 1, 11)]] # January 10, 112×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-10 4.83963 2007-01-11 2.94652julia> ts[Year(2007), Month(1)] # entire January 200731×1 TSFrame with Date Index Index value Date Float64 ────────────────────── 2007-01-01 8.86565 2007-01-02 5.77457 2007-01-03 1.9952 2007-01-04 2.84001 2007-01-05 1.13547 2007-01-06 4.096 2007-01-07 8.18981 2007-01-08 5.27544 ⋮ ⋮ 2007-01-25 7.03886 2007-01-26 6.68316 2007-01-27 4.29167 2007-01-28 0.81052 2007-01-29 0.232917 2007-01-30 9.7533 2007-01-31 9.38362 16 rows omittedjulia> ts[Year(2007), Quarter(2)]91×1 TSFrame with Date Index Index value Date Float64 ────────────────────── 2007-04-01 9.87501 2007-04-02 1.73686 2007-04-03 2.43978 2007-04-04 8.15163 2007-04-05 3.44449 2007-04-06 3.51752 2007-04-07 4.0729 2007-04-08 4.32205 ⋮ ⋮ 2007-06-24 0.810912 2007-06-25 0.931967 2007-06-26 1.62204 2007-06-27 8.22153 2007-06-28 3.91035 2007-06-29 5.29617 2007-06-30 8.43773 76 rows omitted
Finally, one can also use the dot notation to get a column as a vector.
julia> ts.value # get the value column as a vector431-element Vector{Float64}: 8.865646843081969 5.774571919717318 1.9952046244436006 2.8400084462204775 1.135472705569004 4.096003059998256 8.189811881987815 5.275436387659934 1.7840059352672033 4.839626395560415 ⋮ 3.073409473418348 5.954373238968136 4.6928472601902085 5.06277771477449 1.6300408714573833 7.984683888857418 9.514861474587416 0.9525932348548527 1.6670370875061258
Summary statistics
The describe() method prints summary statistics of the TSFrame object. The output is a DataFrame which includes the number of missing values, data types of columns along with computed statistical values.
julia> TSFrames.describe(ts)2×7 DataFrame Row │ variable mean min median max nmissing eltype ⋯ │ Symbol Union… Any Any Any Int64 DataTy ⋯ ─────┼────────────────────────────────────────────────────────────────────────── 1 │ Index 2007-01-01 2007-08-04 2008-03-06 0 Date ⋯ 2 │ value 4.90384 0.00235538 4.8284 9.98329 0 Float6 1 column omitted
Plotting
A TSFrame object can be plotted using the plot() function of the Plots package. The plotting functionality is provided by RecipesBase package so all the flexibility and functionality of the Plots package is available for users.
using Plots
plot(ts, size=(600,400); legend=false)Applying a function over a period
The apply method allows you to aggregate the TSFrame object over a period type (Dates.Period(@ref)) and return the output of applying the function on each period. For example, to convert frequency of daily timeseries to monthly you may use first(), last(), or Statistics.mean() functions and the period as Dates.Month.
julia> using Statisticsjulia> ts_monthly = apply(ts, Month(1), last) # convert to monthly series using the last value for each month15×1 TSFrame with Date Index Index value_last Date Float64 ──────────────────────── 2007-01-01 9.38362 2007-02-01 1.39475 2007-03-01 1.21137 2007-04-01 2.20453 2007-05-01 9.8781 2007-06-01 8.43773 2007-07-01 8.63218 2007-08-01 2.70219 2007-09-01 3.05141 2007-10-01 1.41861 2007-11-01 8.8896 2007-12-01 3.39404 2008-01-01 0.726228 2008-02-01 4.69285 2008-03-01 1.66704julia> ts_weekly = apply(ts, Week(1), Statistics.std) # compute weekly standard deviation62×1 TSFrame with Date Index Index value_std Date Float64 ─────────────────────── 2007-01-01 3.01475 2007-01-08 1.38168 2007-01-15 1.10941 2007-01-22 2.56774 2007-01-29 3.37676 2007-02-05 1.95796 2007-02-12 3.39738 2007-02-19 2.75081 ⋮ ⋮ 2008-01-21 2.72303 2008-01-28 3.60533 2008-02-04 2.82191 2008-02-11 3.16561 2008-02-18 2.78407 2008-02-25 1.8414 2008-03-03 4.35044 47 rows omittedjulia> apply(ts, Week(1), Statistics.std, last) # same as above but index contains the last date of the week62×1 TSFrame with Date Index Index value_std Date Float64 ─────────────────────── 2007-01-07 3.01475 2007-01-14 1.38168 2007-01-21 1.10941 2007-01-28 2.56774 2007-02-04 3.37676 2007-02-11 1.95796 2007-02-18 3.39738 2007-02-25 2.75081 ⋮ ⋮ 2008-01-27 2.72303 2008-02-03 3.60533 2008-02-10 2.82191 2008-02-17 3.16561 2008-02-24 2.78407 2008-03-02 1.8414 2008-03-06 4.35044 47 rows omittedjulia> apply(ts, Week(1), Statistics.std, last, renamecols=false) # do not rename column62×1 TSFrame with Date Index Index value Date Float64 ───────────────────── 2007-01-07 3.01475 2007-01-14 1.38168 2007-01-21 1.10941 2007-01-28 2.56774 2007-02-04 3.37676 2007-02-11 1.95796 2007-02-18 3.39738 2007-02-25 2.75081 ⋮ ⋮ 2008-01-27 2.72303 2008-02-03 3.60533 2008-02-10 2.82191 2008-02-17 3.16561 2008-02-24 2.78407 2008-03-02 1.8414 2008-03-06 4.35044 47 rows omitted
Joins: Row and column binding with other objects
TSFrames provides methods to join two TSFrame objects by columns: join (alias: cbind) or by rows: vcat (alias: rbind). Both the methods provide some basic intelligence while doing the merge.
join merges two datasets based on the Index values of both objects. Depending on the join strategy employed the final object may only contain index values only from the left object (using jointype=:JoinLeft), the right object (using jointype=:JoinRight), intersection of both objects (using jointype=:JoinBoth), or a union of both objects (jointype=:JoinAll) while inserting missing values where index values are missing from any of the other object.
julia> dates = collect(Date(2007,1,1):Day(1):Date(2007,1,30));julia> ts2 = TSFrame(rand(length(dates)), dates)30×1 TSFrame with Date Index Index x1 Date Float64 ───────────────────────── 2007-01-01 0.580498 2007-01-02 0.0634722 2007-01-03 0.199496 2007-01-04 0.612483 2007-01-05 0.330567 2007-01-06 0.717784 2007-01-07 0.476797 2007-01-08 0.605608 ⋮ ⋮ 2007-01-24 0.380159 2007-01-25 0.947078 2007-01-26 0.61767 2007-01-27 0.284386 2007-01-28 0.968559 2007-01-29 0.307711 2007-01-30 0.801643 15 rows omittedjulia> join(ts, ts2; jointype=:JoinAll) # cbind/join on Index column431×2 TSFrame with Date Index Index value x1 Date Float64? Float64? ───────────────────────────────────────── 2007-01-01 8.86565 0.580498 2007-01-02 5.77457 0.0634722 2007-01-03 1.9952 0.199496 2007-01-04 2.84001 0.612483 2007-01-05 1.13547 0.330567 2007-01-06 4.096 0.717784 2007-01-07 8.18981 0.476797 2007-01-08 5.27544 0.605608 ⋮ ⋮ ⋮ 2008-02-29 4.69285 missing 2008-03-01 5.06278 missing 2008-03-02 1.63004 missing 2008-03-03 7.98468 missing 2008-03-04 9.51486 missing 2008-03-05 0.952593 missing 2008-03-06 1.66704 missing 416 rows omitted
vcat also works similarly but merges two datasets by rows. This method also uses certain strategies provided via colmerge argument to check for certain conditions before doing the merge, throwing an error if the conditions are not satisfied.
colmerge can be passed setequal which merges only if both objects have same column names, orderequal which merges only if both objects have same column names and columns are in the same order, intersect merges only the columns which are common to both objects, and union which merges even if the columns differ between the two objects, the resulting object has the columns filled with missing, if necessary.
For vcat, if the values of Index are same in the two objects then all the index values along with values in other columns are kept in the resulting object. So, a vcat operation may result in duplicate Index values and the results from other operations may differ or even throw unknown errors.
julia> dates = collect(Date(2008,4,1):Day(1):Date(2008,4,30));julia> ts3 = TSFrame(DataFrame(values=rand(length(dates)), Index=dates))30×1 TSFrame with Date Index Index values Date Float64 ─────────────────────── 2008-04-01 0.881994 2008-04-02 0.766306 2008-04-03 0.574593 2008-04-04 0.226045 2008-04-05 0.303832 2008-04-06 0.44448 2008-04-07 0.0597581 2008-04-08 0.218164 ⋮ ⋮ 2008-04-24 0.660542 2008-04-25 0.724469 2008-04-26 0.743711 2008-04-27 0.127895 2008-04-28 0.761663 2008-04-29 0.926216 2008-04-30 0.687946 15 rows omittedjulia> vcat(ts, ts3) # do the merge461×2 TSFrame with Date Index Index value values Date Float64? Float64? ─────────────────────────────────────────── 2007-01-01 8.86565 missing 2007-01-02 5.77457 missing 2007-01-03 1.9952 missing 2007-01-04 2.84001 missing 2007-01-05 1.13547 missing 2007-01-06 4.096 missing 2007-01-07 8.18981 missing 2007-01-08 5.27544 missing ⋮ ⋮ ⋮ 2008-04-24 missing 0.660542 2008-04-25 missing 0.724469 2008-04-26 missing 0.743711 2008-04-27 missing 0.127895 2008-04-28 missing 0.761663 2008-04-29 missing 0.926216 2008-04-30 missing 0.687946 446 rows omitted
Rolling window operations
The rollapply applies a function over a fixed-size rolling window on the dataset. In the example below, we compute the 10-day average of dataset values on a rolling basis.
julia> rollapply(ts, mean, 10)422×1 TSFrame with Date Index Index rolling_value_mean Date Float64 ──────────────────────────────── 2007-01-10 4.47958 2007-01-11 3.88767 2007-01-12 3.81801 2007-01-13 3.90904 2007-01-14 4.11194 2007-01-15 4.3742 2007-01-16 4.34464 2007-01-17 3.95579 ⋮ ⋮ 2008-02-29 4.04022 2008-03-01 4.40325 2008-03-02 3.98308 2008-03-03 3.9407 2008-03-04 4.47352 2008-03-05 4.23068 2008-03-06 4.2755 407 rows omitted
Computing rolling difference and percent change
Similar to apply and rollapply there are specific methods to compute rolling differences and percent changes of a TSFrame object. The diff method computes mathematical difference of values in adjacent rows, inserting missing in the first row. pctchange computes the percentage change between adjacent rows.
julia> diff(ts)431×1 TSFrame with Date Index Index value Date Float64? ──────────────────────────── 2007-01-01 missing 2007-01-02 -3.09107 2007-01-03 -3.77937 2007-01-04 0.844804 2007-01-05 -1.70454 2007-01-06 2.96053 2007-01-07 4.09381 2007-01-08 -2.91438 ⋮ ⋮ 2008-02-29 -1.26153 2008-03-01 0.36993 2008-03-02 -3.43274 2008-03-03 6.35464 2008-03-04 1.53018 2008-03-05 -8.56227 2008-03-06 0.714444 416 rows omittedjulia> pctchange(ts)431×1 TSFrame with Date Index Index value Date Float64? ───────────────────────────── 2007-01-01 missing 2007-01-02 -0.348658 2007-01-03 -0.654484 2007-01-04 0.423417 2007-01-05 -0.600187 2007-01-06 2.60731 2007-01-07 0.999464 2007-01-08 -0.355854 ⋮ ⋮ 2008-02-29 -0.211865 2008-03-01 0.0788286 2008-03-02 -0.678034 2008-03-03 3.89846 2008-03-04 0.191639 2008-03-05 -0.899884 2008-03-06 0.749999 416 rows omitted
Computing log of data values
julia> log.(ts)431×1 TSFrame with Date Index Index value_log Date Float64 ──────────────────────── 2007-01-01 2.18218 2007-01-02 1.75346 2007-01-03 0.690747 2007-01-04 1.04381 2007-01-05 0.127049 2007-01-06 1.41001 2007-01-07 2.10289 2007-01-08 1.66306 ⋮ ⋮ 2008-02-29 1.54604 2008-03-01 1.62192 2008-03-02 0.488605 2008-03-03 2.07753 2008-03-04 2.25285 2008-03-05 -0.0485673 2008-03-06 0.511048 416 rows omitted
Creating lagged/leading series
lag() and lead() provide ways to lag or lead a series respectively by a fixed value, inserting missing where required.
julia> lag(ts, 2)431×1 TSFrame with Date Index Index value Date Float64? ─────────────────────────── 2007-01-01 missing 2007-01-02 missing 2007-01-03 8.86565 2007-01-04 5.77457 2007-01-05 1.9952 2007-01-06 2.84001 2007-01-07 1.13547 2007-01-08 4.096 ⋮ ⋮ 2008-02-29 3.07341 2008-03-01 5.95437 2008-03-02 4.69285 2008-03-03 5.06278 2008-03-04 1.63004 2008-03-05 7.98468 2008-03-06 9.51486 416 rows omittedjulia> lead(ts, 2)431×1 TSFrame with Date Index Index value Date Float64? ──────────────────────────── 2007-01-01 1.9952 2007-01-02 2.84001 2007-01-03 1.13547 2007-01-04 4.096 2007-01-05 8.18981 2007-01-06 5.27544 2007-01-07 1.78401 2007-01-08 4.83963 ⋮ ⋮ 2008-02-29 1.63004 2008-03-01 7.98468 2008-03-02 9.51486 2008-03-03 0.952593 2008-03-04 1.66704 2008-03-05 missing 2008-03-06 missing 416 rows omitted
Converting to Matrix and DataFrame
You can easily convert a TSFrame object into a Matrix or fetch the DataFrame for doing operations which are outside of the TSFrames scope.
julia> ts[:, 1] # convert column 1 to a vector of floats431-element Vector{Float64}: 8.865646843081969 5.774571919717318 1.9952046244436006 2.8400084462204775 1.135472705569004 4.096003059998256 8.189811881987815 5.275436387659934 1.7840059352672033 4.839626395560415 ⋮ 3.073409473418348 5.954373238968136 4.6928472601902085 5.06277771477449 1.6300408714573833 7.984683888857418 9.514861474587416 0.9525932348548527 1.6670370875061258julia> Matrix(ts) # convert entire TSFrame into a Matrix431×1 Matrix{Float64}: 8.865646843081969 5.774571919717318 1.9952046244436006 2.8400084462204775 1.135472705569004 4.096003059998256 8.189811881987815 5.275436387659934 1.7840059352672033 4.839626395560415 ⋮ 3.073409473418348 5.954373238968136 4.6928472601902085 5.06277771477449 1.6300408714573833 7.984683888857418 9.514861474587416 0.9525932348548527 1.6670370875061258julia> select(ts.coredata, :Index, :value, DataFrames.nrow) # use the underlying DataFrame for other operations431×3 DataFrame Row │ Index value nrow │ Date Float64 Int64 ─────┼───────────────────────────── 1 │ 2007-01-01 8.86565 431 2 │ 2007-01-02 5.77457 431 3 │ 2007-01-03 1.9952 431 4 │ 2007-01-04 2.84001 431 5 │ 2007-01-05 1.13547 431 6 │ 2007-01-06 4.096 431 7 │ 2007-01-07 8.18981 431 8 │ 2007-01-08 5.27544 431 ⋮ │ ⋮ ⋮ ⋮ 425 │ 2008-02-29 4.69285 431 426 │ 2008-03-01 5.06278 431 427 │ 2008-03-02 1.63004 431 428 │ 2008-03-03 7.98468 431 429 │ 2008-03-04 9.51486 431 430 │ 2008-03-05 0.952593 431 431 │ 2008-03-06 1.66704 431 416 rows omitted
Writing TSFrame into a CSV file
Writing a TSFrame object into a CSV file can be done easily by using the underlying coredata property. This DataFrame can be passed to the CSV.write method for writing into a file.
julia> CSV.write("/tmp/demo_ts.csv", ts)"/tmp/demo_ts.csv"
Broadcasting
Broadcasting can be used on a TSFrame object to apply a function to a subset of it's columns.
julia> using TSFrames, DataFrames;
julia> ts = TSFrame(DataFrame(Index = [1, 2, 3, 4, 5], A = [10.1, 12.4, 42.4, 24.1, 242.5], B = [2, 4, 6, 8, 10]))
(5 x 2) TSFrame with Int64 Index
Index A B
Int64 Float64 Int64
───────────────────────
1 10.1 2
2 12.4 4
3 42.4 6
4 24.1 8
5 242.5 10
julia> sin_A = sin.(ts[:, [:A]]) # get sin of column A
(5 x 1) TSFrame with Int64 Index
Index A_sin
Int64 Float64
──────────────────
1 -0.625071
2 -0.165604
3 -0.999934
4 -0.858707
5 -0.562466
julia> log_ts = log.(ts) # take log of all columns
(5 x 2) TSFrame with Int64 Index
Index A_log B_log
Int64 Float64 Float64
──────────────────────────
1 2.31254 0.693147
2 2.5177 1.38629
3 3.74715 1.79176
4 3.18221 2.07944
5 5.491 2.30259
julia> log_ts = log.(ts[:, [:A, :B]]) # can specify multiple columns
(5 x 2) TSFrame with Int64 Index
Index A_log B_log
Int64 Float64 Float64
──────────────────────────
1 2.31254 0.693147
2 2.5177 1.38629
3 3.74715 1.79176
4 3.18221 2.07944
5 5.491 2.30259
Tables.jl Integration
TSFrame objects are Tables.jl compatible. This integration enables easy conversion between the TSFrame format and other formats which are Tables.jl compatible.
As an example, first consider the following code which converts a TSFrame object into a DataFrame, a TimeArray and a CSV file respectively.
julia> using TSFrames, TimeSeries, Dates, DataFrames, CSV;
julia> dates = Date(2018, 1, 1):Day(1):Date(2018, 12, 31)
Date("2018-01-01"):Day(1):Date("2018-12-31")
julia> ts = TSFrame(DataFrame(Index = dates, x1 = 1:365));
# conversion to DataFrames
julia> df = DataFrame(ts);
# conversion to TimeArray
julia> timeArray = TimeArray(ts, timestamp = :Index);
# writing to CSV
julia> CSV.write("ts.csv", ts);
Next, here is some code which converts a DataFrame, a TimeArray and a CSV file to a TSFrame object.
julia> using TSFrames, DataFrames, CSV, TimeSeries, Dates;
# converting DataFrame to TSFrame
julia> ts = TSFrame(DataFrame(Index=1:10, x1=1:10));
# converting from TimeArray to TSFrame
julia> dates = Date(2018, 1, 1):Day(1):Date(2018, 12, 31)
Date("2018-01-01"):Day(1):Date("2018-12-31")
julia> ta = TimeArray(dates, rand(length(dates)));
julia> ts = TSFrame(ta);
# converting from CSV to TSFrame
julia> CSV.read("ts.csv", TSFrame);This discussion warrants a note about how we've implemented the Tables.jl interfaces. Since TSFrame objects are nothing but a wrapper around a DataFrame, our implementations of these interfaces just call DataFrames.jl's implementations. Moreover, while constructing TSFrame objects out of other Tables.jl compatible types, our constructor first converts the input table to a DataFrame, and then converts the DataFrame to a TSFrame object.