© Copyright Quantopian Inc.
© Modifications Copyright QuantRocket LLC
Licensed under the Creative Commons Attribution 4.0.
Disclaimer
© Copyright Quantopian Inc.
© Modifications Copyright QuantRocket LLC
Licensed under the Creative Commons Attribution 4.0.
Disclaimer
by Maxwell Margenot, Gil Wassermann, James Christopher Hall, and Delaney Granizo-Mackenzie
How can we tell whether an alpha factor is good or not? Unfortunately, there is no cut-off that tells you whether a factor is objectively useful. Instead, we need to compare a particular factor to other options before deciding whether to use it. Our end goal in defining and selecting the best factors is to use them to rank stocks in a long-short equity strategy, covered elsewhere in the lecture series. The more independently predictive the factors we use, the better our ranking scheme and our overall strategy will be.
What we want when comparing factors is to make sure the chosen signal is actually predictive of relative price movements. We do not want to predict the absolute amount the assets in our universe will move up or down. We only care that we can choose assets to long that will do better than the assets we short. In a long-short equity strategy, we hold a long basket and a short basket of assets, determined by the factor values associated with each asset in our universe. If our ranking scheme is predictive, this means that assets in the top basket will tend to outperform assets in the bottom basket. As long this spread is consistent over time our strategy will have a positive return.
An individual factor can have a lot of moving parts to assess, but ideally it should be independent of other factors that you are already trading on in order to keep your portfolio diverse. We discuss the reasoning for this in the Position Concentration Risk lecture.
In this lecture, we detail and explain relevant statistics to evaluate your alpha factor before attempting to implement it in an algorithm. What's important to keep in mind is that all the metrics provided here are relative to other factors you may be trading or evaluating.
Let's have a look at a factor and try to assess its viability. We will calculate the factor values using Pipeline, so make sure you check out the Pipeline tutorial if you are unfamiliar with how Pipeline works.
from zipline.research import run_pipeline, get_forward_returns
from zipline.pipeline import Pipeline, master, EquityPricing
from zipline.pipeline.factors import Returns, AverageDollarVolume
Here we will be using a momentum factor as our example. Momentum factors are a very common form of alpha factor and they come in many shapes and sizes. They all try to get at the same idea, however, that securities in motion will stay in motion. Momentum factors try to quantify trends in financial markets and to "ride the wave", so to speak.
Let's say that we suspect that a momentum factor could potentially be predictive of stock returns.
momentum = Returns(window_length=252, exclude_window_length=21)
This momentum factor takes the change in price over the past year, up until a month ago.
In order to judge whether a factor is viable, we have created a package called Alphalens. Its source code is available on github if you want to get into the nitty-gritty of how it works. We use Alphalens to create a "tear sheet" of a factor, similar to how we use pyfolio to create a tear sheet for analyzing backtests.
import alphalens as al
Alphalens takes your factor and examines how useful it is for predicting relative value through a collection of different metrics. It breaks all the stocks in your chosen universe into different quantiles based on their ranking according to your factor and analyzes the returns, information coefficient (IC), the turnover of each quantile, and provides a breakdown of returns and IC by sector (or other grouping key).
Throughout the course of this lecture we will look at the major sections of an Alphalens tear sheet one by one and detail how to interpret the various individual plots. To do this, we will outline the intermediate steps involved to go from a pipeline definition to a tear sheet. At the end of the notebook, we show an easier way to skip the intermediate steps and generate the whole tear sheet at once.
As always, we need to define our universe. In this case we use a simplified version of the TradableStocksUS universe which does not include a Market Cap filter (to reduce the data dependencies of this lecture). The universe rules are separated into an initial_universe variable consisting of securities master fields and a screen variable consisting of all other rules. Defining an initial_universe limits the computational universe of the pipeline and thus makes it run faster than if all rules were placed in the screen. See the Pipeline Tutorial for a fuller discussion of the distinction between initial_universe and screen.
initial_universe = (
# common stocks only
master.SecuritiesMaster.usstock_SecurityType2.latest.eq("Common Stock")
# primary share only
& master.SecuritiesMaster.usstock_PrimaryShareSid.latest.isnull()
)
screen = (
# dollar volume over $2.5M over trailing 200 days
(AverageDollarVolume(window_length=200) >= 2.5e6)
# price > $5
& (EquityPricing.close.latest > 5)
# no missing data for 200 days (exclude trading halts, IPOs, etc.)
& EquityPricing.close.all_present(window_length=200)
& (EquityPricing.volume.latest > 0).all(window_length=200)
)
Now we will pull values for our factor for all stocks in our universe by using Pipeline. We also want to make sure that we have the sector code for each individual equity, so we add Sector as another factor for our Pipeline. Note that running the Pipeline may take a while.
pipe = Pipeline(
columns = {
'momentum': momentum,
'sector': master.SecuritiesMaster.usstock_Sector.latest
},
initial_universe=initial_universe,
screen=screen
)
results = run_pipeline(pipe, start_date='2010-01-01', end_date='2011-01-01', bundle='usstock-learn-1d')
Let's take a look at the data to get a quick sense of what we have.
my_factor = results['momentum']
print(my_factor.head())
date asset
2010-01-04 Equity(FIBBG000C2V3D6 [A]) 0.812808
Equity(FIBBG000M7KQ09 [AAI]) -0.011086
Equity(FIBBG000F7RCJ1 [AAP]) 0.188061
Equity(FIBBG000B9XRY4 [AAPL]) 1.170468
Equity(FIBBG000C5QZ62 [AAV]) 0.394707
Name: momentum, dtype: float64
Our my_factor variable contains a pandas Series with a factor value for each equity in our universe for each point in time.
Here we create another Series that contains sectors for each equity instead of factor values. This is categorical data that we will use as a parameter for Alphalens later.
sectors = results['sector']
Now that we have the basic components of what we need to analyze our factor, we can start to deal with Alphalens. Note that we will be breaking out individual components of the package, so this is not the typical workflow for using an Alphalens tear sheet. The typical workflow is shown at the end of the notebook.
First we calculate our forward returns. The forward returns are the returns that we would have received for holding each security over the day periods ending on the given date, passed in through the periods parameter. In our case, we look $1$, $5$, and $10$ days in advance. We can consider this a budget backtest. The tear sheet does not factor in any commission or slippage cost, rather, it only considers values as if we had magically already held the specified equities for the specified number of days up to the current day.
periods = [1, 5, 10]
forward_returns = get_forward_returns(my_factor, periods=periods, bundle="usstock-learn-1d")
Next, we pass our factor data and forward returns to Alphalens:
factor_data = al.utils.get_clean_factor(
my_factor,
forward_returns,
groupby=sectors
)
Dropped 0.9% entries from factor data: 0.9% in forward returns computation and 0.0% in binning phase (set max_loss=0 to see potentially suppressed Exceptions). max_loss is 35.0%, not exceeded: OK!
The factor_data variable here is similar to the my_factor variable above. It has a factor value for every equity in our universe at each point in time. Our Alphalens function here has also provided a sector grouping to go along with the factor value.
factor_data.head()
| 1D | 5D | 10D | factor | group | factor_quantile | ||
|---|---|---|---|---|---|---|---|
| date | asset | ||||||
| 2010-01-04 | Equity(FIBBG000C2V3D6 [A]) | 0.007403 | -0.008690 | -0.020599 | 0.812808 | Technology | 5 |
| Equity(FIBBG000M7KQ09 [AAI]) | -0.007663 | -0.022989 | 0.045977 | -0.011086 | Industrials | 1 | |
| Equity(FIBBG000F7RCJ1 [AAP]) | -0.002470 | 0.003953 | -0.028903 | 0.188061 | Consumer Discretionary | 3 | |
| Equity(FIBBG000B9XRY4 [AAPL]) | 0.015555 | 0.005922 | -0.022787 | 1.170468 | Technology | 5 | |
| Equity(FIBBG000C5QZ62 [AAV]) | 0.047546 | 0.118098 | 0.078221 | 0.394707 | Energy | 3 |
As explained above, the forward returns are the returns that we would have received for holding each security for the specified number of days, ending on the given date. These, too, are broken out by sector.
This function also separates our factor into quantiles for each date, replacing the factor value with its appropriate quantile on a given day. Since we will be holding baskets of the top and bottom quantiles, we only care about the factor insofar as it relates to movement into and out of these baskets.
Alphalens provides four categories of analysis on alpha factors:
Each of these topics is covered in its own separate tear sheet as well as an all-encompassing full tear sheet.
A full tear sheet begins with a table of statistics depicting the distribution of the factor values. This table can be displayed on its own with the following code:
al.plotting.plot_factor_distribution_table(factor_data)
| min | max | mean | std | count | avg daily count | count % | |
|---|---|---|---|---|---|---|---|
| Factor Quantile | |||||||
| 1 | -0.943 | 0.384 | -0.104 | 0.182 | 94,571 | 373.8 | 20.0% |
| 2 | -0.173 | 0.667 | 0.146 | 0.154 | 94,427 | 373.2 | 20.0% |
| 3 | -0.032 | 1.042 | 0.319 | 0.197 | 94,424 | 373.2 | 20.0% |
| 4 | 0.092 | 1.820 | 0.552 | 0.297 | 94,427 | 373.2 | 20.0% |
| 5 | 0.242 | 111.613 | 1.634 | 2.871 | 94,523 | 373.6 | 20.0% |
The table includes a row for each of the factor quantiles. By default, factors are divided into 5 equal-sized quantiles; this can be overridden using the quantiles parameter to the get_clean_factor function (or using the more typical functions shown later in the notebook).
For each quantile, we see the min, max, mean, and standard deviation of factor values. This gives us a sense of which factor values are ending up in which quantiles.
We also see the total count of values per quantile for the whole analysis, the average daily count per quantile, and the percentage count per quantile. Here, because we used equal-sized quantiles (the default option), the counts are the same for each quantile. Alternatively, we could have specified the bins parameter to get_clean_factor (or the more typically used functions shown later in the notebook) instead of quantiles. Using the bins parameter results in equal-width quantiles (spaced according to the overall range of factor values) rather than equal-sized quantiles. In that scenario, the counts will typically be different for different quantiles.
If we are solely interested in returns, we can create a tear sheet that only contains the returns analysis. The following code block generates all of our returns plots once we have stored the forward returns data:
al.tears.create_returns_tear_sheet(factor_data, by_group=True);
| 1D | 5D | 10D | |
|---|---|---|---|
| Ann. alpha | -0.045 | -0.053 | -0.040 |
| beta | 0.206 | 0.233 | 0.181 |
| Mean Relative Return Top Quantile (bps) | 1.130 | 0.890 | 0.886 |
| Mean Relative Return Bottom Quantile (bps) | -0.304 | -0.037 | -0.026 |
| Mean Spread (bps) | 1.434 | 0.861 | 0.869 |
The tear sheet calculates the $\alpha$ and $\beta$ of our factor with respect to the market. These are calculated by creating a regression of the market returns for each period against a long-short factor portfolio and extracting the parameters. These signify the excess return associated with our factor and the market beta, respectively.
These point estimates were also calculated agnostic of the sector groupings so they give us an overall view of what our spread would look like if we traded this factor with a long-short equity algorithm and didn't examine which sectors those returns were coming from.
The quantized factor is used to create separate baskets, with returns for each basket calculated over each given day period. We get this information both as bar charts and as violin plots.
These charts provide the simplest overall summary of the factor's predictive value. For a predictive factor, we want to see the mean returns monotonically increasing (or monotonically decreasing) by quantile from left to right.
The bar charts provide us the mean return for each factor quantile, for each time horizon. This gives us a set of point estimates to get a preliminary understanding of how the factor value tracks with returns.
The primary difference between a bar chart and a violin plot is that a violin plot shows the density of our data. The fatter the violin, the higher the density of returns in that region. Here we plot the violins for the $1$, $5$, and $10$ day forward returns for each quantile.
This plot shows the cumulative returns of a factor-weighted cross-sectional equity strategy based on the input factor. The portfolio holds every asset in the universe weighted according to its factor value and rebalanced daily. By default, the portfolio is a long-short portfolio, where every asset with a factor value above the daily mean is held long and every asset with a factor value below the daily mean is held short. Since the portfolio is factor-weighted, assets with factor values near the mean have smaller absolute weights and assets with factor values far from the mean have larger absolute weights. Note that this means the cumulative return plot can potentially be dominated by a small number of outliers whose values are very far away from the mean. For this reason, you should complement this plot with the Cumulative Return by Quantile plots, which uses an equal-weighted approach that is less sensitive to outliers.
This next plot aggregates the relative returns of each individual quantile into a plot of cumulative returns separated by basket for the 1-period forward returns. Within each quantile, assets are equal-weighted, not factor-weighted as in the previous plot. That makes this plot less sensitive to outliers. What we want here is to see five discrete "fingers" with few to no crossovers. This will give us an idea of which quantiles tend to drive the returns (ideally the first and fifth).
This plot is identical to the preceding plot, except that this plot shows the actual returns of each quantile whereas the preceding plot shows the relative returns (that is, relative to the overall daily mean return). In combination, these plots can reveal (for example) whether a bottom quantile of stocks, which might be candidates for shorting in a long-short strategy, had negative returns (and thus could be shorted profitably) or simply had less positive returns than higher quantiles (and thus could provide a hedge but would lose money).
Next we calculate the basis points of the spread, based on subtracting the mean return of the lowest quantile from the mean return of the highest quantile (simulating going short on the lowest and long on the highest). We also get the error and plot it all together, giving us a time series of the basis points with confidence intervals for each time period.
Because we provided sectors as a grouping key, next we get two pie plots showing the composition of the top and bottom quantiles by sector. This helps us see whether the top and bottom quantiles are concentrated in a few sectors or evenly spread across many sectors.
Because we provided a groupby argument, we also see individual plots showing the mean return by quantile for each group (sector), the relative cumulative return by quantile for each group (sector), and the actual cumulative return by quantile for each group. While it is good to consider breakdowns by quantile, it is also important to see how your factor is exposed to the different facets of the market. This is a good way to assess if your factor is behaving as it should in your universe. For example, if you intend your universe to only touch a certain sector, it is worthwhile to confirm that your factor and universe indeed only touch that sector.
Note that you are not limited to a single grouping key. You can provide a list of grouping keys to the groupby argument, and Alphalens will show group-specific plots for each set of grouping keys.
We use the information coefficient (IC) to assess the predictive power of a factor. The IC of a factor is its Spearman Rank Correlation. For more background on the mathematics associated with the IC, check out the Spearman Rank Correlation Lecture. To break it down, we calculate the IC between the factor values and the forward returns for each period. The IC assesses the monotonic relationship between factors and returns. What this means, intuitively, is that it provides a measure for whether higher factor values can be associated with higher returns.
A higher IC indicates that higher factor values are more closely associated with higher return values (and lower factor values with lower return values). A very negative IC indicates that higher factor values are closely associated with lower return values. An IC of $0$ indicates no relationship.
Similarly to the returns tear sheet, we can also create a tear sheet that just gives us data on the information coefficient.
al.tears.create_information_tear_sheet(factor_data, by_group=True)
| 1D | 5D | 10D | |
|---|---|---|---|
| IC Mean | 0.002 | 0.003 | 0.003 |
| IC Std. | 0.125 | 0.139 | 0.124 |
| Risk-Adjusted IC | 0.017 | 0.023 | 0.027 |
| t-stat(IC) | 0.269 | 0.373 | 0.436 |
| p-value(IC) | 0.788 | 0.709 | 0.663 |
| IC Skew | -0.184 | -0.038 | -0.096 |
| IC Kurtosis | 0.531 | -0.097 | -0.123 |
This first table provides us with statistics around the distribution of the information coefficient for each time horizon. This includes mean, standard deviation, skewness, and kurtosis, to give us a clear view of how the distribution is shaped.
Here we plot the IC as a time series for each period along with a 1-month moving average to smooth it out. What we want here is consistency over time and a consistently informative signal.
Histograms are good to show us the distribution of the IC. These will clearly show any strange outliers and how they affect the overall curve.
A QQ-plot compares the distribution of the IC to the normal distribution. It plots the quantiles of one distribution against the quantiles of the other, typically with a reference line at $y = x$. If the points in the QQ-plot fall entirely along this line, this indicates that the two distributions are the same. In practice, a QQ-plot serves as a measure of similarity between distributions. Generally, what we want to see here is an S-shaped curve. This indicates that the tails of the IC distribution are fatter and contain more information.
Looking at the returns by quantile for each individual sector in the returns tear sheet helps to show which sectors are driving the bulk of our returns as well as whether quantiles in each sector are broken out how they should be (with the lowest quantiles giving the lowest returns up to the highest quantiles giving the highest returns). If an individual sector has little to no signal (IC), it makes sense for the quantile returns to be all over the place. Looking at IC by group helps to make sure that everything is behaving nicely.
When considering the impact of actually implementing a signal in a strategy, turnover is a critical thing to consider. This plot shows the turnover of the top and bottom quantiles of your factor, the baskets that you would actually be trading on with a long-short approach. Excessive turnover will eat away at the profits of your strategy through commission costs. Sometimes a signal just isn't good enough to fight against the tide on the scale that you have to deal with through your broker.
We can examine two separate metrics of turnover using Alphalens, percent turnover and factor autocorrelation.
al.tears.create_turnover_tear_sheet(factor_data);
| 1D | 5D | 10D | |
|---|---|---|---|
| Quantile 1 Mean Turnover | 0.061 | 0.141 | 0.206 |
| Quantile 2 Mean Turnover | 0.137 | 0.300 | 0.410 |
| Quantile 3 Mean Turnover | 0.153 | 0.334 | 0.448 |
| Quantile 4 Mean Turnover | 0.124 | 0.277 | 0.381 |
| Quantile 5 Mean Turnover | 0.053 | 0.123 | 0.178 |
| 1D | 5D | 10D | |
|---|---|---|---|
| Mean Factor Rank Autocorrelation | 0.993 | 0.966 | 0.932 |
The first table is a plot of turnover for each quantile by period. This indicates the percentage of names that move in and out of each basket.
Factor autocorrelation is the measure of correlation between the current value of the factor and its previous value. The idea behind its calculation is to provide another measure of the turnover of the factor quantiles. If the autocorrelation is low, it implies that the current value of the factor has little to do with the previous value and that portfolio positions are changing frequently from time period to time period. If the next value of the factor is significantly influenced by its last value, this means that your ranking scheme is more consistent (though this has no influence on its ability to forecast relative price movements).
These plot shows a fairly ordinary turnover for the factor, implying that we will not be hit too hard by the constant changing of portfolio positions, though this varies from time period to time period. We cannot see the costs directly, however, because Alphalens does not model commission costs. It simply provides metrics that we can use to judge a factor by itself. To properly model undermining influences such as slippage and commissions you will need to implement a strategy that uses your factor in a backtest.
The second set of plots in the turnover tear sheet is the factor autocorrelation over time. In this case, we have fairly high autocorrelation, corroborating the turnover plots from above that suggest more consistent portfolio positions.
We can also generate all of the above plots and statistics in one complete tear sheet. If you've followed the above steps to generate factor data and forward returns, you can generate a full tear sheet all at once as follows:
al.tears.create_full_tear_sheet(factor_data, by_group=True);
However, an even easier approach is to go directly from a pipeline definition to a full tear sheet, letting Alphalens take care of the intermediate steps of executing the pipeline, computing forward returns, and formatting the factor data. Starting with our pipeline definition from earlier:
pipe = Pipeline(
columns = {
'momentum' : momentum,
'sector' : master.SecuritiesMaster.usstock_Sector.latest
},
initial_universe=initial_universe,
screen=screen
)
We can simply pass the pipeline definition itself to the Alphalens from_pipeline function, specifying the column name that contains the factor we want to analyze, the optional column name(s) containing the grouping key, and other parameters, and get a full tearsheet:
al.from_pipeline(
pipe,
start_date='2010-01-01',
end_date='2011-01-01',
bundle='usstock-learn-1d',
factor="momentum", # pipeline column name containing the factor
groupby="sector", # optional pipeline column name(s) to group by
periods=[1, 5, 10] # forward returns periods
)
Besides convenience, an important benefit of from_pipeline is that it supports segmented analysis, which means it can analyze large date ranges even if the overall amount of data is too large to fit into memory. To take advantage of this feature, pass a pandas offset alias (such as "Y" for year) as the segment parameter. This causes Alphalens to split the overall date range into year-long segments. Alphalens runs each segment sequentially and concatenates the results to produce the final tear sheet. This cuts down on memory usage because Alphalens can discard intermediate data structures and keep only the necessary outputs.
al.from_pipeline(
pipe,
start_date='2008-01-10',
end_date='2011-12-10',
bundle='usstock-learn-1d',
factor="momentum", # pipeline column name containing the factor
groupby="sector", # optional pipeline column name(s) to group by
periods=[1, 5, 10], # forward returns periods
segment="Y",
)
Coming up with new factors is all well and good, but often you will need many independently predictive factors to produce a signal that is stable enough to trade. Methods of aggregating factors together will be discussed in future lectures, but the most simple initial approach would just be to normalize the values of each factor you would like to include in the aggregate, add the new normalized factor together, and rank by this new combined factor value.
Once you have a factor that looks good, the next step is to implement it in an algorithm. Unfortunately, it isn't enough to simply have a good signal. Trading algorithms have to take into account many other considerations that are not included in Alphalens. We need to include how the market at large will react to the trades we're making (market impact/slippage) as well as the transaction costs associated with making those trades. These influences can erode our profits if we do not properly assess their impact through extensive testing.
To this end, we have full backtests using Zipline. It allows for slippage and transaction cost modeling and lets you set limitations for the amount of leverage (debt) that your algorithm can take on to make its trades. Learn more about leverage in the Leverage lecture.
This presentation is for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation for any security; nor does it constitute an offer to provide investment advisory or other services by QuantRocket LLC ("QuantRocket"). Nothing contained herein constitutes investment advice or offers any opinion with respect to the suitability of any security, and any views expressed herein should not be taken as advice to buy, sell, or hold any security or as an endorsement of any security or company. In preparing the information contained herein, the authors have not taken into account the investment needs, objectives, and financial circumstances of any particular investor. Any views expressed and data illustrated herein were prepared based upon information believed to be reliable at the time of publication. QuantRocket makes no guarantees as to their accuracy or completeness. All information is subject to change and may quickly become unreliable for various reasons, including changes in market conditions or economic circumstances.