Back test evaluation period limited to 25 years?

[PerryB]

Hello,

I’m new to StrataSearch. When I tried to setup the Setups > Trade Setting, I got an error message: “The Evaluation Period, including Lead-up Days, cannot exceed 25 years”. Why is the Evaluation Period limited to 25 years? I’m using 4.0.921 Beta.

Perry

[Overload]

This restriction was originally put in a number of years ago when computers often ran at 750mhz with 512mb of RAM. At that time, such restrictions were necessary to make an automated search run at a reasonable pace, and without overloading a computer.

Computers run much faster now, so we could certainly eliminate this limitation, and we probably will at some point. But we've also learned over the years that an automated search will be much more efficient when it's focused on a specific time period and market rather than a search for everything all at once. For example, running a search for All Securities against a 25 year time frame would be an extremely slow search, and would probably not take advantage of the benefits that come from tracking individual Sectors. On the other hand, by focusing on a more recent time period and limiting the search to a specific sector, one can complete the search quickly and have the freedom to move onto another custom search.

Another feature worth mentioning is that a OneClick Search provides the ability to evaluate "Alternate Data". This can be a much more efficient approach than simply running against one big Evaluation Period. For example, let's say we ran against 20 years of data in our first OneClick Search. Every combination that was run was required to evaluate all 20 years of data, and this was quite slow. But let's say in the second OneClick Search, we broke it up so that our main search was only the most recent 10 years, but we had an Alternate Data evaluation for the 10 prior years. In this case, the initial 10 year period is evaluated, and only if that period passes the test will the prior 10 years be evaluated. This multi-level approach helps make the search much more efficient.

Sorry for the long answer. But, put simply, we haven't lifted the restriction because the demand hasn't been very high to do so. Most users looking to evaluate larger time frames might be doing it via multiple layers, as mentioned.

Feel free to ask if you have additional questions or comments on this.

Pete

[PerryB]

Pete,

Thanks for the explanation. I, for one, would rather see a warning message rather than a hard limit on the number of years to test. As it happens, I’m testing for long term trend signals.

Speaking of “Alternate Data”, I tried re-running a strategy with only changing the Evaluation Period date range. The Processor started by displaying Creating new combinations… and the quite. Does this mean that it won’t do any runs because of a Evaluation Period date change only? So I should convert the Strategy into a OneClick search and ask it to evaluate alternate dates?

Thanks,

Perry

[Overload]

Thanks for the explanation. I, for one, would rather see a warning message rather than a hard limit on the number of years to test. As it happens, I’m testing for long term trend signals.

Thanks for the input. This is definitely on the list as a future enhancement.

Speaking of “Alternate Data”, I tried re-running a strategy with only changing the Evaluation Period date range. The Processor started by displaying Creating new combinations… and the quite. Does this mean that it won’t do any runs because of a Evaluation Period date change only? So I should convert the Strategy into a OneClick search and ask it to evaluate alternate dates?

For efficiency reasons, a Results Database will only allow a single Strategy be placed in the database once. A "single Strategy" is defined as having the same Strategy Name, Entry and Exit Strings, Rank, Order Types, Sector, and Long/Short selection. The Evaluation Period is not included in that, so it assumes that Strategy has already been run. You can, however, easily rerun the Strategy. Just right-click on the result in the Combination Results Listing and select "Rerun Combination".

Pete

[Dacamic]

I, for one, would rather see a warning message rather than a hard limit on the number of years to test. As it happens, I’m testing for long term trend signals.

If I may ask, what data source do you use to do testing of this nature?

[PerryB]

Steve,

I don’t know the capabilities of the various data sources very well, but Reuters/Metastock goes back to 1980; and Yahoo has at least the S&P500 Index back to 1950.

Perry

[taowave]

Hi Pete,

Glad that question was asked and you answered that.I could never figure out what the issue was..

Allan

For efficiency reasons, a Results Database will only allow a single Strategy be placed in the database once. A "single Strategy" is defined as having the same Strategy Name, Entry and Exit Strings, Rank, Order Types, Sector, and Long/Short selection. The Evaluation Period is not included in that, so it assumes that Strategy has already been run. You can, however, easily rerun the Strategy. Just right-click on the result in the Combination Results Listing and select "Rerun Combination".

[Dacamic]

I don’t know the capabilities of the various data sources very well, but Reuters/Metastock goes back to 1980; and Yahoo has at least the S&P500 Index back to 1950.

As you might already know, Yahoo adjusts their data for dividends. They do offer a ton of historical data, but the dividend adjustment significantly reduces the value of that data for very long-term back tests. The same phenomenon, though, might not affect their index-based data (it's been awhile since I've looked at it).

[ovtrading]

I'm curious what sector you would test on with 25 years worth of data.

[Dacamic]

I'm curious what sector you would test on with 25 years worth of data.

In descending order of preference, i would probably limit my tests to the following groups:

1. Indexes;
2. A self-made sector whose members are selected based upon longevity;
3. Russell 2000; and,
4. All Securities.

As your question implies, sector turnover creates risk that data in back tests is not representative of present day activity. To mitigate that risk, i would focus very long-term tests on sectors that have a significant portion of its members with activity during the entire evaluation period. This approach isn't the only one that makes sense, rather it reflects my preferences and system development processes.

[JLG]

Perry,

Perhaps this is not too late to be of some help. Like you I have been restricted by the daily bar limit in StrataSearch. Here is a workaround that I have used.

1. Divide long histories into equal periods of less than twenty five years.
2. Use another program to separate each time period into a separate security and place each into a separate Ascii file.
3. Adjust the dates of each security so they cover the same 25 year time period using security name to identify source dates.
4. Import these securities.
5. Combine these securities into a sector.
6. Develop search criteria that encourage significant market participation.
7. Run single system searches against that sector with a limit on the number of securities equal to the total number of historical periods.

This is a bit of a boot strap method, is clearly suboptimal, abandons some serial continuity and overstates the value of infrequent strategies but at least it can be used with long histories.