We will complement the imported futures data with recent historical data from Interactive Brokers. The two data sources will be combined in a Zipline bundle in the next notebook.
First, start IB Gateway:
from quantrocket.ibg import start_gateways
start_gateways(wait=True)
{'ibg1': {'status': 'running'}}Next, we need to collect contract details for the IBKR futures we are interested in. For the purpose of this tutorial, we are only collecting ES futures:
from quantrocket.master import collect_ibkr_listings
collect_ibkr_listings(
countries="US",
sec_types="FUT",
symbols="ES"
)
{'status': 'the IBKR listing details will be collected asynchronously'}Monitor flightlog for completion:
quantrocket.master: INFO Collecting US FUT listings from IBKR website (ES only)
quantrocket.master: INFO Requesting details for 1 US FUT listings found on IBKR website
quantrocket.master: INFO Generating rollover dates for all IBKR FUT contracts
quantrocket.master: INFO Saved 29 US FUT listings to securities master database
quantrocket.master: INFO Building consolidated securities master from active vendors: ibkr, usstock
quantrocket.master: INFO Completed building consolidated securities master with 16217 recordsNext we define a universe of futures for easy reference. We want to include the IBKR futures we just collected as well as the imported futures contracts. To do so, we query the futures from the securities master database:
from quantrocket.master import get_securities
contracts = get_securities(
exchanges="CME",
sec_types="FUT",
symbols="ES",
)
Then upload the sids to create the "futures" universe:
from quantrocket.master import create_universe
create_universe("futures", sids=contracts.index.tolist())
{'code': 'futures',
'provided': 125,
'inserted': 125,
'total_after_insert': 125}NOTE: You should periodically collect new contracts from IBKR and append them to your universe, as explained in the usage guide. This step can be automated on your countdown crontab.
Next we collect historical data for the universe we created. To do so, first create an IBKR history database tied to the 'futures' universe:
from quantrocket.history import create_ibkr_db
create_ibkr_db(
"ibkr-futures-1d",
universes="futures",
bar_size="1 day",
)
{'status': 'successfully created quantrocket.v2.history.ibkr-futures-1d.sqlite'}Then collect the data:
from quantrocket.history import collect_history
collect_history("ibkr-futures-1d")
{'status': 'the historical data will be collected asynchronously'}Monitor flightlog for completion:
quantrocket.history: INFO [ibkr-futures-1d] Collecting history from IBKR for 48 securities in ibkr-futures-1d
...
quantrocket.history: INFO [ibkr-futures-1d] Saved 6031 total records for 16 total securities to quantrocket.v2.history.ibkr-futures-1d.sqliteYou may see a warning such as the following:
quantrocket.history: WARNING Ignoring 77 of 125 securities with no ibkr_ConId field: QF000000654904, QF000000654905, QF000000654906, QF000000654907, QF000000654908 and 72 more. You must collect securities master listings from IBKR before before you can collect historical data from IBKR, see http://qrok.it/h/ibhsid for help.This is expected and harmless, as it simply reflects that the imported futures contracts do not have the ibkr_ConId field populated since they did not originate from Interactive Brokers. (ConId stands for "contract ID" and is Interactive Brokers' unique security identifier.)
You will also see warnings like this:
WARNING [ibkr-futures-1d] IBKR reports ESU6 FUT (sid IB81037229) cannot be found in their system, this is commonly due to a stock delisting or switching exchanges or a derivative contract expiring, see http://qrok.it/h/err/200 for more help (error code 200: No security definition has been found for the request)This is expected due to IBKR removing futures that expired more than 2 years ago from their system.