Zipline Bundle¶

In this notebook, we create a Zipline bundle that ingests data from two sources: the custom database containing our imported futures data, and the IBKR database containing recent futures data. Creating a bundle is a convenient way to combine the two data sources, even if you don't intend to use Zipline. We can keep the bundle up-to-date by updating the IBKR history database and re-running the ingestion, while the imported data provides deeper historical coverage.

First, create an empty bundle using the create_bundle_from_db function, specifying both source databases in the from_db argument. Make sure to list the IBKR database first ('ibkr-futures-1d' in this example): if a SID is found in both databases, only the price history from the database listed first will be ingested. Since the IBKR database will be updated going forward, we want to give it priority.

Choose a start date corresponding to the start date of your imported data. For a list of available calendars, see the usage guide.

In [1]:

from quantrocket.zipline import create_bundle_from_db

create_bundle_from_db(
    "futures-1d-bundle",
    from_db=[
        "ibkr-futures-1d",
        "imported-futures-1d",
    ],
    calendar="CME",
    start_date="2000-01-02",

)

Out[1]:

{'status': 'success', 'msg': 'successfully created futures-1d-bundle bundle'}

Next, ingest the data:

In [2]:

from quantrocket.zipline import ingest_bundle
ingest_bundle("futures-1d-bundle")

Out[2]:

{'status': 'the data will be ingested asynchronously'}

Monitor flightlog for completion:

quantrocket.zipline: INFO [futures-1d-bundle] Ingesting futures-1d-bundle bundle
quantrocket.zipline: INFO [futures-1d-bundle] Ingested 6658 total records for 18 total securities in futures-1d-bundle bundle

You can spot-check the bundle to make sure that early data from the imported history db is present:

In [4]:

from quantrocket import get_prices
prices = get_prices(
    "futures-1d-bundle",
    start_date="2008-09-02",
    end_date="2008-09-15",
    fields="Close"
)
prices

Out[4]:

	Sid	QF000000654947	QF000000654948
Field	Date
Close	2008-09-02	1276.50	1278.00
	2008-09-03	1275.25	1276.75
	2008-09-04	1236.50	1238.00
	2008-09-05	1241.00	1242.50
	2008-09-08	1267.00	1268.50
	2008-09-09	1226.50	1227.75
	2008-09-10	1233.25	1234.50
	2008-09-11	1251.00	1252.00
	2008-09-12	1257.25	1258.50
	2008-09-15	1195.00	1196.00

The bundle is now ready to use. To update the bundle, collect the IBKR history database again, then re-run the ingestion. Since the imported database is non-updating, no further action with it is necessary.

Automatic collection can be scheduled on your crontab. For example:

# collect IBKR history, wait for it to finish, then re-ingest the Zipline bundle
0 20 * * mon-fri quantrocket history collect 'ibkr-futures-1d' && quantrocket history wait 'ibkr-futures-1d' && quantrocket zipline ingest 'futures-1d-bundle'