Data¶

The data model and how data is read: DataFetcher and the containers behind it, currencies and FX, the data store, file import, Postgres stores and downloads.

data ¶

Market and reference data: the containers that hold it, the DataFetcher every calculation reads through, and loaders that build them.

MarketData ¶

MarketData(file_path: str, date_format: str = '%Y-%m-%d')

Time-series data container backed by a MultiIndex DataFrame.

The source file must contain at least IDENTIFIER and DATE columns. After loading the DataFrame is indexed on (IDENTIFIER, DATE) and sorted, enabling fast .loc slicing by identifier or list of identifiers.

data `property` ¶

data: DataFrame

Return a copy of the underlying DataFrame.

identifiers `property` ¶

identifiers: list[str]

Unique identifiers present in the dataset.

Cached on the frame's identity, so the scan of the whole MultiIndex happens once per frame. fetch_fx_rates consults this on every call to decide whether a pair exists.

identifier_set `property` ¶

identifier_set: frozenset[str]

:attr:identifiers as a set, cached the same way.

columns `property` ¶

columns: list[str]

Non-index column names.

Cached on the frame's identity, like identifiers, because it is consulted on every price read.

sessions `property` ¶

sessions: DatetimeIndex

The distinct dates the dataset carries, ascending.

Cached on the frame's identity, like identifiers. Materialising the DATE level is an O(rows) take over the whole frame, and date_range and last_session_on_or_before both read it.

date_range `property` ¶

date_range: tuple[Timestamp, Timestamp]

(earliest, latest) timestamps in the dataset.

from_dataframe `classmethod` ¶

from_dataframe(
    df: DataFrame, date_format: str = "%Y-%m-%d"
) -> MarketData

Create a MarketData instance from an existing DataFrame.

drop_non_positive_prices ¶

drop_non_positive_prices(
    exempt: Collection[str] = (),
) -> None

Read every price at or below zero as missing; see the function.

last_session_on_or_before ¶

last_session_on_or_before(
    date: str | Timestamp,
) -> pd.Timestamp | None

The latest date the dataset carries at or before date.

A date inside the coverage that has no rows is an ordinary closed market, and this is the session that was in force through it. None when the dataset begins after date, since then there is no earlier session to be in force.

session_columns ¶

session_columns(
    date: str | Timestamp,
) -> tuple[dict[str, dict[str, object]], set[str]]

Every column's values for one session, keyed by identifier.

The read the daily loops make once per session, done without touching pandas. The frame is indexed (IDENTIFIER, DATE), so a single day's rows are scattered through it rather than adjacent, and get() has to filter the whole frame to find them. This looks up the row positions for the date from a per-frame index instead, so the read is a gather and two dict builds. Where an (identifier, date) pair is repeated, the first row wins, as it does for get() followed by .iloc[0].

Returns:

Name	Type	Description
`tuple`	`dict[str, dict[str, object]]`	`({column: {identifier: value}}, identifiers)`, exactly
	`set[str]`	what a :class:`~beacon.data.session.SessionPanel` holds, so no
	`tuple[dict[str, dict[str, object]], set[str]]`	DataFrame is built on the way. Empty for a date the data has no
	`tuple[dict[str, dict[str, object]], set[str]]`	rows on.

session_rows ¶

session_rows(
    date: str | Timestamp,
) -> (
    tuple[np.ndarray, np.ndarray, dict[str, np.ndarray]]
    | None
)

The frame's identifiers, one session's row positions, and the column arrays, with nothing gathered or built; None for a date with no rows. What a lazy session panel holds.

codes_for ¶

codes_for(identifiers: list[str]) -> np.ndarray

Each identifier's code among this frame's rows, -1 for a name with no rows.

The last answer is kept for the same list object, so a caller asking about the same names day after day looks them up once.

session_values ¶

session_values(
    date: Timestamp, codes: ndarray, column: str
) -> np.ndarray | None

column on date for the names with codes, NaN where a name has no row; None when the column is missing or not a float column.

The values the session panel gives, as one array: where a name has several rows on the day the first is taken, as the panel's reversed fill takes it.

last_values ¶

last_values(
    codes: ndarray,
    start: Timestamp,
    end: Timestamp,
    column: str,
) -> np.ndarray | None

column's last value from start to end for the names with codes, NaN where a name has none in the window; None when the column is missing or not a float column.

What a read of the window gives as each name's last value, its last row with one, found session by session from the end instead of by slicing every name's history.

get ¶

get(
    identifier: str | list[str],
    start_date: str | None = None,
    end_date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Return data for one or more identifiers, optionally filtered.

Parameters:

Name	Type	Description	Default
`identifier`	`str \| list[str]`	Single identifier or list of identifiers.	required
`start_date`	`str \| None`	Date string to slice the start of the date range.	`None`
`end_date`	`str \| None`	Date string to slice the end of the date range.	`None`
`columns`	`list[str] \| None`	Subset of columns to return.	`None`

Returns:

Name	Type	Description
	`DataFrame`	pd.DataFrame: Single identifier: indexed by `DATE`. List of
`identifiers`	`DataFrame`	MultiIndexed by `(IDENTIFIER, DATE)`. Empty
	`DataFrame`	DataFrame if no matching data is found.

ReferenceData ¶

ReferenceData(file_path: str)

Reference data container with validity ranges.

The source file must contain IDENTIFIER and DATE_FROM columns. DATE_TO is optional: a missing column, or NaT in it, marks a currently-active record.

Indexed on IDENTIFIER (non-unique, since an identifier may have multiple validity periods).

data `property` ¶

data: DataFrame

Return a copy of the underlying DataFrame.

identifiers `property` ¶

identifiers: list[str]

Unique identifiers present in the dataset, cached on the frame's identity.

columns `property` ¶

columns: list[str]

Column names (including DATE_FROM, DATE_TO).

from_dataframe `classmethod` ¶

from_dataframe(df: DataFrame) -> ReferenceData

Create a ReferenceData instance from an existing DataFrame.

get ¶

get(
    identifier: str | list[str],
    date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Return reference data for one or more identifiers.

Parameters:

Name	Type	Description	Default
`identifier`	`str \| list[str]`	Single identifier or list of identifiers.	required
`date`	`str \| None`	Point-in-time date. If provided, only rows where `DATE_FROM <= date` and (`DATE_TO >= date` or `DATE_TO` is NaT) are returned.	`None`
`columns`	`list[str] \| None`	Subset of columns to return.	`None`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Indexed by `IDENTIFIER`. Empty DataFrame if no
`DataFrame`	match.

DataFetcher ¶

DataFetcher(
    market_data: MarketData,
    reference_data: ReferenceData | None = None,
    corporate_actions: CorporateActions | None = None,
    features: FeatureData | None = None,
    fx_policy: str = DEFAULT_FX_POLICY,
    max_price_staleness_days: int | None = None,
    free_float_backfill_days: int = DEFAULT_FREE_FLOAT_BACKFILL_DAYS,
)

One query interface over market, reference, corporate-action and feature data.

Parameters:

Name	Type	Description	Default
`market_data`	`MarketData`	Time-series market data container. Currency pairs and short rates live in it as series of their own, with their values in `CLOSE`.	required
`reference_data`	`ReferenceData \| None`	Reference data container, or None if there is none. It says which series are currency pairs and short rates, and which can be held (see `beacon.data.kinds`).	`None`
`corporate_actions`	`CorporateActions \| None`	Action history. Absent means an empty history rather than None, so callers never have to check before asking: "this instrument paid nothing" and "we hold no action data" give the same answer to every question this class can be asked.	`None`
`features`	`FeatureData \| None`	Feature table. Absent means an empty table, on the same terms.	`None`
`fx_policy`	`str`	How an FX rate is read on a day the pair did not print one. `"CARRY_FORWARD"` (the default) uses the last rate published; `"EXACT_DAY"` answers only when the pair printed a rate on that very day. It applies to every conversion made through this fetcher.	`DEFAULT_FX_POLICY`
`max_price_staleness_days`	`int \| None`	How many calendar days a name may go without trading before `stale_identifiers` reports it as not worth holding. None (the default) keeps every name, whenever it last traded.	`None`
`free_float_backfill_days`	`int`	How many calendar days a reported free float carries forward over blank cells (default 90; 0 turns carrying off). See `beacon.data.free_float`.	`DEFAULT_FREE_FLOAT_BACKFILL_DAYS`

Raises:

Type	Description
`ValueError`	If `fx_policy` is not a known policy, `max_price_staleness_days` is below 1, `free_float_backfill_days` is negative or not an integer, the market data has a `RATE` column, the reference data describes a series wrongly, or a short rate looks like a percent.

identifiers `property` ¶

identifiers: list[str]

Unique identifiers present in market data.

fx_pairs `property` ¶

fx_pairs: list[str]

Spot currency pairs with market data: the identifiers reference data gives ASSET_CLASS Currency and SECURITY_TYPE Spot.

instrument_identifiers `property` ¶

instrument_identifiers: list[str]

Market identifiers a fund can hold: every series but the ones reference data marks as not tradable, such as currency pairs and short rates.

tradable_identifiers `property` ¶

tradable_identifiers: list[str]

What a universe can be drawn from: the reference data's identifiers (else the market data's), less those not tradable.

market_columns `property` ¶

market_columns: list[str]

Column names in the market data.

reference_identifiers `property` ¶

reference_identifiers: list[str] | None

Unique identifiers in the reference data, or None if not loaded.

reference_columns `property` ¶

reference_columns: list[str] | None

Column names in the reference data, or None if not loaded.

date_range `property` ¶

date_range: tuple[Timestamp, Timestamp]

(earliest, latest) timestamps in the market data.

corporate_actions `property` ¶

corporate_actions: CorporateActions

The action history. Empty rather than None when none was loaded.

features `property` ¶

features: FeatureData

The feature table. Empty rather than None when none was loaded.

Exposed for persistence and for discovery, on the same terms as market. Point-in-time reads go through fetch_feature and fetch_features, not through this.

market `property` ¶

market: MarketData

The market-data container itself.

Exposed for persistence (beacon.data.store): writing a fetcher to disk means reading back everything it holds, and the summarising properties cannot reconstruct a frame. Query through fetch_market_data instead: this is the whole dataset, not an answer to a question.

source `property` ¶

source: str | None

Where this data was loaded from, or None if nothing recorded it.

Describes the load, not every row: a later sync merges rows from somewhere else without changing where the store came from. Modelling mixed provenance would need a source per row, which nothing asks for.

store_path `property` ¶

store_path: Path | None

The store this was loaded from, if it came from one.

reference `property` ¶

reference: ReferenceData | None

The reference-data container, or None if none was loaded.

Exposed for persistence, on the same terms as :attr:market.

with_settings ¶

with_settings(
    fx_policy: str = KEEP,
    max_price_staleness_days: int | None = KEEP,
    free_float_backfill_days: int = KEEP,
) -> DataFetcher

This data read under other settings.

The same object when nothing differs. Otherwise a new fetcher over the same data, with its own caches, since a cached FX series or free-float history depends on the settings it was read under. Pass KEEP (the default) for a setting to leave as it is.

Raises:

Type	Description
`ValueError`	As the constructor does, for a setting it refuses.

is_tradable ¶

is_tradable(identifier: str) -> bool

Whether a fund can hold identifier: false for a series reference data marks as not tradable.

short_rate ¶

short_rate(currency: str) -> ShortRate | None

A currency's short rate, as reference data describes it, or None.

Where a currency has several, the one month rate is read, else the overnight, else the first by identifier.

fetch_short_rates ¶

fetch_short_rates(
    currency: str,
    start_date: str | None = None,
    end_date: str | None = None,
) -> pd.Series

A currency's short rate series, annual decimals, from its CLOSE. Empty when the data holds none.

short_rate_on ¶

short_rate_on(
    currency: str, date: str | Timestamp
) -> float | None

A currency's short rate in force on date: the last on or before it. None when the data holds none that early.

resolve_session ¶

resolve_session(
    date: str | Timestamp,
) -> pd.Timestamp | None

The market session date resolves to, backfilling inside the data.

A date the data has no bar for but which sits inside its coverage is a day the market was shut. The last session on or before it is the one that was actually in force through the closure: reading it is not an approximation, it is what the day was. Past the last bar nothing is known, so that answers None rather than a stale print wearing a current date; the bound is the data's own coverage rather than a day count, because no day count can tell a long closure from the unknown future.

Parameters:

Name	Type	Description	Default
`date`	`str \| Timestamp`	The date asked about.	required

Returns:

Type	Description
`Timestamp \| None`	pd.Timestamp \| None: The session, or None when date falls
`Timestamp \| None`	outside the data's coverage on either side.

fetch_feature ¶

fetch_feature(
    identifier: str,
    field: str,
    date: str | Timestamp | None = None,
    feature_type: str | None = None,
    max_age_days: int | None = MAX_AGE_DAYS,
) -> float | None

One feature value, as it was knowable on a date.

The point-in-time read. A value published after date is invisible, which is what keeps a backtest from screening on numbers nobody had.

Returns:

Type	Description
`float \| None`	float \| None: The value, or None when nothing is knowable. An
`float \| None`	instrument with no coverage is an ordinary answer, not an error:
`float \| None`	most datasets cover most names most of the time and not all of
`float \| None`	them all of it.

fetch_features ¶

fetch_features(
    identifiers: list[str],
    fields: list[str],
    date: str | Timestamp | None = None,
    feature_type: str | None = None,
    max_age_days: int | None = MAX_AGE_DAYS,
) -> dict[str, dict[str, float | None]]

Several features for several instruments, on one date.

The batch form, on the same argument the reference batch endpoint made: a client that has to fan out per name will, and moving the fan-out inside the server only relocates the cost.

Returns:

Name	Type	Description
`dict`	`dict[str, dict[str, float \| None]]`	identifier -> field -> value. Every requested pair is
	`dict[str, dict[str, float \| None]]`	present, null where nothing is knowable, so a caller reads a value
	`dict[str, dict[str, float \| None]]`	rather than testing for a key.

replace_features ¶

replace_features(features: FeatureData) -> None

Swap the feature table, stamping the refresh.

The only mutating method on a fetcher, and it exists because an import has to land somewhere the next request can see. It replaces rather than edits: merged_with builds the new table, so a read in flight keeps the frame it started with instead of watching rows appear under it.

feature_types ¶

feature_types() -> list[str]

Datasets the loaded features carry, for discovery.

feature_fields ¶

feature_fields(
    feature_type: str | None = None,
) -> list[str]

Fields the loaded features carry, optionally within one dataset.

record_origin ¶

record_origin(
    source: str, path: Path | None = None
) -> None

Note where this fetcher's data was loaded from.

delisting_dates ¶

delisting_dates() -> dict[str, pd.Timestamp]

The last date each identifier is listed, for those whose life ends.

Resolved in one pass rather than per identifier per day: an index over five thousand names and ten years would otherwise make twelve million point-in-time lookups to find a few hundred delistings.

A name is treated as still listed if any of its records is open-ended, even when another record for it has an end date.

Returns:

Name	Type	Description
`dict`	`dict[str, Timestamp]`	identifier -> last listed date. Names that never leave are
	`dict[str, Timestamp]`	absent, so an empty mapping means a constant universe and callers
	`dict[str, Timestamp]`	can skip the work entirely.

record_refresh ¶

record_refresh(
    dataset: str, when: datetime | None = None
) -> None

Note that a dataset has just been refreshed.

Parameters:

Name	Type	Description	Default
`dataset`	`str`	One of `DATASETS`: `"market"`, `"reference"`, `"corporate_actions"`, `"features"` or `"fx"`.	required
`when`	`datetime \| None`	The moment. None uses now, which is what a real sync wants; tests pass an explicit time so an age can be asserted rather than approximated.	`None`

Raises:

Type	Description
`ValueError`	If the dataset is not one this fetcher holds.

last_refreshed ¶

last_refreshed(dataset: str) -> datetime | None

When a dataset was last loaded or synced.

Returns:

Type	Description
`datetime \| None`	datetime or None: The moment, or None when the dataset is not
`datetime \| None`	loaded at all, which is a different statement from "loaded and
`datetime \| None`	never refreshed" and should not be collapsed into it. Loading
`datetime \| None`	counts as a refresh.

Raises:

Type	Description
`ValueError`	If the dataset is not one of `DATASETS`.

age_seconds ¶

age_seconds(
    dataset: str, now: datetime | None = None
) -> float | None

How long ago a dataset was last refreshed, in seconds.

Parameters:

Name	Type	Description	Default
`dataset`	`str`	Which dataset.	required
`now`	`datetime \| None`	The reference moment, for tests.	`None`

Returns:

Type	Description
`float \| None`	float or None: The age, or None when the dataset is not loaded.
`float \| None`	Never negative: a clock adjustment between the two readings would
`float \| None`	otherwise report data refreshed in the future, which is noise
`float \| None`	rather than information.

merge_market_data ¶

merge_market_data(frame: DataFrame) -> int

Fold freshly ingested rows into the market data.

Newly fetched rows win where they overlap an existing identifier and date. A re-sync of a window is a correction (a restated close, a backfilled volume), so keeping the older value would make the sync pointless.

The swap at the end is a single assignment, so a reader either sees the whole old dataset or the whole new one. This process is single-threaded and cooperatively scheduled, so there is no torn state to guard against; a reader that started before the swap simply finishes against the data it began with.

Parameters:

Name	Type	Description	Default
`frame`	`DataFrame`	Long-form rows carrying `IDENTIFIER` and `DATE`.	required

Returns:

Name	Type	Description
`int`	`int`	Rows added, counting only genuinely new identifier/date
	`int`	pairs. A re-sync that restates existing rows returns 0, which is
	`int`	the truthful answer to "how much did this add".

merge_reference_data ¶

merge_reference_data(frame: DataFrame) -> int

Fold freshly ingested reference records in.

A record matching an existing identifier and DATE_FROM replaces it.

Parameters:

Name	Type	Description	Default
`frame`	`DataFrame`	Rows carrying `IDENTIFIER` and `DATE_FROM`.	required

Returns:

Name	Type	Description
`int`	`int`	Records added, counting only genuinely new identifier and
	`int`	`DATE_FROM` pairs.

fetch_corporate_actions ¶

fetch_corporate_actions(
    identifier: str,
    start_date: str | Timestamp | None = None,
    end_date: str | Timestamp | None = None,
    types: list[str] | None = None,
) -> pd.DataFrame

Corporate actions for one identifier over a window.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`start_date`	`str \| Timestamp \| None`	Earliest ex-date, inclusive.	`None`
`end_date`	`str \| Timestamp \| None`	Latest ex-date, inclusive.	`None`
`types`	`list[str] \| None`	Restrict to these action types.	`None`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Matching actions, oldest first; empty when there are
`DataFrame`	none.

fetch_trailing_dividend ¶

fetch_trailing_dividend(
    identifier: str, as_of: str | Timestamp
) -> float

Ordinary dividends per share over the trailing twelve months.

fetch_trailing_dividend_yield ¶

fetch_trailing_dividend_yield(
    identifier: str,
    as_of: str | Timestamp,
    price: float | None = None,
) -> float | None

Trailing dividend yield, priced off the market data by default.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`as_of`	`str \| Timestamp`	End of the trailing window.	required
`price`	`float \| None`	Price to divide by. None reads the close on or before as_of from the market data.	`None`

Returns:

Name	Type	Description
	`float \| None`	float or None: The yield, or None when no positive price is
`available`	`float \| None`	a missing price is a reason to say nothing rather than
	`float \| None`	to guess.

fetch_market_data ¶

fetch_market_data(
    identifier: str | list[str],
    start_date: str | None = None,
    end_date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Fetch time-series market data for one or more identifiers.

Parameters:

Name	Type	Description	Default
`identifier`	`str \| list[str]`	One identifier or a list of identifiers.	required
`start_date`	`str \| None`	Date string to filter the start of the date range.	`None`
`end_date`	`str \| None`	Date string to filter the end of the date range.	`None`
`columns`	`list[str] \| None`	Subset of columns to return.	`None`

Returns:

Name	Type	Description
	`DataFrame`	pd.DataFrame: Single identifier: indexed by `DATE`. Multiple
`identifiers`	`DataFrame`	MultiIndexed by `(IDENTIFIER, DATE)`. Empty
	`DataFrame`	DataFrame if no matching data is found.

warm_session ¶

warm_session(
    identifiers: list[str], date: str | Timestamp
) -> None

Read one session's rows for identifiers in a single slice.

A hint, not a contract: every read this serves answers identically without it, only slower. Without it, a methodology walking a universe makes one frame slice per name per column, each costing what the whole frame costs rather than what one row does, so a preview's cost per name climbs with the size of its universe.

A selection rule prices every candidate and the weighting scheme then prices the survivors, the same names on the same day. The second warm is a subset of the first, so it keeps the panel rather than rebuilding it, and the reads that follow are free.

Nothing goes stale under it: the panel answers only for the identifiers it was built with, only on its own session, and a merge clears it. Calling this with a different session or a name it does not hold replaces it, so the caller never has to say when it is done.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	The instruments about to be read one at a time.	required
`date`	`str \| Timestamp`	The session they will be read on. Resolve it first with :meth:`resolve_session`, since a panel for a closed day holds nothing and every read would fall back to the frame.	required

fetch_shares_outstanding ¶

fetch_shares_outstanding(
    identifier: str,
    date: str,
    column: str = "SHARES_OUTSTANDING",
) -> float | None

Return shares outstanding for identifier on date.

Sourced from the column market-data field. Returns None if the column is not present or there is no value on that date.

fetch_free_float_factor ¶

fetch_free_float_factor(
    identifier: str, date: str, column: str = "FREE_FLOAT"
) -> float | None

Return the free-float factor in force for identifier on date.

That day's value when there is one. Otherwise the last value before it, if no older than free_float_backfill_days: free float moves on corporate events and reviews, so a blank cell means nothing was reported, not that the float changed. Never a later value.

Returns None if the column is absent, or nothing was reported within the window. Callers refuse through :func:~beacon.data.free_float.require_free_float rather than choosing a fallback of their own.

prices_on ¶

prices_on(
    identifiers: list[str], date: str, column: str = "CLOSE"
) -> dict[str, float | None]

:meth:fetch_price for many names on one day, in one read.

The batch face of the same answer: warm the day's session once, take the whole column from it, and convert NaN to None, exactly what fetch_price returns name by name. The daily valuation asks this for every holding every day.

A name the session panel does not answer for goes through fetch_price rather than being assumed absent, so a panel that somehow missed it costs a slower read instead of a wrong one.

Returns:

Name	Type	Description
`dict`	`dict[str, float \| None]`	identifier -> price, or None where there is none.

price_array_on ¶

price_array_on(
    identifiers: list[str], date: str, column: str = "CLOSE"
) -> np.ndarray | None

:meth:prices_on as one float array in the order of identifiers, NaN where it gives None; None when the column cannot be read this way, and the caller reads :meth:prices_on instead.

For a caller that asks about the same list of names day after day: the names are looked up once per list, so pass the same list object.

last_market_values ¶

last_market_values(
    identifiers: list[str],
    start: str | Timestamp,
    end: str | Timestamp,
    column: str,
) -> np.ndarray | None

Each name's last column value from start to end, as a :meth:fetch_market_data read of the window would give it, in the order of identifiers, NaN where a name has none; None when the column cannot be read this way, and the caller reads the window instead.

fetch_price ¶

fetch_price(
    identifier: str, date: str, column: str = "CLOSE"
) -> float | None

Return identifier's price on date, or None if it did not print.

The scalar form of the single-day, single-name fetch a methodology makes for every name in a universe. It reads the same value fetch_market_data(identifier, date, date) does (the same column of the same row) and returns it rather than a one-row frame to slice, which is what lets a warmed session serve it. Returns None too when the column is absent from the market data.

last_prices ¶

last_prices(
    identifiers: list[str],
    as_of: str,
    lookback_days: int,
    currency: str | None = None,
) -> pd.Series

Each instrument's last price on or before as_of, looking back lookback_days calendar days, converted into currency; an empty Series when none is priced.

fetch_prices ¶

fetch_prices(
    identifiers: list[str],
    start_date: str | None = None,
    end_date: str | None = None,
    currency: str | None = None,
    column: str = "CLOSE",
) -> pd.DataFrame

Prices by date and instrument, converted into currency.

The read to use whenever prices of several instruments are compared or added up: each is converted from its own currency day by day under this fetcher's FX policy. With currency=None nothing is converted. See beacon.data.prices.

Raises:

Type	Description
`CalculationError`	If an instrument's currency has no rate into currency.

fetch_fx_rates ¶

fetch_fx_rates(
    from_currency: str,
    to_currency: str,
    start_date: str | None = None,
    end_date: str | None = None,
) -> pd.Series

Return the stored FX rate series converting from_currency into to_currency.

The pair is the spot pair whose reference row has from_currency as its BASE_CURRENCY and to_currency as its QUOTE_CURRENCY, and its rate is its CLOSE. Returns an empty Series if that exact pair is not stored: this does not invert or cross rates. :meth:fx_rate_on and :meth:fx_rates_on do.

fx_route ¶

fx_route(
    from_currency: str, to_currency: str
) -> str | None

How a rate for this pair is found: direct, inverse or a cross.

Returns:

Type	Description
`str \| None`	str \| None: "direct" for a stored pair, "inverse" for one over the
`str \| None`	stored reverse pair, "cross via USD" for a rate built from two
`str \| None`	legs, "same currency" when no conversion is needed, and None when
`str \| None`	no rate can be found.

fx_rate_on ¶

fx_rate_on(
    from_currency: str,
    to_currency: str,
    date: str | Timestamp,
) -> float | None

The rate converting from_currency into to_currency on date.

The single currency conversion in the library, so the number displayed and the number weighted by are the same quantity. The pair is found as stored, else as the inverse of the stored reverse pair, else as a cross through USD (see beacon.data.fx and :meth:fx_route).

The series is fetched once per ordered pair and cached, because a run asks this on every foreign name on every day and each fetch slices the whole market frame.

Parameters:

Name	Type	Description	Default
`from_currency`	`str`	The currency being converted out of.	required
`to_currency`	`str`	The currency being converted into.	required
`date`	`str \| Timestamp`	The date the rate is wanted on.	required

Returns:

Type	Description
`float \| None`	float \| None: The rate in force on date, or None when the pair
`float \| None`	is unknown or when its history begins after date. Under the
`float \| None`	default `CARRY_FORWARD` policy the last rate on or before date
`float \| None`	is used; under `EXACT_DAY` only a rate printed on date itself,
`float \| None`	else None. Callers treat None as "cannot convert" rather than as a
`float \| None`	rate of one. Nothing here invents parity on a caller's behalf: a
`float \| None`	rate of 1.0 is a claim about two currencies, and the only one this
`float \| None`	makes is that a currency converts into itself.
`float \| None`	Carried forward only, never backward: a rate dated after the
`float \| None`	day it would be applied to is look-ahead. As with
`float \| None`	`resolve_session`, there is no earlier observation to be in force,
`float \| None`	so there is no answer rather than a substitute for one.

latest_rows ¶

latest_rows(
    identifiers: list[str],
    as_of: Timestamp,
    columns: list[str] | None = None,
    recent_days: int = RECENT_DAYS,
) -> pd.DataFrame

One row per name: its most recent bar at or before as_of.

The batch "what is the last thing we know about these names" read, shared by the reference endpoint and the staleness gate.

Read in two stages: the last recent_days first, which answers almost every name, then only the names it missed again without a lower bound. The result is reduced to one row per name once, with a grouped tail, rather than sliced per name downstream.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	Names to look up.	required
`as_of`	`Timestamp`	The date to look back from, inclusive.	required
`columns`	`list[str] \| None`	Columns to read, or None for all of them.	`None`
`recent_days`	`int`	How far back the cheap first stage reaches.	`RECENT_DAYS`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: MultiIndexed as the market data is, holding at most
`DataFrame`	one row per identifier. Names with no bar at or before as_of are
`DataFrame`	absent rather than present-and-empty.

last_priced_on ¶

last_priced_on(
    identifiers: list[str], as_of: Timestamp
) -> dict[str, pd.Timestamp]

When each name last printed a bar at or before as_of.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	Names to look up.	required
`as_of`	`Timestamp`	The date to look back from.	required

Returns:

Name	Type	Description
`dict`	`dict[str, Timestamp]`	identifier -> the date of its last bar. A name with no bar
	`dict[str, Timestamp]`	at all is absent from the mapping, which is a different thing from
	`dict[str, Timestamp]`	one whose last bar is old.

stale_identifiers ¶

stale_identifiers(
    identifiers: list[str], as_of: Timestamp
) -> set[str]

Which names have not traded recently enough to be worth holding.

Empty when no threshold is set, which is the default: staleness is something an installation opts into with max_price_staleness_days, and until it does this costs one comparison and reads nothing.

A name with no price at all is not reported here. That is a different condition with a different remedy (the weighting already refuses it by name), and folding the two together would quietly excuse a missing instrument as a quiet one.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	Names to test.	required
`as_of`	`Timestamp`	The date staleness is measured from.	required

Returns:

Name	Type	Description
`set`	`set[str]`	Identifiers whose last bar is more than
	`set[str]`	`max_price_staleness_days` calendar days before as_of.

fx_rates_on ¶

fx_rates_on(
    from_currency: str, to_currency: str, days: Index
) -> pd.Series | None

:meth:fx_rate_on over many days at once, as a Series.

The vectorised face of the same rule, for a caller that needs a rate for every day of a run rather than one date, such as chained levels. Asking per day would be one search per day per currency where a single reindex answers the lot. The policy, the carry semantics, the routes and the meaning of "no rate" are the same as :meth:fx_rate_on; the two methods differ only in how many answers they return.

Parameters:

Name	Type	Description	Default
`from_currency`	`str`	The currency being converted out of.	required
`to_currency`	`str`	The currency being converted into.	required
`days`	`Index`	The dates wanted, ascending.	required

Returns:

Type	Description
`Series \| None`	pd.Series \| None: One rate per day, indexed by days, or None when
`Series \| None`	the pair is unknown entirely. Individual days the policy cannot
`Series \| None`	answer for are NaN: a day is missing, not the pair.

fetch_reference_data ¶

fetch_reference_data(
    identifier: str | list[str],
    date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Fetch reference data for one or more identifiers.

Parameters:

Name	Type	Description	Default
`identifier`	`str \| list[str]`	One identifier or a list of identifiers.	required
`date`	`str \| None`	Point-in-time date. Only rows valid at this date are returned.	`None`
`columns`	`list[str] \| None`	Subset of columns to return.	`None`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Indexed by `IDENTIFIER`. Empty DataFrame if no
`DataFrame`	reference data is loaded or identifier is not found.

fetch_classification ¶

fetch_classification(
    identifier: str,
    date: str | Timestamp | None = None,
    scheme: str = DEFAULT_SCHEME,
) -> str | None

One instrument's classification as it stood on a date.

Reference data already carries validity ranges, so a name that moved from Industrials to Technology has two rows and this returns whichever was in force. That matters for anything historical: attributing a 2021 return to a sector the company only joined in 2023 is a real way to get a breakdown wrong.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`date`	`str \| Timestamp \| None`	The as-of date. None takes the currently-active record (the one with no end date), falling back to the latest start date if every record has been closed off.	`None`
`scheme`	`str`	Which column to read, e.g. `"SECTOR"`, `"INDUSTRY"`, `"COUNTRY"`. Free-form, because which columns a client loads is its own business.	`DEFAULT_SCHEME`

Returns:

Type	Description
`str \| None`	str or None: The classification, or None when it is unknown: no
`str \| None`	reference data, no such instrument, no such column, or no record
`str \| None`	valid on that date.

fetch_classifications ¶

fetch_classifications(
    identifiers: list[str],
    date: str | Timestamp | None = None,
    scheme: str = DEFAULT_SCHEME,
) -> dict[str, str | None]

Classifications for several instruments at once.

Every identifier appears, with None where the classification is unknown, so a caller can see what is missing rather than finding it silently absent.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	The instruments.	required
`date`	`str \| Timestamp \| None`	As-of date, as for :meth:`fetch_classification`.	`None`
`scheme`	`str`	Which column to read.	`DEFAULT_SCHEME`

Returns:

Name	Type	Description
`dict`	`dict[str, str \| None]`	Identifier to classification.

group_by_classification ¶

group_by_classification(
    identifiers: list[str],
    date: str | Timestamp | None = None,
    scheme: str = DEFAULT_SCHEME,
) -> dict[str, list[str]]

Instruments grouped by classification, ready for GroupBounds.

Unclassified instruments are collected under UNCLASSIFIED rather than dropped. A name missing from every bucket is how a constraint set quietly stops covering part of the universe.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	The instruments.	required
`date`	`str \| Timestamp \| None`	As-of date.	`None`
`scheme`	`str`	Which column to read.	`DEFAULT_SCHEME`

Returns:

Name	Type	Description
`dict`	`dict[str, list[str]]`	Classification to the identifiers carrying it, each list in
	`dict[str, list[str]]`	the order the identifiers were given.

pair_reference ¶

pair_reference(
    identifier: str,
    base: str | None = None,
    quote: str | None = None,
    date_from: str = "1900-01-01",
) -> dict[str, object]

A reference row that makes identifier a spot currency pair.

base and quote default to the identifier's two halves, so pair_reference("GBPUSD") describes pounds quoted in dollars.

Raises:

Type	Description
`ValueError`	If the currencies are not given and the identifier is not six letters.

short_rate_reference ¶

short_rate_reference(
    identifier: str,
    currency: str,
    tenor: str | None = None,
    day_count: str = ACT_360,
    date_from: str = "1900-01-01",
) -> dict[str, object]

A reference row that makes identifier currency's short rate.

load_data ¶

load_data(env: Environment) -> DataFetcher

Read data files from the environment config and return a DataFetcher.

For each dataset, a DataFrame is checked first; if not provided, the file path is used instead. Raises ValueError if no market data is available from either source.

adjustment ¶

Back-adjusting a price series for corporate actions.

adjusted = adjust_closes(closes, actions.get("CMPA"))

A raw close series has a step in it wherever a split happened, and it understates what a holder made wherever a dividend was paid. ADJ_CLOSE removes both.

What `ADJ_CLOSE` means here¶

Splits and dividends, which is the vendor convention: Yahoo and Bloomberg both ship a column of this name that reinvests cash. It is what most people mean when they say "adjusted", and shipping a splits-only series under the same name would give a number that looks right and is not.

The consequence is worth stating: an adjusted series is no longer a price. It answers "what would a holder have made", so its level is not what anything traded at on that day. A chart of it is a total-return chart.

Adjusted backwards from the latest date¶

The most recent close equals the raw close, and only history moves. Every vendor does it this way and the reason is practical: the right-hand edge of a chart matches the quote a user can see elsewhere, so the series is checkable. Adjusting forwards from the start would make the last number unrecognisable, which is the one people look at.

It also means the series changes when a new action lands: every historical value shifts. That is inherent to the convention rather than a defect, and a client caching an adjusted series needs to know it is not immutable.

Missing prices, and why a dividend needs one¶

A cash adjustment is a fraction of the price the day before the ex-date: paying 0.75 out of a 220 stock is a different event from paying it out of a 5 stock. So a dividend whose preceding close is missing cannot be applied, and is skipped with a warning rather than guessed at, because a guessed factor silently misstates every earlier value in the series.

Structural actions (a merger, a spin-off) are not a scaling of the same instrument's price, so they are left unadjusted.

adjust_closes ¶

adjust_closes(
    closes: Series, actions: DataFrame
) -> pd.Series

Back-adjust a close series for splits and cash.

Parameters:

Name	Type	Description	Default
`closes`	`Series`	Date-indexed closes, ascending.	required
`actions`	`DataFrame`	This instrument's actions, with `EX_DATE`, `TYPE`, `VALUE`, as `CorporateActions.get` returns.	required

Returns:

Type	Description
`Series`	pd.Series: Adjusted closes on the same index. The last value equals
`Series`	the last raw close; earlier ones are scaled by every action after
`Series`	them.

base ¶

Base data containers for market and reference data.

MarketData ¶

MarketData(file_path: str, date_format: str = '%Y-%m-%d')

Time-series data container backed by a MultiIndex DataFrame.

The source file must contain at least IDENTIFIER and DATE columns. After loading the DataFrame is indexed on (IDENTIFIER, DATE) and sorted, enabling fast .loc slicing by identifier or list of identifiers.

data `property` ¶

data: DataFrame

Return a copy of the underlying DataFrame.

identifiers `property` ¶

identifiers: list[str]

Unique identifiers present in the dataset.

Cached on the frame's identity, so the scan of the whole MultiIndex happens once per frame. fetch_fx_rates consults this on every call to decide whether a pair exists.

identifier_set `property` ¶

identifier_set: frozenset[str]

:attr:identifiers as a set, cached the same way.

columns `property` ¶

columns: list[str]

Non-index column names.

Cached on the frame's identity, like identifiers, because it is consulted on every price read.

sessions `property` ¶

sessions: DatetimeIndex

The distinct dates the dataset carries, ascending.

Cached on the frame's identity, like identifiers. Materialising the DATE level is an O(rows) take over the whole frame, and date_range and last_session_on_or_before both read it.

date_range `property` ¶

date_range: tuple[Timestamp, Timestamp]

(earliest, latest) timestamps in the dataset.

from_dataframe `classmethod` ¶

from_dataframe(
    df: DataFrame, date_format: str = "%Y-%m-%d"
) -> MarketData

Create a MarketData instance from an existing DataFrame.

drop_non_positive_prices ¶

drop_non_positive_prices(
    exempt: Collection[str] = (),
) -> None

Read every price at or below zero as missing; see the function.

last_session_on_or_before ¶

last_session_on_or_before(
    date: str | Timestamp,
) -> pd.Timestamp | None

The latest date the dataset carries at or before date.

A date inside the coverage that has no rows is an ordinary closed market, and this is the session that was in force through it. None when the dataset begins after date, since then there is no earlier session to be in force.

session_columns ¶

session_columns(
    date: str | Timestamp,
) -> tuple[dict[str, dict[str, object]], set[str]]

Every column's values for one session, keyed by identifier.

The read the daily loops make once per session, done without touching pandas. The frame is indexed (IDENTIFIER, DATE), so a single day's rows are scattered through it rather than adjacent, and get() has to filter the whole frame to find them. This looks up the row positions for the date from a per-frame index instead, so the read is a gather and two dict builds. Where an (identifier, date) pair is repeated, the first row wins, as it does for get() followed by .iloc[0].

Returns:

Name	Type	Description
`tuple`	`dict[str, dict[str, object]]`	`({column: {identifier: value}}, identifiers)`, exactly
	`set[str]`	what a :class:`~beacon.data.session.SessionPanel` holds, so no
	`tuple[dict[str, dict[str, object]], set[str]]`	DataFrame is built on the way. Empty for a date the data has no
	`tuple[dict[str, dict[str, object]], set[str]]`	rows on.

session_rows ¶

session_rows(
    date: str | Timestamp,
) -> (
    tuple[np.ndarray, np.ndarray, dict[str, np.ndarray]]
    | None
)

The frame's identifiers, one session's row positions, and the column arrays, with nothing gathered or built; None for a date with no rows. What a lazy session panel holds.

codes_for ¶

codes_for(identifiers: list[str]) -> np.ndarray

Each identifier's code among this frame's rows, -1 for a name with no rows.

The last answer is kept for the same list object, so a caller asking about the same names day after day looks them up once.

session_values ¶

session_values(
    date: Timestamp, codes: ndarray, column: str
) -> np.ndarray | None

column on date for the names with codes, NaN where a name has no row; None when the column is missing or not a float column.

The values the session panel gives, as one array: where a name has several rows on the day the first is taken, as the panel's reversed fill takes it.

last_values ¶

last_values(
    codes: ndarray,
    start: Timestamp,
    end: Timestamp,
    column: str,
) -> np.ndarray | None

column's last value from start to end for the names with codes, NaN where a name has none in the window; None when the column is missing or not a float column.

What a read of the window gives as each name's last value, its last row with one, found session by session from the end instead of by slicing every name's history.

get ¶

get(
    identifier: str | list[str],
    start_date: str | None = None,
    end_date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Return data for one or more identifiers, optionally filtered.

Parameters:

Name	Type	Description	Default
`identifier`	`str \| list[str]`	Single identifier or list of identifiers.	required
`start_date`	`str \| None`	Date string to slice the start of the date range.	`None`
`end_date`	`str \| None`	Date string to slice the end of the date range.	`None`
`columns`	`list[str] \| None`	Subset of columns to return.	`None`

Returns:

Name	Type	Description
	`DataFrame`	pd.DataFrame: Single identifier: indexed by `DATE`. List of
`identifiers`	`DataFrame`	MultiIndexed by `(IDENTIFIER, DATE)`. Empty
	`DataFrame`	DataFrame if no matching data is found.

ReferenceData ¶

ReferenceData(file_path: str)

Reference data container with validity ranges.

The source file must contain IDENTIFIER and DATE_FROM columns. DATE_TO is optional: a missing column, or NaT in it, marks a currently-active record.

Indexed on IDENTIFIER (non-unique, since an identifier may have multiple validity periods).

data `property` ¶

data: DataFrame

Return a copy of the underlying DataFrame.

identifiers `property` ¶

identifiers: list[str]

Unique identifiers present in the dataset, cached on the frame's identity.

columns `property` ¶

columns: list[str]

Column names (including DATE_FROM, DATE_TO).

from_dataframe `classmethod` ¶

from_dataframe(df: DataFrame) -> ReferenceData

Create a ReferenceData instance from an existing DataFrame.

get ¶

get(
    identifier: str | list[str],
    date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Return reference data for one or more identifiers.

Parameters:

Name	Type	Description	Default
`identifier`	`str \| list[str]`	Single identifier or list of identifiers.	required
`date`	`str \| None`	Point-in-time date. If provided, only rows where `DATE_FROM <= date` and (`DATE_TO >= date` or `DATE_TO` is NaT) are returned.	`None`
`columns`	`list[str] \| None`	Subset of columns to return.	`None`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Indexed by `IDENTIFIER`. Empty DataFrame if no
`DataFrame`	match.

as_of_position ¶

as_of_position(
    index: Index, date: str | Timestamp
) -> int | None

Where the last observation at or before date sits, or None.

The one way a point-in-time lookup is done, so that look-ahead is not something a caller has to remember to avoid.

The natural expression, index.searchsorted(date, side="right") - 1, returns -1 when every observation is dated after date, and in pandas -1 is a legal index meaning the last element: the newest observation, the largest look-ahead available. Clamping to 0 instead still returns a future value. This returns None, which a caller cannot accidentally index with.

Parameters:

Name	Type	Description	Default
`index`	`Index`	Ascending dates. Sorting is the caller's business; a search over an unsorted index is meaningless under any convention.	required
`date`	`str \| Timestamp`	The point in time being asked about.	required

Returns:

Type	Description
`int \| None`	int \| None: Position of the last entry at or before date, or None
`int \| None`	when the index begins after it.

count_through ¶

count_through(values: ndarray, key: Any) -> int

How many of ascending values are at or before key.

The end of a slice that stops at key, for a window read from arrays, where :func:as_of_position would build an index on every call. A count is not a position: 0 means nothing is at or before key, and the caller reads values[:count], never values[count - 1] unguarded.

drop_non_positive_prices ¶

drop_non_positive_prices(
    frame: DataFrame, exempt: Collection[str] = ()
) -> None

Read a price at or below zero as no price at all, in place.

exempt names the series whose CLOSE is not a price, such as a short rate, which may well be below zero.

A price cannot be zero or negative, and one that is would fail a backtest the moment it is traded and turn an index's market values negative. Treated as a missing bar instead, every reader handles it as one: the last good close is carried and the day recorded as a price gap. One warning per load names the identifiers and each one's first such date.

corporate_actions ¶

Corporate-action history.

A store of every action per instrument, so a caller can ask what a constituent paid over the last year and the server can answer /data/corporate-actions/{ticker}. It sits beside MarketData and ReferenceData and is loaded the same way.

One row is one action: an identifier, an ex-date, a type and a value. What the value means depends on the type, and conflating the two is the mistake this module is arranged to prevent:

Cash actions (DIVIDEND, SPECIAL_DIVIDEND, RETURN_OF_CAPITAL) carry an amount per share. They add up. Two dividends of 0.25 make 0.50.
Ratio actions (SPLIT, REVERSE_SPLIT, STOCK_DIVIDEND) carry a multiplier on the share count. They compound. Two 2-for-1 splits make a factor of 4, not 4-for-1 in any additive sense.
Structural actions (RIGHTS_ISSUE, SPIN_OFF, MERGER) carry no directly aggregable value.

So there is no single "total actions" helper. Summing a split ratio into a dividend total would produce a number with no meaning, and the only way to stop that happening is to not offer the operation.

What trailing twelve months means here¶

Twelve calendar months back from the as-of date, not 365 days. The two differ across a leap day, and "twelve months" is what a yield is quoted on. The window is half-open (as_of - 1 year < ex_date <= as_of), so an action exactly one year old has rolled out and one dated today is in. Without that, a dividend paid on the anniversary would be counted in two consecutive years' figures.

CorporateActions ¶

CorporateActions(frame: DataFrame)

A history of corporate actions, indexed by identifier and ex-date.

The source must carry IDENTIFIER, EX_DATE, TYPE and VALUE, and every TYPE must be one of ACTION_TYPES (case-insensitive). Anything else, such as PAY_DATE, CURRENCY or DECLARED_DATE, is carried through; PAY_DATE is parsed as a date.

Raises:

Type	Description
`CalculationError`	If a required column is missing or a type is unknown.

data `property` ¶

data: DataFrame

A copy of the whole history.

identifiers `property` ¶

identifiers: list[str]

Identifiers with at least one action.

is_empty `property` ¶

is_empty: bool

Whether the store holds nothing.

from_dataframe `classmethod` ¶

from_dataframe(frame: DataFrame) -> CorporateActions

Build from an existing DataFrame, matching the other containers.

empty `classmethod` ¶

empty() -> CorporateActions

A store with no actions in it.

Worth having: most tests and most universes have no action history, and the alternative is every caller checking for None before asking.

get ¶

get(
    identifier: str,
    start_date: str | Timestamp | None = None,
    end_date: str | Timestamp | None = None,
    types: list[str] | None = None,
) -> pd.DataFrame

Actions for one identifier, oldest first.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`start_date`	`str \| Timestamp \| None`	Earliest ex-date to include, inclusive.	`None`
`end_date`	`str \| Timestamp \| None`	Latest ex-date to include, inclusive.	`None`
`types`	`list[str] \| None`	Restrict to these action types. None returns all.	`None`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Matching rows, or an empty frame with the right
`DataFrame`	columns when there are none, so a caller can read `VALUE` off
`DataFrame`	the result without checking first.

trailing_cash ¶

trailing_cash(
    identifier: str,
    as_of: str | Timestamp,
    types: list[str] | None = None,
) -> float

Cash per share over the twelve months ending as_of.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`as_of`	`str \| Timestamp`	End of the window, inclusive.	required
`types`	`list[str] \| None`	Which cash actions to count. None counts ordinary dividends only, the conventional basis for a trailing yield, since a special dividend is by definition not expected to repeat.	`None`

Returns:

Name	Type	Description
`float`	`float`	The total. 0.0 when there is nothing in the window.

Raises:

Type	Description
`CalculationError`	If a non-cash type is requested. A split ratio added to a dividend total would produce a number with no meaning.

trailing_dividend ¶

trailing_dividend(
    identifier: str, as_of: str | Timestamp
) -> float

Ordinary dividends per share over the trailing twelve months.

trailing_dividend_yield ¶

trailing_dividend_yield(
    identifier: str, as_of: str | Timestamp, price: float
) -> float

Trailing dividends as a fraction of price.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`as_of`	`str \| Timestamp`	End of the trailing window.	required
`price`	`float`	Price to divide by, normally the close on as_of.	required

Returns:

Name	Type	Description
`float`	`float`	The yield.

Raises:

Type	Description
`CalculationError`	If price is not positive. A yield on a zero price is not a large number, it is an undefined one.

cumulative_ratio ¶

cumulative_ratio(
    identifier: str,
    start_date: str | Timestamp | None = None,
    end_date: str | Timestamp | None = None,
) -> float

Compounded share-count multiplier over a window.

Two 2-for-1 splits give 4.0, not 4 in any additive sense, which is why this is separate from the cash helpers rather than a mode of them.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`start_date`	`str \| Timestamp \| None`	Earliest ex-date, inclusive.	`None`
`end_date`	`str \| Timestamp \| None`	Latest ex-date, inclusive.	`None`

Returns:

Name	Type	Description
`float`	`float`	The product of every ratio in the window. 1.0 when there are
	`float`	none, which is the identity and leaves a share count unchanged.

Raises:

Type	Description
`CalculationError`	If a ratio is not positive. A zero or negative multiplier would erase or invert a position.

ratio_schedule ¶

ratio_schedule() -> dict[pd.Timestamp, dict[str, float]]

Every share-count change, by ex-date and name.

What holds a fixed number of shares (an index's units, a backtest's position) multiplies them by the ratio on the ex-date, when the stored close falls by the same ratio. A cancelled action is left out, and two on the same name and day compound.

Returns:

Name	Type	Description
`dict`	`dict[Timestamp, dict[str, float]]`	ex-date -> {identifier: share-count multiplier}. Empty when
	`dict[Timestamp, dict[str, float]]`	the history holds no splits, reverse splits or stock dividends.

Raises:

Type	Description
`CalculationError`	If a ratio is not positive. A zero or negative multiplier would erase or invert a position.

as_records ¶

as_records(
    identifier: str, date: str | Timestamp
) -> list[dict[str, object]]

Actions on one date, in the shape IndexCalculator expects.

The calculator takes {"type", "asset", "value", "ex_date"}. Building that here keeps the mapping in one place rather than at every call site.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`date`	`str \| Timestamp`	The ex-date.	required

Returns:

Name	Type	Description
`list`	`list[dict[str, object]]`	One dict per action, empty when there are none.

kind_of ¶

kind_of(action_type: str) -> Kind

What the value of an action of this type means.

Parameters:

Name	Type	Description	Default
`action_type`	`str`	One of ACTION_TYPES.	required

Returns:

Type	Description
`Kind`	The kind. An unrecognised type is STRUCTURAL, with a warning: the
`Kind`	answer that causes a client to display no quantity, which is the safe
`Kind`	direction to be wrong in. Rendering an unknown action as cash would put
`Kind`	a number on screen that means something else entirely.

without_cancelled ¶

without_cancelled(actions: DataFrame) -> pd.DataFrame

actions less any whose status is cancelled.

A cancelled action never happened, so nothing that applies actions (a split to units held, a dividend to a total return, either to an adjusted close) or sums them (a trailing dividend) may count it. Rows with no status, or a frame with no status column, are kept.

status_of ¶

status_of(value: object) -> Status | None

Normalise a stored status, or None if it is not one this vocabulary has.

Validated rather than passed through: the status arrives from whatever wrote the store, and forwarding an unrecognised string would make a client branch on a value no schema documents. None says "unknown", which is what an unreadable status actually means.

ratios_between ¶

ratios_between(
    schedule: dict[Timestamp, dict[str, float]],
    after: Timestamp,
    through: Timestamp,
) -> dict[str, float]

The share-count multipliers with an ex-date in (after, through].

For a loop that steps from one session to the next: an action whose ex-date fell on a day the loop did not visit still applies on the next day it does. Several on one name compound.

Parameters:

Name	Type	Description	Default
`schedule`	`dict[Timestamp, dict[str, float]]`	Output of `CorporateActions.ratio_schedule`.	required
`after`	`Timestamp`	The last day already processed (excluded).	required
`through`	`Timestamp`	Today (included).	required

Returns:

Name	Type	Description
`dict`	`dict[str, float]`	identifier -> multiplier. Empty on a day nothing changed.

features ¶

Features: everything about an instrument that is not price, reference or action.

Fundamentals, alternative datasets, macroeconomic series, and values somebody derived and imported all share one shape, so they share one table, with one set of point-in-time rules, rather than a surface per kind each with its own schema.

The shape¶

Field-value pairs, keyed by instrument and date:

IDENTIFIER   ticker, ISIN, or whatever the loaded data keys on
DATE         the date the value became knowable (see below)
TYPE         which dataset it came from
FIELD        the datapoint: revenue, total_debt, pe_ratio, card_spend
VALUE        the number
DETAIL       free-form context, no defaults and no fixed vocabulary

TYPE is what lets several datasets share the table without colliding. revenue from a vendor and revenue from a user's own model are different series, and a query has to be able to say which it means.

DATE is the announcement date, not the period end¶

This is the decision the whole table rests on.

A fundamental has at least two dates: the period it describes (Q1 2024) and the date it was published (2024-05-15). Only the second decides visibility. A backtest standing on 2024-04-01 must not see Q1 revenue that nobody knew until May. That is look-ahead bias, the most common way a fundamental-driven strategy shows returns it could never have earned, and the dangerous part is that the obvious point-in-time query looks correct while producing it.

So DATE holds when the value became knowable, and the period it describes goes in DETAIL.

The column is called DATE, matching what every other dataset here names its date, rather than something like AS_OF that encodes the rule. The point-in-time behaviour is a property of how the table is read: it belongs to the accessor, not to the column name. Storing the period end in DATE would make every downstream query wrong in the same direction at once.

DETAIL is deliberately unconstrained. A restatement, a fiscal-period label, a vendor revision number, a units note: all real, all things somebody will need to record, and a fixed schema invented here would be invented wrong.

Restatements are kept, not overwritten¶

A vendor revising Q1 revenue in August does not erase what it said in May, and a backtest standing in June must still see the May figure. So the same (IDENTIFIER, TYPE, FIELD) published on a different DATE is kept as a new row rather than deduplicated, and the accessor resolves which one is in force. Overwriting would make the table smaller and the history unrecoverable.

Two rows with the same (IDENTIFIER, DATE, TYPE, FIELD) are a genuine duplicate (the same claim loaded twice), and the last one wins, because that is what re-importing a corrected file should do.

FeatureData ¶

FeatureData(file_path: str)

Per-instrument datapoints, keyed by identifier and as-of date.

Indexed on (IDENTIFIER, DATE), non-unique: an instrument carries many fields on one date, and one field across many dates.

data `property` ¶

data: DataFrame

A copy of the underlying frame.

is_empty `property` ¶

is_empty: bool

Whether anything is loaded.

identifiers `property` ¶

identifiers: list[str]

Instruments the table carries anything for.

columns `property` ¶

columns: list[str]

Column names, excluding the index keys.

types `property` ¶

types: list[str]

The datasets loaded, so a client can populate a control.

from_dataframe `classmethod` ¶

from_dataframe(df: DataFrame) -> FeatureData

Create a FeatureData instance from an existing DataFrame.

empty `classmethod` ¶

empty() -> FeatureData

A container holding nothing.

So a dataset without features is still a dataset, and callers can ask it questions without checking for None first.

fields ¶

fields(feature_type: str | None = None) -> list[str]

The fields loaded, optionally within one dataset.

Parameters:

Name	Type	Description	Default
`feature_type`	`str \| None`	Restrict to one `TYPE`. None returns every field across every dataset, which will collide names where two datasets share one.	`None`

value_as_of ¶

value_as_of(
    identifier: str,
    field: str,
    date: Timestamp | str | None = None,
    feature_type: str | None = None,
    max_age_days: int | None = MAX_AGE_DAYS,
) -> float | None

The value in force on a date, or None.

The most recent row whose own DATE is on or before date, not the row for that date. Fundamentals are quarterly and a backtest runs daily, so "the latest thing knowable on this date" is the only question worth asking.

It never looks forward. A value published on 2024-05-15 is invisible to a query standing on 2024-04-01, however recently the period it describes ended. That is the whole reason DATE holds the announcement date, and this is where it pays off or fails to.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`field`	`str`	The datapoint.	required
`date`	`Timestamp \| str \| None`	Stand here. None uses the latest date in the table, which is the right default for "what do we know now" and the wrong one for a backtest, which always passes its own.	`None`
`feature_type`	`str \| None`	Restrict to one dataset. None searches every type, which will pick arbitrarily between two carrying the same field name, so a caller that has both should say which.	`None`
`max_age_days`	`int \| None`	How stale is too stale. None disables the check.	`MAX_AGE_DAYS`

Returns:

Type	Description
`float \| None`	float \| None: The value, or None when nothing is knowable: no
`float \| None`	coverage, nothing published yet, nothing recent enough, or a
`float \| None`	value that is not a number.

table_as_of ¶

table_as_of(
    identifiers: list[str],
    field: str,
    dates: list[Timestamp],
    feature_type: str | None = None,
    max_age_days: int | None = MAX_AGE_DAYS,
) -> pd.DataFrame

value_as_of for many instruments on many dates, as one table.

The same answer, read from the whole table at once rather than name by name: on each date, each instrument's latest row on or before it, missing when there is none, it is older than max_age_days, or its value is not a number.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	The instruments, the table's columns.	required
`field`	`str`	The datapoint.	required
`dates`	`list[Timestamp]`	The dates to stand on, the table's rows.	required
`feature_type`	`str \| None`	Restrict to one dataset, as for `value_as_of`.	`None`
`max_age_days`	`int \| None`	How stale is too stale. None disables the check.	`MAX_AGE_DAYS`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Dates by instrument.

history ¶

history(
    identifier: str,
    field: str,
    date: Timestamp | str | None = None,
    feature_type: str | None = None,
) -> pd.DataFrame

Every row for one instrument and field, up to a date.

Exposed because a restatement is kept rather than overwritten (see the module docstring), so "what did we believe, and when" is a question this table can answer and a caller may need to.

merged_with ¶

merged_with(frame: DataFrame) -> FeatureData

A new table with frame added.

Merge rather than replace, because a feature table is append-only by nature: a restatement is a new row (see the module docstring), and a second upload of a different TYPE should not discard the first. A row matching an existing identifier, date, type and field replaces it, which is what re-uploading a corrected file means.

Returns a new instance rather than mutating: the loaded data is shared by every request in flight, and editing it underneath them would make a read depend on when it happened to run.

rows_for ¶

rows_for(
    identifier: str,
    date: Timestamp | str | None = None,
    feature_type: str | None = None,
    fields: list[str] | None = None,
) -> list[dict[str, Any]]

Every field in force for one instrument, resolved point-in-time.

One row per field, carrying the value, its dataset, its detail and the date it became knowable, so a client can show when a number is from, which for a fundamental is most of what makes it interpretable. A field with nothing knowable yet still gets a row, with value, detail and date None. Unlike value_as_of, no staleness bound is applied.

type_coverage ¶

type_coverage() -> list[dict[str, Any]]

Per-dataset coverage, for the catalogue.

coverage ¶

coverage() -> dict[str, Any]

How much of each dataset is present, for /data/coverage.

fetcher ¶

DataFetcher: one query interface over market, reference, corporate-action and feature data.

Accepts single identifiers or lists and passes through column names as-is.

DataFetcher ¶

DataFetcher(
    market_data: MarketData,
    reference_data: ReferenceData | None = None,
    corporate_actions: CorporateActions | None = None,
    features: FeatureData | None = None,
    fx_policy: str = DEFAULT_FX_POLICY,
    max_price_staleness_days: int | None = None,
    free_float_backfill_days: int = DEFAULT_FREE_FLOAT_BACKFILL_DAYS,
)

One query interface over market, reference, corporate-action and feature data.

Parameters:

Name	Type	Description	Default
`market_data`	`MarketData`	Time-series market data container. Currency pairs and short rates live in it as series of their own, with their values in `CLOSE`.	required
`reference_data`	`ReferenceData \| None`	Reference data container, or None if there is none. It says which series are currency pairs and short rates, and which can be held (see `beacon.data.kinds`).	`None`
`corporate_actions`	`CorporateActions \| None`	Action history. Absent means an empty history rather than None, so callers never have to check before asking: "this instrument paid nothing" and "we hold no action data" give the same answer to every question this class can be asked.	`None`
`features`	`FeatureData \| None`	Feature table. Absent means an empty table, on the same terms.	`None`
`fx_policy`	`str`	How an FX rate is read on a day the pair did not print one. `"CARRY_FORWARD"` (the default) uses the last rate published; `"EXACT_DAY"` answers only when the pair printed a rate on that very day. It applies to every conversion made through this fetcher.	`DEFAULT_FX_POLICY`
`max_price_staleness_days`	`int \| None`	How many calendar days a name may go without trading before `stale_identifiers` reports it as not worth holding. None (the default) keeps every name, whenever it last traded.	`None`
`free_float_backfill_days`	`int`	How many calendar days a reported free float carries forward over blank cells (default 90; 0 turns carrying off). See `beacon.data.free_float`.	`DEFAULT_FREE_FLOAT_BACKFILL_DAYS`

Raises:

Type	Description
`ValueError`	If `fx_policy` is not a known policy, `max_price_staleness_days` is below 1, `free_float_backfill_days` is negative or not an integer, the market data has a `RATE` column, the reference data describes a series wrongly, or a short rate looks like a percent.

identifiers `property` ¶

identifiers: list[str]

Unique identifiers present in market data.

fx_pairs `property` ¶

fx_pairs: list[str]

Spot currency pairs with market data: the identifiers reference data gives ASSET_CLASS Currency and SECURITY_TYPE Spot.

instrument_identifiers `property` ¶

instrument_identifiers: list[str]

Market identifiers a fund can hold: every series but the ones reference data marks as not tradable, such as currency pairs and short rates.

tradable_identifiers `property` ¶

tradable_identifiers: list[str]

What a universe can be drawn from: the reference data's identifiers (else the market data's), less those not tradable.

market_columns `property` ¶

market_columns: list[str]

Column names in the market data.

reference_identifiers `property` ¶

reference_identifiers: list[str] | None

Unique identifiers in the reference data, or None if not loaded.

reference_columns `property` ¶

reference_columns: list[str] | None

Column names in the reference data, or None if not loaded.

date_range `property` ¶

date_range: tuple[Timestamp, Timestamp]

(earliest, latest) timestamps in the market data.

corporate_actions `property` ¶

corporate_actions: CorporateActions

The action history. Empty rather than None when none was loaded.

features `property` ¶

features: FeatureData

The feature table. Empty rather than None when none was loaded.

Exposed for persistence and for discovery, on the same terms as market. Point-in-time reads go through fetch_feature and fetch_features, not through this.

market `property` ¶

market: MarketData

The market-data container itself.

Exposed for persistence (beacon.data.store): writing a fetcher to disk means reading back everything it holds, and the summarising properties cannot reconstruct a frame. Query through fetch_market_data instead: this is the whole dataset, not an answer to a question.

source `property` ¶

source: str | None

Where this data was loaded from, or None if nothing recorded it.

Describes the load, not every row: a later sync merges rows from somewhere else without changing where the store came from. Modelling mixed provenance would need a source per row, which nothing asks for.

store_path `property` ¶

store_path: Path | None

The store this was loaded from, if it came from one.

reference `property` ¶

reference: ReferenceData | None

The reference-data container, or None if none was loaded.

Exposed for persistence, on the same terms as :attr:market.

with_settings ¶

with_settings(
    fx_policy: str = KEEP,
    max_price_staleness_days: int | None = KEEP,
    free_float_backfill_days: int = KEEP,
) -> DataFetcher

This data read under other settings.

The same object when nothing differs. Otherwise a new fetcher over the same data, with its own caches, since a cached FX series or free-float history depends on the settings it was read under. Pass KEEP (the default) for a setting to leave as it is.

Raises:

Type	Description
`ValueError`	As the constructor does, for a setting it refuses.

is_tradable ¶

is_tradable(identifier: str) -> bool

Whether a fund can hold identifier: false for a series reference data marks as not tradable.

short_rate ¶

short_rate(currency: str) -> ShortRate | None

A currency's short rate, as reference data describes it, or None.

Where a currency has several, the one month rate is read, else the overnight, else the first by identifier.

fetch_short_rates ¶

fetch_short_rates(
    currency: str,
    start_date: str | None = None,
    end_date: str | None = None,
) -> pd.Series

A currency's short rate series, annual decimals, from its CLOSE. Empty when the data holds none.

short_rate_on ¶

short_rate_on(
    currency: str, date: str | Timestamp
) -> float | None

A currency's short rate in force on date: the last on or before it. None when the data holds none that early.

resolve_session ¶

resolve_session(
    date: str | Timestamp,
) -> pd.Timestamp | None

The market session date resolves to, backfilling inside the data.

A date the data has no bar for but which sits inside its coverage is a day the market was shut. The last session on or before it is the one that was actually in force through the closure: reading it is not an approximation, it is what the day was. Past the last bar nothing is known, so that answers None rather than a stale print wearing a current date; the bound is the data's own coverage rather than a day count, because no day count can tell a long closure from the unknown future.

Parameters:

Name	Type	Description	Default
`date`	`str \| Timestamp`	The date asked about.	required

Returns:

Type	Description
`Timestamp \| None`	pd.Timestamp \| None: The session, or None when date falls
`Timestamp \| None`	outside the data's coverage on either side.

fetch_feature ¶

fetch_feature(
    identifier: str,
    field: str,
    date: str | Timestamp | None = None,
    feature_type: str | None = None,
    max_age_days: int | None = MAX_AGE_DAYS,
) -> float | None

One feature value, as it was knowable on a date.

The point-in-time read. A value published after date is invisible, which is what keeps a backtest from screening on numbers nobody had.

Returns:

Type	Description
`float \| None`	float \| None: The value, or None when nothing is knowable. An
`float \| None`	instrument with no coverage is an ordinary answer, not an error:
`float \| None`	most datasets cover most names most of the time and not all of
`float \| None`	them all of it.

fetch_features ¶

fetch_features(
    identifiers: list[str],
    fields: list[str],
    date: str | Timestamp | None = None,
    feature_type: str | None = None,
    max_age_days: int | None = MAX_AGE_DAYS,
) -> dict[str, dict[str, float | None]]

Several features for several instruments, on one date.

The batch form, on the same argument the reference batch endpoint made: a client that has to fan out per name will, and moving the fan-out inside the server only relocates the cost.

Returns:

Name	Type	Description
`dict`	`dict[str, dict[str, float \| None]]`	identifier -> field -> value. Every requested pair is
	`dict[str, dict[str, float \| None]]`	present, null where nothing is knowable, so a caller reads a value
	`dict[str, dict[str, float \| None]]`	rather than testing for a key.

replace_features ¶

replace_features(features: FeatureData) -> None

Swap the feature table, stamping the refresh.

The only mutating method on a fetcher, and it exists because an import has to land somewhere the next request can see. It replaces rather than edits: merged_with builds the new table, so a read in flight keeps the frame it started with instead of watching rows appear under it.

feature_types ¶

feature_types() -> list[str]

Datasets the loaded features carry, for discovery.

feature_fields ¶

feature_fields(
    feature_type: str | None = None,
) -> list[str]

Fields the loaded features carry, optionally within one dataset.

record_origin ¶

record_origin(
    source: str, path: Path | None = None
) -> None

Note where this fetcher's data was loaded from.

delisting_dates ¶

delisting_dates() -> dict[str, pd.Timestamp]

The last date each identifier is listed, for those whose life ends.

Resolved in one pass rather than per identifier per day: an index over five thousand names and ten years would otherwise make twelve million point-in-time lookups to find a few hundred delistings.

A name is treated as still listed if any of its records is open-ended, even when another record for it has an end date.

Returns:

Name	Type	Description
`dict`	`dict[str, Timestamp]`	identifier -> last listed date. Names that never leave are
	`dict[str, Timestamp]`	absent, so an empty mapping means a constant universe and callers
	`dict[str, Timestamp]`	can skip the work entirely.

record_refresh ¶

record_refresh(
    dataset: str, when: datetime | None = None
) -> None

Note that a dataset has just been refreshed.

Parameters:

Name	Type	Description	Default
`dataset`	`str`	One of `DATASETS`: `"market"`, `"reference"`, `"corporate_actions"`, `"features"` or `"fx"`.	required
`when`	`datetime \| None`	The moment. None uses now, which is what a real sync wants; tests pass an explicit time so an age can be asserted rather than approximated.	`None`

Raises:

Type	Description
`ValueError`	If the dataset is not one this fetcher holds.

last_refreshed ¶

last_refreshed(dataset: str) -> datetime | None

When a dataset was last loaded or synced.

Returns:

Type	Description
`datetime \| None`	datetime or None: The moment, or None when the dataset is not
`datetime \| None`	loaded at all, which is a different statement from "loaded and
`datetime \| None`	never refreshed" and should not be collapsed into it. Loading
`datetime \| None`	counts as a refresh.

Raises:

Type	Description
`ValueError`	If the dataset is not one of `DATASETS`.

age_seconds ¶

age_seconds(
    dataset: str, now: datetime | None = None
) -> float | None

How long ago a dataset was last refreshed, in seconds.

Parameters:

Name	Type	Description	Default
`dataset`	`str`	Which dataset.	required
`now`	`datetime \| None`	The reference moment, for tests.	`None`

Returns:

Type	Description
`float \| None`	float or None: The age, or None when the dataset is not loaded.
`float \| None`	Never negative: a clock adjustment between the two readings would
`float \| None`	otherwise report data refreshed in the future, which is noise
`float \| None`	rather than information.

merge_market_data ¶

merge_market_data(frame: DataFrame) -> int

Fold freshly ingested rows into the market data.

Newly fetched rows win where they overlap an existing identifier and date. A re-sync of a window is a correction (a restated close, a backfilled volume), so keeping the older value would make the sync pointless.

The swap at the end is a single assignment, so a reader either sees the whole old dataset or the whole new one. This process is single-threaded and cooperatively scheduled, so there is no torn state to guard against; a reader that started before the swap simply finishes against the data it began with.

Parameters:

Name	Type	Description	Default
`frame`	`DataFrame`	Long-form rows carrying `IDENTIFIER` and `DATE`.	required

Returns:

Name	Type	Description
`int`	`int`	Rows added, counting only genuinely new identifier/date
	`int`	pairs. A re-sync that restates existing rows returns 0, which is
	`int`	the truthful answer to "how much did this add".

merge_reference_data ¶

merge_reference_data(frame: DataFrame) -> int

Fold freshly ingested reference records in.

A record matching an existing identifier and DATE_FROM replaces it.

Parameters:

Name	Type	Description	Default
`frame`	`DataFrame`	Rows carrying `IDENTIFIER` and `DATE_FROM`.	required

Returns:

Name	Type	Description
`int`	`int`	Records added, counting only genuinely new identifier and
	`int`	`DATE_FROM` pairs.

fetch_corporate_actions ¶

fetch_corporate_actions(
    identifier: str,
    start_date: str | Timestamp | None = None,
    end_date: str | Timestamp | None = None,
    types: list[str] | None = None,
) -> pd.DataFrame

Corporate actions for one identifier over a window.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`start_date`	`str \| Timestamp \| None`	Earliest ex-date, inclusive.	`None`
`end_date`	`str \| Timestamp \| None`	Latest ex-date, inclusive.	`None`
`types`	`list[str] \| None`	Restrict to these action types.	`None`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Matching actions, oldest first; empty when there are
`DataFrame`	none.

fetch_trailing_dividend ¶

fetch_trailing_dividend(
    identifier: str, as_of: str | Timestamp
) -> float

Ordinary dividends per share over the trailing twelve months.

fetch_trailing_dividend_yield ¶

fetch_trailing_dividend_yield(
    identifier: str,
    as_of: str | Timestamp,
    price: float | None = None,
) -> float | None

Trailing dividend yield, priced off the market data by default.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`as_of`	`str \| Timestamp`	End of the trailing window.	required
`price`	`float \| None`	Price to divide by. None reads the close on or before as_of from the market data.	`None`

Returns:

Name	Type	Description
	`float \| None`	float or None: The yield, or None when no positive price is
`available`	`float \| None`	a missing price is a reason to say nothing rather than
	`float \| None`	to guess.

fetch_market_data ¶

fetch_market_data(
    identifier: str | list[str],
    start_date: str | None = None,
    end_date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Fetch time-series market data for one or more identifiers.

Parameters:

Name	Type	Description	Default
`identifier`	`str \| list[str]`	One identifier or a list of identifiers.	required
`start_date`	`str \| None`	Date string to filter the start of the date range.	`None`
`end_date`	`str \| None`	Date string to filter the end of the date range.	`None`
`columns`	`list[str] \| None`	Subset of columns to return.	`None`

Returns:

Name	Type	Description
	`DataFrame`	pd.DataFrame: Single identifier: indexed by `DATE`. Multiple
`identifiers`	`DataFrame`	MultiIndexed by `(IDENTIFIER, DATE)`. Empty
	`DataFrame`	DataFrame if no matching data is found.

warm_session ¶

warm_session(
    identifiers: list[str], date: str | Timestamp
) -> None

Read one session's rows for identifiers in a single slice.

A hint, not a contract: every read this serves answers identically without it, only slower. Without it, a methodology walking a universe makes one frame slice per name per column, each costing what the whole frame costs rather than what one row does, so a preview's cost per name climbs with the size of its universe.

A selection rule prices every candidate and the weighting scheme then prices the survivors, the same names on the same day. The second warm is a subset of the first, so it keeps the panel rather than rebuilding it, and the reads that follow are free.

Nothing goes stale under it: the panel answers only for the identifiers it was built with, only on its own session, and a merge clears it. Calling this with a different session or a name it does not hold replaces it, so the caller never has to say when it is done.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	The instruments about to be read one at a time.	required
`date`	`str \| Timestamp`	The session they will be read on. Resolve it first with :meth:`resolve_session`, since a panel for a closed day holds nothing and every read would fall back to the frame.	required

fetch_shares_outstanding ¶

fetch_shares_outstanding(
    identifier: str,
    date: str,
    column: str = "SHARES_OUTSTANDING",
) -> float | None

Return shares outstanding for identifier on date.

Sourced from the column market-data field. Returns None if the column is not present or there is no value on that date.

fetch_free_float_factor ¶

fetch_free_float_factor(
    identifier: str, date: str, column: str = "FREE_FLOAT"
) -> float | None

Return the free-float factor in force for identifier on date.

That day's value when there is one. Otherwise the last value before it, if no older than free_float_backfill_days: free float moves on corporate events and reviews, so a blank cell means nothing was reported, not that the float changed. Never a later value.

Returns None if the column is absent, or nothing was reported within the window. Callers refuse through :func:~beacon.data.free_float.require_free_float rather than choosing a fallback of their own.

prices_on ¶

prices_on(
    identifiers: list[str], date: str, column: str = "CLOSE"
) -> dict[str, float | None]

:meth:fetch_price for many names on one day, in one read.

The batch face of the same answer: warm the day's session once, take the whole column from it, and convert NaN to None, exactly what fetch_price returns name by name. The daily valuation asks this for every holding every day.

A name the session panel does not answer for goes through fetch_price rather than being assumed absent, so a panel that somehow missed it costs a slower read instead of a wrong one.

Returns:

Name	Type	Description
`dict`	`dict[str, float \| None]`	identifier -> price, or None where there is none.

price_array_on ¶

price_array_on(
    identifiers: list[str], date: str, column: str = "CLOSE"
) -> np.ndarray | None

:meth:prices_on as one float array in the order of identifiers, NaN where it gives None; None when the column cannot be read this way, and the caller reads :meth:prices_on instead.

For a caller that asks about the same list of names day after day: the names are looked up once per list, so pass the same list object.

last_market_values ¶

last_market_values(
    identifiers: list[str],
    start: str | Timestamp,
    end: str | Timestamp,
    column: str,
) -> np.ndarray | None

Each name's last column value from start to end, as a :meth:fetch_market_data read of the window would give it, in the order of identifiers, NaN where a name has none; None when the column cannot be read this way, and the caller reads the window instead.

fetch_price ¶

fetch_price(
    identifier: str, date: str, column: str = "CLOSE"
) -> float | None

Return identifier's price on date, or None if it did not print.

The scalar form of the single-day, single-name fetch a methodology makes for every name in a universe. It reads the same value fetch_market_data(identifier, date, date) does (the same column of the same row) and returns it rather than a one-row frame to slice, which is what lets a warmed session serve it. Returns None too when the column is absent from the market data.

last_prices ¶

last_prices(
    identifiers: list[str],
    as_of: str,
    lookback_days: int,
    currency: str | None = None,
) -> pd.Series

Each instrument's last price on or before as_of, looking back lookback_days calendar days, converted into currency; an empty Series when none is priced.

fetch_prices ¶

fetch_prices(
    identifiers: list[str],
    start_date: str | None = None,
    end_date: str | None = None,
    currency: str | None = None,
    column: str = "CLOSE",
) -> pd.DataFrame

Prices by date and instrument, converted into currency.

The read to use whenever prices of several instruments are compared or added up: each is converted from its own currency day by day under this fetcher's FX policy. With currency=None nothing is converted. See beacon.data.prices.

Raises:

Type	Description
`CalculationError`	If an instrument's currency has no rate into currency.

fetch_fx_rates ¶

fetch_fx_rates(
    from_currency: str,
    to_currency: str,
    start_date: str | None = None,
    end_date: str | None = None,
) -> pd.Series

Return the stored FX rate series converting from_currency into to_currency.

The pair is the spot pair whose reference row has from_currency as its BASE_CURRENCY and to_currency as its QUOTE_CURRENCY, and its rate is its CLOSE. Returns an empty Series if that exact pair is not stored: this does not invert or cross rates. :meth:fx_rate_on and :meth:fx_rates_on do.

fx_route ¶

fx_route(
    from_currency: str, to_currency: str
) -> str | None

How a rate for this pair is found: direct, inverse or a cross.

Returns:

Type	Description
`str \| None`	str \| None: "direct" for a stored pair, "inverse" for one over the
`str \| None`	stored reverse pair, "cross via USD" for a rate built from two
`str \| None`	legs, "same currency" when no conversion is needed, and None when
`str \| None`	no rate can be found.

fx_rate_on ¶

fx_rate_on(
    from_currency: str,
    to_currency: str,
    date: str | Timestamp,
) -> float | None

The rate converting from_currency into to_currency on date.

The single currency conversion in the library, so the number displayed and the number weighted by are the same quantity. The pair is found as stored, else as the inverse of the stored reverse pair, else as a cross through USD (see beacon.data.fx and :meth:fx_route).

The series is fetched once per ordered pair and cached, because a run asks this on every foreign name on every day and each fetch slices the whole market frame.

Parameters:

Name	Type	Description	Default
`from_currency`	`str`	The currency being converted out of.	required
`to_currency`	`str`	The currency being converted into.	required
`date`	`str \| Timestamp`	The date the rate is wanted on.	required

Returns:

Type	Description
`float \| None`	float \| None: The rate in force on date, or None when the pair
`float \| None`	is unknown or when its history begins after date. Under the
`float \| None`	default `CARRY_FORWARD` policy the last rate on or before date
`float \| None`	is used; under `EXACT_DAY` only a rate printed on date itself,
`float \| None`	else None. Callers treat None as "cannot convert" rather than as a
`float \| None`	rate of one. Nothing here invents parity on a caller's behalf: a
`float \| None`	rate of 1.0 is a claim about two currencies, and the only one this
`float \| None`	makes is that a currency converts into itself.
`float \| None`	Carried forward only, never backward: a rate dated after the
`float \| None`	day it would be applied to is look-ahead. As with
`float \| None`	`resolve_session`, there is no earlier observation to be in force,
`float \| None`	so there is no answer rather than a substitute for one.

latest_rows ¶

latest_rows(
    identifiers: list[str],
    as_of: Timestamp,
    columns: list[str] | None = None,
    recent_days: int = RECENT_DAYS,
) -> pd.DataFrame

One row per name: its most recent bar at or before as_of.

The batch "what is the last thing we know about these names" read, shared by the reference endpoint and the staleness gate.

Read in two stages: the last recent_days first, which answers almost every name, then only the names it missed again without a lower bound. The result is reduced to one row per name once, with a grouped tail, rather than sliced per name downstream.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	Names to look up.	required
`as_of`	`Timestamp`	The date to look back from, inclusive.	required
`columns`	`list[str] \| None`	Columns to read, or None for all of them.	`None`
`recent_days`	`int`	How far back the cheap first stage reaches.	`RECENT_DAYS`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: MultiIndexed as the market data is, holding at most
`DataFrame`	one row per identifier. Names with no bar at or before as_of are
`DataFrame`	absent rather than present-and-empty.

last_priced_on ¶

last_priced_on(
    identifiers: list[str], as_of: Timestamp
) -> dict[str, pd.Timestamp]

When each name last printed a bar at or before as_of.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	Names to look up.	required
`as_of`	`Timestamp`	The date to look back from.	required

Returns:

Name	Type	Description
`dict`	`dict[str, Timestamp]`	identifier -> the date of its last bar. A name with no bar
	`dict[str, Timestamp]`	at all is absent from the mapping, which is a different thing from
	`dict[str, Timestamp]`	one whose last bar is old.

stale_identifiers ¶

stale_identifiers(
    identifiers: list[str], as_of: Timestamp
) -> set[str]

Which names have not traded recently enough to be worth holding.

Empty when no threshold is set, which is the default: staleness is something an installation opts into with max_price_staleness_days, and until it does this costs one comparison and reads nothing.

A name with no price at all is not reported here. That is a different condition with a different remedy (the weighting already refuses it by name), and folding the two together would quietly excuse a missing instrument as a quiet one.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	Names to test.	required
`as_of`	`Timestamp`	The date staleness is measured from.	required

Returns:

Name	Type	Description
`set`	`set[str]`	Identifiers whose last bar is more than
	`set[str]`	`max_price_staleness_days` calendar days before as_of.

fx_rates_on ¶

fx_rates_on(
    from_currency: str, to_currency: str, days: Index
) -> pd.Series | None

:meth:fx_rate_on over many days at once, as a Series.

The vectorised face of the same rule, for a caller that needs a rate for every day of a run rather than one date, such as chained levels. Asking per day would be one search per day per currency where a single reindex answers the lot. The policy, the carry semantics, the routes and the meaning of "no rate" are the same as :meth:fx_rate_on; the two methods differ only in how many answers they return.

Parameters:

Name	Type	Description	Default
`from_currency`	`str`	The currency being converted out of.	required
`to_currency`	`str`	The currency being converted into.	required
`days`	`Index`	The dates wanted, ascending.	required

Returns:

Type	Description
`Series \| None`	pd.Series \| None: One rate per day, indexed by days, or None when
`Series \| None`	the pair is unknown entirely. Individual days the policy cannot
`Series \| None`	answer for are NaN: a day is missing, not the pair.

fetch_reference_data ¶

fetch_reference_data(
    identifier: str | list[str],
    date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Fetch reference data for one or more identifiers.

Parameters:

Name	Type	Description	Default
`identifier`	`str \| list[str]`	One identifier or a list of identifiers.	required
`date`	`str \| None`	Point-in-time date. Only rows valid at this date are returned.	`None`
`columns`	`list[str] \| None`	Subset of columns to return.	`None`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Indexed by `IDENTIFIER`. Empty DataFrame if no
`DataFrame`	reference data is loaded or identifier is not found.

fetch_classification ¶

fetch_classification(
    identifier: str,
    date: str | Timestamp | None = None,
    scheme: str = DEFAULT_SCHEME,
) -> str | None

One instrument's classification as it stood on a date.

Reference data already carries validity ranges, so a name that moved from Industrials to Technology has two rows and this returns whichever was in force. That matters for anything historical: attributing a 2021 return to a sector the company only joined in 2023 is a real way to get a breakdown wrong.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`date`	`str \| Timestamp \| None`	The as-of date. None takes the currently-active record (the one with no end date), falling back to the latest start date if every record has been closed off.	`None`
`scheme`	`str`	Which column to read, e.g. `"SECTOR"`, `"INDUSTRY"`, `"COUNTRY"`. Free-form, because which columns a client loads is its own business.	`DEFAULT_SCHEME`

Returns:

Type	Description
`str \| None`	str or None: The classification, or None when it is unknown: no
`str \| None`	reference data, no such instrument, no such column, or no record
`str \| None`	valid on that date.

fetch_classifications ¶

fetch_classifications(
    identifiers: list[str],
    date: str | Timestamp | None = None,
    scheme: str = DEFAULT_SCHEME,
) -> dict[str, str | None]

Classifications for several instruments at once.

Every identifier appears, with None where the classification is unknown, so a caller can see what is missing rather than finding it silently absent.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	The instruments.	required
`date`	`str \| Timestamp \| None`	As-of date, as for :meth:`fetch_classification`.	`None`
`scheme`	`str`	Which column to read.	`DEFAULT_SCHEME`

Returns:

Name	Type	Description
`dict`	`dict[str, str \| None]`	Identifier to classification.

group_by_classification ¶

group_by_classification(
    identifiers: list[str],
    date: str | Timestamp | None = None,
    scheme: str = DEFAULT_SCHEME,
) -> dict[str, list[str]]

Instruments grouped by classification, ready for GroupBounds.

Unclassified instruments are collected under UNCLASSIFIED rather than dropped. A name missing from every bucket is how a constraint set quietly stops covering part of the universe.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	The instruments.	required
`date`	`str \| Timestamp \| None`	As-of date.	`None`
`scheme`	`str`	Which column to read.	`DEFAULT_SCHEME`

Returns:

Name	Type	Description
`dict`	`dict[str, list[str]]`	Classification to the identifiers carrying it, each list in
	`dict[str, list[str]]`	the order the identifiers were given.

free_float ¶

The free float in force on a date, and the one refusal when there is none.

Every float-adjusted read (the weighting, the market values and the special-dividend divisor) resolves a missing value the same way. Free float moves on corporate events and index-provider reviews, not daily, so a value a few weeks old is still the right number, and the last known value carries forward within a window:

Global, set as free_float_backfill_days on the DataFetcher beside fx_policy and max_price_staleness_days. It changes numbers, so it is one setting for the installation and published on /health, rather than a default buried at each read.
Default 90 days. A quarterly review cycle fits inside it with room to spare. 0 turns carrying off, so only a value dated that day is used.
Forward only. A value dated after the date is never used.
Beyond the window, refuse. Everywhere, including the dividend path. A float last seen a year ago is not the float.

validated_window ¶

validated_window(days: int) -> int

Check a backfill window, returning it unchanged.

Raises:

Type	Description
`ValueError`	If it is negative or not an integer. There is no "unlimited": a float last seen a year ago is not the float.

carried_forward ¶

carried_forward(
    history: Series, date: Timestamp, window_days: int
) -> float | None

The last value on or before date, if it is no older than the window.

Parameters:

Name	Type	Description	Default
`history`	`Series`	One name's observed free floats, indexed by date, sorted, with the missing cells already dropped.	required
`date`	`Timestamp`	The date the free float is wanted on.	required
`window_days`	`int`	How many calendar days old the value may be.	required

Returns:

Type	Description
`float \| None`	float \| None: The value in force, or None when there is none on or
`float \| None`	before date, or the latest one is older than the window.

require_free_float ¶

require_free_float(
    provider: Any,
    identifier: str,
    date: str,
    calculation_name: str,
) -> float

The free float to scale identifier's market cap by, or a refusal.

Every float-adjusted read goes through here (the weighting, the market values and the special-dividend divisor), so the three give one answer to a missing value and say the same thing when they refuse.

Takes the provider as an argument rather than living on the fetcher, so a test double that answers fetch_free_float_factor gets the same rule.

Parameters:

Name	Type	Description	Default
`provider`	`Any`	Anything with `fetch_free_float_factor`, which applies the backfill window.	required
`identifier`	`str`	The instrument.	required
`date`	`str`	The date wanted, YYYY-MM-DD.	required
`calculation_name`	`str`	What is asking, for the error.	required

Raises:

Type	Description
`CalculationError`	If there is no value within the window, or the value is outside 0-1. Using the full market cap instead would weight the name as though every share were freely traded, which is a different index from the one specified.

fx ¶

Which rate converts one currency into another, when only some pairs are stored.

A pair is stored as its own market-data identifier, such as GBPUSD. With only GBPUSD and EURUSD stored, USD to GBP is one over GBPUSD and GBP to EUR is GBPUSD divided by EURUSD, so neither needs a stored pair of its own.

The rate for a pair is found in this order:

the stored pair itself
the inverse of the stored reverse pair
a cross through USD, each leg found by rules 1 and 2

A cross is built so the settings keep their meaning. Under carry-forward a leg's rate is the one in force on the day, so the cross on a day uses both legs as in force that day. Under exact-day a leg must have printed on the day, so the cross exists only on days both legs printed. Neither ever uses a rate dated after the day, and the no-look-ahead search in DataFetcher applies to the derived series exactly as to a stored one.

A stored rate of zero or below has no inverse; it is dropped, and that day is treated as one with no rate.

rate_series ¶

rate_series(
    stored: StoredRates,
    source: str,
    target: str,
    exact_day: bool,
) -> tuple[pd.Series, str | None]

The series converting source into target, and how it was found.

Parameters:

Name	Type	Description	Default
`stored`	`StoredRates`	Reads a stored pair, returning an empty series when absent.	required
`source`	`str`	The currency converted out of, upper case.	required
`target`	`str`	The currency converted into, upper case.	required
`exact_day`	`bool`	True under the EXACT_DAY policy, where a cross exists only on days both legs printed.	required

Returns:

Name	Type	Description
`tuple`	`Series`	The rate series indexed by date and sorted, and the route:
	`str \| None`	DIRECT, INVERSE or CROSS. An empty series and None when no route
	`tuple[Series, str \| None]`	exists.

identifiers ¶

Searching and enumerating the identifiers a data source actually holds.

Answers "which identifiers do you have, and which of them match what the user is typing" across market data, reference data and corporate actions, so a client can offer type-ahead over everything the source holds, not only the members of configured universes.

Built once, searched many times¶

This is called on every keystroke. A scan that reads reference data per request would be doing the expensive part (pulling names out of pandas) over and over for an answer that only changes when the data does.

So the index is built once and cached against a fingerprint of the fetcher's refresh timestamps. A sync moves those, the fingerprint changes, and the next request rebuilds. Nothing else invalidates it, because nothing else can change which identifiers exist.

Matching strings are folded at build time, not per request. That is the single biggest thing keeping a 100k-row search inside its budget: lowercasing a hundred thousand strings per keystroke costs more than the matching does.

Ranking is the server's job¶

Once limit is applied the caller cannot re-rank what it was not sent, so the order has to be right here. Best first:

exact identifier match
identifier prefix
name prefix
identifier substring
name substring

with an alphabetical tie-break inside each tier. Someone typing CMP0 wants CMP000, not the first company whose description happens to contain that fragment.

Why `total` forces a full scan¶

total is the number of matches before the limit, so the client can say "showing 20 of 340". That cannot be known without examining every candidate, so there is no early exit: the limit bounds what is returned, never what is looked at. Worth knowing before trying to optimise the loop away.

IdentifierEntry `dataclass` ¶

IdentifierEntry(
    identifier: str,
    name: str | None = None,
    datasets: tuple[str, ...] = (),
    exchange: str | None = None,
    currency: str | None = None,
    folded_identifier: str = "",
    folded_name: str = "",
)

One identifier the data source holds.

Attributes:

Name	Type	Description
`identifier`	`str`	The symbol itself.
`name`	`str \| None`	Display name, or None when reference data carries none. A row without a name is still a useful suggestion, so it is returned rather than dropped.
`datasets`	`tuple[str, ...]`	Which datasets actually cover it. This is what lets a client offer a reference-only name in a reference view and mark it unavailable for prices, rather than suggesting something it cannot then serve.
`exchange`	`str \| None`	Where it trades, when known.
`currency`	`str \| None`	Its denomination, when known.

rank_against ¶

rank_against(query: str) -> int

Which match tier this entry falls into for a folded query.

Returns:

Name	Type	Description
`int`	`int`	One of the tier constants, or NO_MATCH.

SearchResult `dataclass` ¶

SearchResult(
    entries: tuple[IdentifierEntry, ...],
    total: int,
    truncated: bool,
)

What a search found.

Attributes:

Name	Type	Description
`entries`	`tuple[IdentifierEntry, ...]`	The rows to return, ranked and windowed.
`total`	`int`	Matches before the limit, so a client can say "showing 20 of 340".
`truncated`	`bool`	Whether the limit hid anything. Derivable from `total` and the window, but explicit beats arithmetic at a call site.

IdentifierIndex ¶

IdentifierIndex(
    entries: tuple[IdentifierEntry, ...], version: str
)

An in-memory index of the identifiers a fetcher holds.

Parameters:

Name	Type	Description	Default
`entries`	`tuple[IdentifierEntry, ...]`	Pre-built rows, ordered alphabetically by identifier.	required
`version`	`str`	A fingerprint of the data this was built from, used both to decide when to rebuild and as an ETag.	required

empty `classmethod` ¶

empty() -> IdentifierIndex

An index over no data.

A server with no data source still answers searches: "nothing matches" and "this engine is misconfigured" are different statements, and an empty suggestion list must not look like a broken install.

build `classmethod` ¶

build(fetcher: DataFetcher) -> IdentifierIndex

Index everything a fetcher holds.

Parameters:

Name	Type	Description	Default
`fetcher`	`DataFetcher`	The data source.	required

Returns:

Name	Type	Description
`IdentifierIndex`	`IdentifierIndex`	Entries ordered by identifier, so an enumeration
	`IdentifierIndex`	with no query is alphabetical and every tie-break below is stable
	`IdentifierIndex`	without re-sorting.

search ¶

search(
    query: str | None = None,
    limit: int = DEFAULT_LIMIT,
    offset: int = 0,
    datasets: tuple[str, ...] = (),
) -> SearchResult

Find identifiers matching a fragment, or enumerate them all.

Parameters:

Name	Type	Description	Default
`query`	`str \| None`	Fragment to match against identifier and name, case-insensitively. None or empty means no filter: the first `limit` in index order, after `offset`.	`None`
`limit`	`int`	Maximum rows to return.	`DEFAULT_LIMIT`
`offset`	`int`	Rows to skip, for walking a full enumeration.	`0`
`datasets`	`tuple[str, ...]`	Only return identifiers covered by all of these.	`()`

Returns:

Name	Type	Description
`SearchResult`	`SearchResult`	Ranked rows, plus the total before the limit.

fingerprint ¶

fingerprint(fetcher: DataFetcher) -> str

A short version tag for the data an index would be built from.

Made from the market, reference and corporate-action refresh timestamps alone. Every path that can change which identifiers exist (both merge methods) records a refresh, so the timestamps are a complete signal.

Deliberately not derived from the identifier count: reading that means a unique() over the whole market panel, which on a hundred thousand names is exactly the work a per-request cache key must not do. It also has to be computable without building the index, or checking the cache would cost as much as missing it.

Used both to decide when to rebuild and as the ETag a client revalidates against.

importing ¶

Load data from CSV files or an Excel workbook, in the layout beacon.data.layout sets out.

Give CSV files named after their sheet (market.csv, reference.csv, ...), or one Excel workbook whose sheets carry those names. Names are matched without regard to case or spaces, so a sheet called "Corporate Actions" is the corporate_actions sheet, and a column called "close" is CLOSE.

from beacon.data import importing

data = importing.load_files(["market.csv", "reference.csv"])

Everything is checked before anything is loaded. If any row is wrong, load_files raises DataImportError listing every problem, each with its sheet, row and column, so the files can be fixed in one pass.

template writes a blank template to start from, as an Excel workbook or a zip of CSV files.

DataImportError ¶

DataImportError(problems: list[Problem], total: int)

Bases: InvalidRuleError

The supplied data has problems, and nothing was loaded.

Attributes:

Name	Type	Description
`problems`	`list[Problem]`	Up to `MAX_PROBLEMS` of them, each naming its sheet, row and column.
`total`		How many problems there are in all.
`findings`		The same problems in the shape every other refusal with findings uses (`path`, `severity`, `code`, `message`), plus `sheet`, `row` and `column`. This is what the API server sends.

read ¶

read(
    paths: Iterable[str | Path],
) -> tuple[dict[str, pd.DataFrame], list[layout.Problem]]

Read the files into sheets, and the problems reading them.

Every value is read as text, so the checks see exactly what was written, and a date or number in the wrong form is reported rather than guessed.

load_files ¶

load_files(paths: Iterable[str | Path]) -> DataFetcher

Read, check and load CSV files or an Excel workbook.

Raises:

Type	Description
`DataImportError`	If a file cannot be read or any row has a problem. Nothing is loaded in that case.

template ¶

template(fmt: str = 'xlsx') -> bytes

A blank template with every sheet's columns and example rows.

Parameters:

Name	Type	Description	Default
`fmt`	`str`	"xlsx" for an Excel workbook, or "csv" for a zip of CSV files.	`'xlsx'`

Raises:

Type	Description
`ValueError`	For any other format.

ingest ¶

Fetching market and reference data from an external source.

Downloads history and reference records per identifier and reshapes them into the long form MarketData and ReferenceData load. The yfinance-backed downloaders need the data extra.

The downloader is injected¶

Everything here takes a downloader: a callable from (identifier, start, end) to a date-indexed OHLCV frame. The yfinance-backed one is built by :func:yfinance_downloader, and that is the only place the optional dependency is touched.

The point is not abstraction for its own sake. It means this module imports and runs with no network and no extra installed, so the reshaping, the partial- failure handling and the progress reporting are all testable against a fake that returns known numbers, none of which could be tested honestly against a live market-data service.

One identifier at a time, and failures do not spread¶

A sync over five hundred names must not collapse because one was delisted last month. Each identifier is fetched separately, a failure is recorded against that identifier alone, and the run continues. The result says what came back and what did not, so a caller can tell a successful sync from a sync that succeeded at nothing.

Fetching them one at a time also makes progress mean something: "142 of 500" is a real statement, whereas a single bulk request can only ever report 0% and then 100%.

IngestResult `dataclass` ¶

IngestResult(
    market: DataFrame,
    reference: DataFrame = pd.DataFrame(),
    fetched: list[str] = list(),
    failed: dict[str, str] = dict(),
)

What a sync managed to fetch.

Attributes:

Name	Type	Description
`market`	`DataFrame`	Long-form market data, ready for `MarketData.from_dataframe`. Empty when nothing was fetched.
`reference`	`DataFrame`	Long-form reference data. Empty when none was requested or none came back.
`fetched`	`list[str]`	Identifiers that returned data.
`failed`	`dict[str, str]`	Identifier to the reason it did not, so a caller can tell a successful sync from one that succeeded at nothing.

rows `property` ¶

rows: int

Market-data rows fetched.

succeeded `property` ¶

succeeded: bool

Whether anything at all came back.

summary ¶

summary() -> dict[str, object]

A JSON-ready description, for a job result or a log line.

yfinance_downloader ¶

yfinance_downloader() -> Downloader

Build a downloader backed by yfinance.

Returns:

Name	Type	Description
`Downloader`	`Downloader`	Callable fetching one identifier's history.

Raises:

Type	Description
`MissingDependencyError`	If yfinance is not installed. Raised here rather than at import, so this module stays usable with an injected downloader and no extra installed.

yfinance_reference_downloader ¶

yfinance_reference_downloader() -> Callable[
    [str], dict[str, object]
]

Build a reference-data downloader backed by yfinance.

Returns:

Name	Type	Description
`Callable`	`Callable[[str], dict[str, object]]`	Takes an identifier and returns its info mapping.

Raises:

Type	Description
`MissingDependencyError`	If yfinance is not installed.

normalise_history ¶

normalise_history(
    identifier: str, frame: DataFrame
) -> pd.DataFrame

Reshape a downloaded OHLCV frame into the long form MarketData expects.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument the frame belongs to.	required
`frame`	`DataFrame`	Date-indexed frame with Open/High/Low/Close/Volume columns.	required

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Rows of `IDENTIFIER`, `DATE` and the mapped columns.
`DataFrame`	Empty when the input is empty.

Raises:

Type	Description
`DataNotFoundError`	If there is no close price. Every other column is optional (a series with no volume is still a usable price series), but the calculator and the engine both read CLOSE, so a frame without one would be accepted here and fail much later.

normalise_reference ¶

normalise_reference(
    identifier: str,
    info: dict[str, object],
    valid_from: str = DEFAULT_VALID_FROM,
) -> dict[str, object]

Reshape a downloaded info mapping into a reference-data row.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`info`	`dict[str, object]`	The source's field mapping.	required
`valid_from`	`str`	DATE_FROM to stamp on the record.	`DEFAULT_VALID_FROM`

Returns:

Name	Type	Description
`dict`	`dict[str, object]`	A reference row. Fields the source did not supply are simply
	`dict[str, object]`	absent rather than filled with a placeholder.

ingest_market_data ¶

ingest_market_data(
    identifiers: list[str],
    downloader: Downloader,
    start: str | None = None,
    end: str | None = None,
    on_progress: Callable[[int, int, str], None]
    | None = None,
) -> IngestResult

Fetch history for several identifiers.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	What to fetch.	required
`downloader`	`Downloader`	Where to fetch it from.	required
`start`	`str \| None`	Inclusive start date.	`None`
`end`	`str \| None`	End date, passed to the downloader as given. Yahoo Finance treats it as exclusive: pass the day after the last one wanted.	`None`
`on_progress`	`Callable[[int, int, str], None] \| None`	Called with `(done, total, identifier)` after each one, so a job can report real progress rather than 0% then 100%.	`None`

Returns:

Name	Type	Description
`IngestResult`	`IngestResult`	The combined data and a per-identifier outcome.

ingest_reference_data ¶

ingest_reference_data(
    identifiers: list[str],
    downloader: Callable[[str], dict[str, object]],
    valid_from: str = DEFAULT_VALID_FROM,
    on_progress: Callable[[int, int, str], None]
    | None = None,
) -> IngestResult

Fetch reference records for several identifiers.

Parameters:

Name	Type	Description	Default
`identifiers`	`list[str]`	What to fetch.	required
`downloader`	`Callable[[str], dict[str, object]]`	Where to fetch it from.	required
`valid_from`	`str`	DATE_FROM to stamp on each record.	`DEFAULT_VALID_FROM`
`on_progress`	`Callable[[int, int, str], None] \| None`	Called with `(done, total, identifier)` after each one.	`None`

Returns:

Name	Type	Description
`IngestResult`	`IngestResult`	The reference rows and a per-identifier outcome.

kinds ¶

What each series in the data is, as its reference data says.

Every market series carries its value in CLOSE: an instrument's price, a currency pair's rate, a short-term interest rate. Reference data tells them apart:

ASSET_CLASS: Equity, Currency or Fixed Income. A row without one, or reference data without the column, is Equity. Another value is kept as given and read like an equity: a priced series.
SECURITY_TYPE: required for a currency or fixed income row, because it says how CLOSE is read. Spot is a currency pair; Short Rate is a short-term interest rate.
BASE_CURRENCY and QUOTE_CURRENCY, on a spot pair: CLOSE is units of the quote currency per unit of the base. CURRENCY, when given, must be the quote currency.
CURRENCY, on a short rate: the currency the rate is for. CLOSE is an annual decimal (0.0525 for 5.25%). TENOR (ON, 1M, 3M, ...) and DAY_COUNT (ACT/360, the default, or ACT/365) are optional.
TRADABLE: whether a fund can hold the series. Spot pairs and short rates are references and default to not tradable; everything else defaults to tradable. A value in the data overrides the default.

Values are matched without regard to case. An identifier's latest reference row decides what it is.

SeriesError ¶

Bases: ValueError

Reference data describes a series wrongly, a short rate looks like a percent, or market data still has a RATE column.

ShortRate `dataclass` ¶

ShortRate(
    identifier: str,
    currency: str,
    tenor: str | None,
    day_count: float,
)

A currency's short-term interest rate series.

Attributes:

Name	Type	Description
`identifier`	`str`	Its market identifier.
`currency`	`str`	The currency the rate is for.
`tenor`	`str \| None`	Its tenor, such as `"ON"` or `"1M"`, or None.
`day_count`	`float`	Days in a year for its interest: 360.0 or 365.0.

Kinds `dataclass` ¶

Kinds(
    pairs: dict[tuple[str, str], str] = dict(),
    short_rates: dict[str, ShortRate] = dict(),
    untradable: frozenset[str] = frozenset(),
)

What each identifier in the reference data is.

Attributes:

Name	Type	Description
`pairs`	`dict[tuple[str, str], str]`	Each spot pair's identifier, by (base, quote) currency.
`short_rates`	`dict[str, ShortRate]`	Each currency's short rate, the preferred one when it has several.
`untradable`	`frozenset[str]`	Identifiers a fund cannot hold.

pair_identifiers `property` ¶

pair_identifiers: list[str]

Every spot pair's identifier, sorted.

short_rate_identifiers `property` ¶

short_rate_identifiers: list[str]

Every chosen short rate's identifier, sorted.

Described `dataclass` ¶

Described(
    asset_class: str,
    security_type: str | None,
    pair: tuple[str, str] | None,
    short_rate: ShortRate | None,
    tradable: bool,
)

What one reference row says its series is.

Attributes:

Name	Type	Description
`asset_class`	`str`	Its asset class, `Equity` when not given.
`security_type`	`str \| None`	Its security type, or None.
`pair`	`tuple[str, str] \| None`	(base, quote) for a spot pair, else None.
`short_rate`	`ShortRate \| None`	The rate it is, for a short rate, else None.
`tradable`	`bool`	Whether a fund can hold it.

describe ¶

describe(identifier: str, row: Series) -> Described

What row, a reference row of identifier, says its series is.

Raises:

Type	Description
`SeriesError`	If a currency or fixed income row has no security type, a spot pair lacks its currencies or is quoted in another, a short rate has no currency or an unknown day count, or a TRADABLE value is not a yes or no.

classify ¶

classify(reference: DataFrame | None) -> Kinds

Read what each identifier is from reference, indexed by identifier.

Raises:

Type	Description
`SeriesError`	If a row describes its series wrongly (see `describe`), or two spot pairs quote the same currencies.

problems ¶

problems(
    reference: DataFrame,
) -> Iterator[tuple[int | None, str | None, str]]

Everything wrong with how reference, a frame with an IDENTIFIER column, describes its series: each as (row position or None, column or None, message). An equity needs an EXCHANGE.

short_rates_in ¶

short_rates_in(reference: DataFrame) -> set[str]

The identifiers reference, a frame with an IDENTIFIER column, describes as short rates. Rows that describe their series wrongly are left out: problems reports them.

refuse_rate_column ¶

refuse_rate_column(columns: list[str]) -> None

Refuse market data that still carries a RATE column.

Raises:

Type	Description
`SeriesError`	If columns holds `RATE`.

pair_reference ¶

pair_reference(
    identifier: str,
    base: str | None = None,
    quote: str | None = None,
    date_from: str = "1900-01-01",
) -> dict[str, object]

A reference row that makes identifier a spot currency pair.

base and quote default to the identifier's two halves, so pair_reference("GBPUSD") describes pounds quoted in dollars.

Raises:

Type	Description
`ValueError`	If the currencies are not given and the identifier is not six letters.

short_rate_reference ¶

short_rate_reference(
    identifier: str,
    currency: str,
    tenor: str | None = None,
    day_count: str = ACT_360,
    date_from: str = "1900-01-01",
) -> dict[str, object]

A reference row that makes identifier currency's short rate.

layout ¶

The table layout data is supplied in, and the checks every row must pass.

One layout serves every way a user brings their own data: CSV files, an Excel workbook, and the tables or views of a Postgres database. Each dataset is a sheet (or file, or table) with a fixed name and fixed columns:

market (required): IDENTIFIER, DATE, CLOSE; optionally OPEN, HIGH, LOW, VOLUME, SHARES_OUTSTANDING, FREE_FLOAT. Every series is here, currency pairs and short rates included, its value in CLOSE.
reference (required): IDENTIFIER, NAME, CURRENCY, DATE_FROM; EXCHANGE for an equity; optionally DATE_TO, the columns that say what a series is (ASSET_CLASS, SECURITY_TYPE, BASE_CURRENCY, QUOTE_CURRENCY, TENOR, DAY_COUNT, TRADABLE; see beacon.data.kinds) and descriptive columns such as SECTOR and COUNTRY.
corporate_actions: IDENTIFIER, EX_DATE, TYPE, VALUE; optionally PAY_DATE, STATUS.
features: IDENTIFIER, DATE, FIELD, VALUE; optionally TYPE (default "imported") and DETAIL.

Extra columns are kept, so a reference sheet can carry any descriptive fields.

check reports every problem at once, each naming its sheet, row and column, so a user can fix a file in one pass. Nothing is loaded from sheets that fail it: a partly valid import is refused whole rather than half-loaded.

Sheet `dataclass` ¶

Sheet(
    name: str,
    required: tuple[str, ...],
    optional: tuple[str, ...] = (),
    dates: tuple[str, ...] = (),
    numbers: tuple[str, ...] = (),
    positive: tuple[str, ...] = (),
    key: tuple[str, ...] = (),
    needed: bool = False,
)

One dataset's name, columns and rules.

Attributes:

Name	Type	Description
`name`	`str`	The sheet, file or table name, lower case.
`required`	`tuple[str, ...]`	Columns every row must fill.
`optional`	`tuple[str, ...]`	Known columns that may be left out or blank.
`dates`	`tuple[str, ...]`	Columns that must hold dates.
`numbers`	`tuple[str, ...]`	Columns that must hold numbers.
`positive`	`tuple[str, ...]`	Numeric columns that must be above zero.
`key`	`tuple[str, ...]`	Columns that together identify a row; no two rows may share them. Empty when repeats are allowed.
`needed`	`bool`	Whether the sheet must be supplied at all.

Problem `dataclass` ¶

Problem(
    sheet: str,
    row: int | None,
    column: str | None,
    code: str,
    message: str,
)

One thing wrong with the supplied data.

Attributes:

Name	Type	Description
`sheet`	`str`	The sheet, file or table.
`row`	`int \| None`	The row as a spreadsheet shows it (the header is row 1), or None for a problem with the sheet as a whole.
`column`	`str \| None`	The column, or None.
`code`	`str`	A stable code, such as MISSING_COLUMN or BAD_DATE.
`message`	`str`	What is wrong, in a sentence.

DataImportError ¶

DataImportError(problems: list[Problem], total: int)

Bases: InvalidRuleError

The supplied data has problems, and nothing was loaded.

Attributes:

Name	Type	Description
`problems`	`list[Problem]`	Up to `MAX_PROBLEMS` of them, each naming its sheet, row and column.
`total`		How many problems there are in all.
`findings`		The same problems in the shape every other refusal with findings uses (`path`, `severity`, `code`, `message`), plus `sheet`, `row` and `column`. This is what the API server sends.

sheet_name ¶

sheet_name(raw: str) -> str

A sheet or file name as the layout spells it: "Corporate Actions" is corporate_actions.

tidy ¶

tidy(frame: DataFrame) -> pd.DataFrame

Column names in the layout's spelling, rows numbered from zero, and fully blank rows (a common spreadsheet leftover) dropped.

check ¶

check(
    sheets: dict[str, DataFrame],
) -> tuple[list[Problem], int]

Every problem in the supplied sheets.

Parameters:

Name	Type	Description	Default
`sheets`	`dict[str, DataFrame]`	Sheet name (lower case) to its rows, as read from the source.	required

Returns:

Name	Type	Description
`tuple`	`list[Problem]`	The first `MAX_PROBLEMS` problems, and how many there are in
	`int`	all. No problems means the data can be loaded.

to_fetcher ¶

to_fetcher(
    sheets: dict[str, DataFrame], **settings: Any
) -> DataFetcher

Build the data from sheets that passed check.

Parameters:

Name	Type	Description	Default
`sheets`	`dict[str, DataFrame]`	The checked sheets.	required
`**settings`	`Any`	Passed to `DataFetcher`: `fx_policy`, `max_price_staleness_days`, `free_float_backfill_days`.	`{}`

loader ¶

Data loader that reads file paths from an Environment and returns a DataFetcher.

load_data ¶

load_data(env: Environment) -> DataFetcher

Read data files from the environment config and return a DataFetcher.

For each dataset, a DataFrame is checked first; if not provided, the file path is used instead. Raises ValueError if no market data is available from either source.

market_caps ¶

A name's market capitalisation from its price and share count, converted into another currency and adjusted for free float, and a set of names' shares of their total.

Each figure is scaled from its own base rather than one from another: multiplication is not associative in floating point, so the converted free-float cap is the converted cap times the float, not the local free-float cap converted. An unknown rate leaves the converted figures unknown, and an unknown float the free-float ones, rather than assuming 1.

MarketCaps `dataclass` ¶

MarketCaps(
    local: float,
    converted: float | None,
    free_float_local: float | None,
    free_float_converted: float | None,
)

A name's caps: in its own currency and converted, in full and free-float adjusted; None where the rate or the float is unknown.

caps_of ¶

caps_of(
    price: float,
    shares: float,
    rate: float | None,
    free_float: float | None,
) -> MarketCaps

The caps of shares at price, converted by rate and adjusted by free_float (the share of the shares that trade freely).

is_stale ¶

is_stale(
    priced_from: Timestamp,
    on: Timestamp,
    lookback_days: int,
) -> bool

Whether a figure printed on priced_from is more than lookback_days old on on.

shares_of_total ¶

shares_of_total(
    values: dict[str, float | None],
) -> dict[str, float | None]

Each value's share of the positive values' total; None for a name with no value, or when nothing is positive.

postgres ¶

Read data from a Postgres database, in the layout beacon.data.layout sets out.

A Postgres store is tables or views named after the layout's sheets (market, reference, and optionally corporate_actions, features), with the same columns, in one schema. To use data you already hold, create views with those names and columns over your own tables.

The engine only reads: it connects with a read-only session, and never creates, changes or deletes anything in the database.

The password is never stored. Name an environment variable that holds it (password_env), and it is read from there each time the engine connects.

from beacon.data.postgres import PostgresSource, load

source = PostgresSource(host="localhost", database="markets",
                        user="analyst", password_env="MARKETS_PASSWORD")
data = load(source)

PostgresSource `dataclass` ¶

PostgresSource(
    host: str,
    database: str,
    user: str,
    port: int = 5432,
    schema: str = "public",
    password_env: str | None = None,
)

Where a Postgres store's tables are.

Attributes:

Name	Type	Description
`host`	`str`	The database server.
`database`	`str`	The database name.
`user`	`str`	The user to connect as. Read access is all it needs.
`port`	`int`	The server's port.
`schema`	`str`	The schema holding the tables or views.
`password_env`	`str \| None`	The environment variable holding the password, or None when the server needs none (trusted or peer authentication).

describe ¶

describe() -> str

Where the data is, without the password.

password ¶

password() -> str | None

The password from its environment variable, or None.

password_missing ¶

password_missing() -> bool

Whether a password variable is named but not set.

connect ¶

connect(source: PostgresSource) -> Any

A read-only connection to the database.

read ¶

read(
    source: PostgresSource,
    connector: Connector | None = None,
) -> tuple[dict[str, pd.DataFrame], list[layout.Problem]]

Read each layout table the schema holds, and the problems reading them.

A table the schema does not have is simply absent, and layout.check says so if it was required. Any other failure (no permission, say) is a problem naming the table. connector defaults to connect, looked up when called.

load ¶

load(
    source: PostgresSource,
    connector: Connector | None = None,
    **settings: Any,
) -> DataFetcher

Read, check and load a Postgres store.

Raises:

Type	Description
`DataImportError`	If the database cannot be reached, a table cannot be read, or any row has a problem. Nothing is loaded in that case.

prices ¶

Prices for several instruments at once, in one currency.

prices = fetcher.fetch_prices(["AAA", "BBB"], "2024-01-02", "2024-06-28",
                              currency="USD")

Comparing or adding up prices across instruments only means something once they are in the same money. This is the one read that says which money: each instrument's prices are converted from the currency it is quoted in (its reference data's CURRENCY) into currency, day by day, under the dataset's FX policy and through the same rate lookup every other conversion uses (DataFetcher.fx_rates_on).

An instrument whose currency has no rate at all into currency is refused, naming it, rather than left in its own currency.
A day the policy has no rate for is left empty (NaN) rather than guessed.
An instrument with no currency on record is read as quoted in currency: a dataset whose reference data does not model currency is single-currency.
With currency=None nothing is converted, and each column stays in its instrument's own currency.

prices_in ¶

prices_in(
    source: _Source,
    identifiers: Iterable[str],
    start: str | None = None,
    end: str | None = None,
    currency: str | None = None,
    column: str = PRICE_COLUMN,
) -> pd.DataFrame

Prices by date and instrument, converted into currency.

Parameters:

Name	Type	Description	Default
`source`	`_Source`	The data, usually a `DataFetcher`.	required
`identifiers`	`Iterable[str]`	The instruments. One without prices in the window is left out.	required
`start`	`str \| None`	First date, inclusive.	`None`
`end`	`str \| None`	Last date, inclusive.	`None`
`currency`	`str \| None`	The currency to convert into, or None to leave each instrument in its own.	`None`
`column`	`str`	The market-data column to read.	`PRICE_COLUMN`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Dates as rows, one column per instrument priced, in the
`DataFrame`	order given.

Raises:

Type	Description
`CalculationError`	If an instrument's currency has no rate into currency.

converted ¶

converted(
    source: _Source, wide: DataFrame, currency: str
) -> pd.DataFrame

wide, prices by date and instrument each in its own currency, in currency: what prices_in converts after reading.

For a caller that wants the prices both ways, so the data is read once.

Raises:

Type	Description
`CalculationError`	If an instrument's currency has no rate into currency.

currencies_of ¶

currencies_of(
    source: _Source, identifiers: list[str], default: str
) -> dict[str, str]

The currency each instrument is quoted in, from its latest record.

The latest record rather than one on a date, so an instrument that has since been delisted still has its currency. default stands in where no currency is on record.

wide_columns ¶

wide_columns(
    source: _Source,
    identifiers: Iterable[str],
    start: str | None,
    end: str | None,
    columns: list[str],
) -> dict[str, pd.DataFrame]

Several market-data columns as dates by instrument, from one read, each in its instruments' own currency.

Every frame has the same dates and the instruments that have any of the columns, in the order given; a column the data lacks is all missing.

reference_as_of ¶

Reference columns for many names on many dates, each as it stood then.

A weight history asks for the same few columns of the same names on every rebalance date. Asking the reference data once per date would read it as many times as there are dates; this reads every validity window of the names once and answers each (name, date) from them, by the rule ReferenceData.get uses: the row whose DATE_FROM is on or before the date and whose DATE_TO is on or after it, or open.

The columns that say what a series is (ASSET_CLASS, SECURITY_TYPE and TRADABLE) are answered as beacon.data.kinds reads them, so a dataset with no ASSET_CLASS column answers Equity, as the engine takes it to be. Any other column the dataset lacks answers None.

ReferenceAsOf ¶

ReferenceAsOf(
    fetcher: DataFetcher,
    identifiers: Iterable[str],
    columns: list[str],
)

Reference columns of a set of names, answerable on any date.

Parameters:

Name	Type	Description	Default
`fetcher`	`DataFetcher`	The loaded data.	required
`identifiers`	`Iterable[str]`	The names that will be asked about.	required
`columns`	`list[str]`	The columns to answer, as stored (upper case).	required

of ¶

of(
    identifier: str, date: str | Timestamp
) -> dict[str, Any] | None

The columns of identifier as they stood on date, keyed in lower case; None when no reference row was valid then.

map_of ¶

map_of(
    identifiers: Iterable[str], date: str | Timestamp
) -> dict[str, dict[str, Any] | None]

Each of identifiers' columns on one date.

windows_of ¶

windows_of(
    identifier: str,
    start: str | Timestamp,
    end: str | Timestamp,
) -> list[dict[str, Any]]

The windows of identifier that overlap start to end, each its date_from, date_to (None when open) and reference columns: what a daily panel needs to read any of its dates.

returns ¶

Each name's daily total return, in one currency.

returns = name_returns(fetcher, ["AAA", "BBB"], "2024-01-02", "2024-06-28",
                       currency="USD")

A day's return is what a holder of one share made, as a share of the previous close, in currency:

(close × split ratio + cash paid per share) ÷ previous close − 1

Splits (and reverse splits and stock dividends) multiply the shares held, so the close on the day is scaled by the ratio rather than read as a fall.
Cash (dividends, special dividends and returns of capital) counts on its ex-date, gross of any tax withheld, converted at that day's rate. That is when a holder is owed it, and how a backtest's NAV counts it.
Mergers, spin-offs and rights issues are not a scaling of one price, so no return accounts for them, as an adjusted close leaves them out.
A cancelled action never happened and is left out.
A day with no close is empty; the next close's return runs from the last close before the gap, so nothing is lost across it.

name_returns ¶

name_returns(
    source: _Source,
    identifiers: Iterable[str],
    start: str | None = None,
    end: str | None = None,
    currency: str | None = None,
    actions: DataFrame | None = None,
    closes: DataFrame | None = None,
) -> pd.DataFrame

Daily total returns by date and name, in currency.

Parameters:

Name	Type	Description	Default
`source`	`_Source`	The data, usually a `DataFetcher`.	required
`identifiers`	`Iterable[str]`	The names. One without prices in the window is left out.	required
`start`	`str \| None`	First date, inclusive. Its returns are empty, having no close before them in the window.	`None`
`end`	`str \| None`	Last date, inclusive.	`None`
`currency`	`str \| None`	The currency returns are measured in; each name's own when None.	`None`
`actions`	`DataFrame \| None`	The corporate actions, with `IDENTIFIER`, `TYPE`, `VALUE` and `EX_DATE`; read from source's `corporate_actions` when None.	`None`
`closes`	`DataFrame \| None`	The names' closes in their own currencies, dates by name, for a caller that has read them already; read from source when None.	`None`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Dates by name, NaN where a name has no close or no
`DataFrame`	close before it.

session ¶

One market session, read for a whole set of names at once.

SessionPanel ¶

SessionPanel(session: Timestamp, frame: DataFrame)

Every market column of one session, for a named set of instruments.

A methodology that walks a universe asks the same question of every name on the same day: what did it close at, how many shares are out, how much of it floats. Asked one name at a time, each answer is a slice of the whole market frame, so the cost of a single lookup grows with the frame rather than with the question, making a preview superlinear in its universe size. Asked once for the whole list it is one slice, and the per-name reads that follow are dictionary lookups.

The values are held as plain dicts rather than as the frame they came from because the per-name read is the whole point: .at on a DataFrame is microseconds, and microseconds across a six-thousand-name universe times three columns is the cost this exists to remove.

A panel is a pure function of (market frame, identifiers, session), so it can be held and reused for exactly as long as all three hold, and no longer. It answers only for the identifiers it was built with and only for its own session; anything else must go back to the data. That is what keeps a reused panel from being a different question's answer: there is no key under which name A's row can be returned for name B, or Monday's for Tuesday's.

Build a panel from rows already reduced to one session.

Parameters:

Name	Type	Description	Default
`session`	`Timestamp`	The session these rows are for.	required
`frame`	`DataFrame`	Rows indexed by `IDENTIFIER`, one per instrument.	required

from_columns `classmethod` ¶

from_columns(
    session: Timestamp,
    values: dict[str, dict[str, object]],
    identifiers: set[str],
) -> SessionPanel

Build a panel from column dictionaries, with no frame in between.

MarketData.session_columns produces exactly what a panel holds, so no DataFrame is constructed only to be taken apart again.

from_rows `classmethod` ¶

from_rows(
    session: Timestamp,
    identifiers: ndarray,
    rows: ndarray,
    arrays: dict[str, ndarray],
) -> SessionPanel

Build a whole-session panel that turns a column into a dict only when that column is read.

A daily valuation reads the close and nothing else, so building every column's dict for every name every day was most of the read. identifiers and arrays are the frame's, one entry per row, and rows the session's row positions, as session_rows gives them.

from_market_frame `classmethod` ¶

from_market_frame(
    session: Timestamp, frame: DataFrame
) -> SessionPanel

Build a panel from what a multi-identifier market fetch returns.

That frame is MultiIndexed by (IDENTIFIER, DATE) and has already been filtered to the one date, so the date level is dropped rather than filtered again. A repeated (identifier, date) keeps its first row, as a single-name fetch followed by .iloc[0] does, so the panel and the read it replaces agree.

covers ¶

covers(identifiers: Iterable[str]) -> bool

Whether this panel answers for every one of identifiers: all of them, when it holds the whole session.

answers ¶

answers(identifier: str, date: str) -> bool

Whether this panel can answer for identifier on date.

Both halves, and nothing looser. A panel holds one session for the set of names it was built from; asked about another date or another name it says no, and the caller goes to the data. There is deliberately no nearest-session behaviour here: a panel that answered for another day would be a different question's answer wearing this one's key.

A panel built from the whole session (complete) answers for every name on its date: one it lacks has no row that day, and the data's own read would find none either.

Parameters:

Name	Type	Description	Default
`identifier`	`str`	The instrument.	required
`date`	`str`	The date as a `%Y-%m-%d` string, compared with `stamp`.	required

column ¶

column(name: str) -> dict[str, object]

Every instrument's value in one column, identifier to value.

The whole day in one call, for a caller valuing a book rather than a name. Values are as stored, NaN included, because deciding what a missing value means is the caller's business; DataFetcher.prices_on turns NaN into None. Empty for a column the panel does not hold, on the same terms as value.

value ¶

value(identifier: str, column: str) -> float | None

One instrument's value in column, or None where there is none.

None for an absent column and for a null value alike, on the same terms as :meth:~beacon.data.fetcher.DataFetcher.fetch_price: both mean "the data does not say", and a caller that has to tell them apart is asking a question about the store rather than about the instrument.

store ¶

A persisted market-data store on disk.

A store is a directory holding a manifest.json and one Parquet file per dataset: market.parquet (required), and reference.parquet, corporate_actions.parquet and features.parquet when the data has them. :func:save writes one from a DataFetcher and :func:load reads one back. A server started with python -m beacon.server loads the store at :func:default_path (or the path it is given), which is what python -m beacon.synthetic writes.

Parquet keeps each column's type, so a store reads back without parsing a date or a number, and packs smaller than gzipped CSV. A store saved by an earlier version, as gzipped CSV (schema version 1), is converted the first time it is loaded. schema_version is how the format changes without stranding anyone's store: a store written by a newer version is refused rather than misread.

Byte-for-byte reproducibility¶

Two runs of the same generator produce the same store. Parquet as written here carries no timestamps, the rows go into row groups of a fixed size, and the manifest's keys are written sorted.

Timestamps are deliberately not recorded in the manifest. The store's age is the file's mtime, and DataFetcher stamps its own refresh time when it loads. Writing a creation time into the manifest would be a third answer to a question that already has two, and would make every store byte-unique.

StoreManifest `dataclass` ¶

StoreManifest(
    schema_version: int,
    source: str,
    datasets: tuple[str, ...],
)

What a store says about itself.

Attributes:

Name	Type	Description
`schema_version`	`int`	On-disk shape, for migration on load.
`source`	`str`	Where the rows came from: "synthetic", "yfinance", "local", "imported", or whatever wrote it.
`datasets`	`tuple[str, ...]`	Which files are present, so a loader knows what to expect rather than probing for each one.

default_path ¶

default_path() -> Path

The store location a spawned server auto-loads.

The platform app-data directory, beside the documents DocumentStore keeps. Requires platformdirs; passing an explicit path anywhere in this module needs nothing beyond pandas, because only the default location is a platform question.

Returns:

Name	Type	Description
`Path`	`Path`	The directory, which may not exist.

size_on_disk ¶

size_on_disk(path: Path) -> int | None

Total bytes a store occupies, or None if it is not readable.

Every file in the directory, not just the ones this module writes: a store a future version adds a file to should report its real size rather than a figure that quietly excludes the new part.

dataset_size_on_disk ¶

dataset_size_on_disk(
    path: Path, dataset: str
) -> int | None

Bytes one dataset's file occupies.

Per dataset rather than the whole store, so a coverage pane showing a size against each row shows numbers that add up. Reporting the store total on every row would display the same figure once per dataset and make any sum of them wrong by that factor.

Returns:

Type	Description
`int \| None`	int \| None: The size, or None when the store does not hold that
`int \| None`	dataset ("there is no file" rather than "the file is empty"), when
`int \| None`	the dataset name is unknown, or when the file cannot be measured.
`int \| None`	Callers that sum these have to treat None as zero themselves, so a
`int \| None`	caller adding up sizes has to notice that a dataset is absent rather
`int \| None`	than have the absence quietly contribute a zero it never checked.

exists ¶

exists(path: Path) -> bool

Whether a readable store lives at this path.

A directory holding a manifest and market data. Market data is the floor: reference data and corporate actions are optional, but a store with no prices is not a data source, and reporting it as one would move the failure from startup to the first request.

read_manifest ¶

read_manifest(path: Path) -> StoreManifest

Read a store's manifest.

Raises:

Type	Description
`ConfigurationError`	If it is missing, unreadable, or written by a newer py-beacon than this one.

flatten_index ¶

flatten_index(frame: DataFrame) -> pd.DataFrame

Index levels back to columns, ready to write.

The containers differ in whether they kept their key columns: CorporateActions sets its index with drop=False so the columns survive, while MarketData and ReferenceData consume theirs. Dropping the index when its names are already columns avoids writing each key twice, which from_dataframe would then read back as a duplicate-column frame.

as_stored ¶

as_stored(frame: DataFrame, dataset: str) -> pd.DataFrame

frame with the types a store has always given back for it.

A store was gzipped CSV before py-beacon 0.9.6, and a load gave back what reading that text gave: numbers as float64 or int64, a date column no container parses as text, a blank string as blank. Parquet keeps whatever it is given, so without this a store saved now would load other types from the same data, and the API would show them.

outdated ¶

outdated(path: Path) -> str | None

Why the store at path is in a format this version no longer reads, or None. Reads only its market file's header.

A store saved before py-beacon 0.7.0 keeps exchange rates in a RATE column, which is refused since they moved to CLOSE.

save ¶

save(
    fetcher: DataFetcher,
    path: Path,
    source: str = SOURCE_LOCAL,
    progress: SaveProgress | None = None,
) -> Path

Write a fetcher's data to a store directory.

Parameters:

Name	Type	Description	Default
`fetcher`	`DataFetcher`	The data source to persist. Reference data, corporate actions and features are written only when present, so a market-only fetcher round-trips as a market-only store rather than as one carrying empty files.	required
`path`	`Path`	Directory to write into; created if absent.	required
`source`	`str`	Recorded in the manifest and reported by `/data/coverage`.	`SOURCE_LOCAL`
`progress`	`SaveProgress \| None`	Called as the files are written, with the share of the save done (by rows) and what is being saved.	`None`

Returns:

Name	Type	Description
`Path`	`Path`	The directory written.

save_features ¶

save_features(features: FeatureData, path: Path) -> None

Write a store's feature table, leaving its other datasets as they are.

Parameters:

Name	Type	Description	Default
`features`	`FeatureData`	The whole feature table, not just new rows: the file is replaced.	required
`path`	`Path`	The store directory. It must already hold a manifest.	required

Raises:

Type	Description
`ConfigurationError`	If path is not a Beacon data store.

load ¶

load(
    path: Path,
    fx_policy: str = DEFAULT_FX_POLICY,
    max_price_staleness_days: int | None = None,
    free_float_backfill_days: int = DEFAULT_FREE_FLOAT_BACKFILL_DAYS,
) -> DataFetcher

Read a store directory into a fetcher.

Parameters:

Name	Type	Description	Default
`path`	`Path`	Directory written by :func:`save`.	required
`fx_policy`	`str`	How the fetcher reads a rate on a day the pair printed none: see :data:`~beacon.data.fetcher.FX_CARRY_FORWARD` and :data:`~beacon.data.fetcher.FX_EXACT_DAY`. A modelling assumption rather than a store property, so it is chosen when the store is opened rather than written into it: the same rows converted under the other assumption are a legitimate second answer, and stamping one into the data would make it the only one.	`DEFAULT_FX_POLICY`
`max_price_staleness_days`	`int \| None`	How many calendar days a name may go untraded before it is dropped; None keeps everything.	`None`
`free_float_backfill_days`	`int`	How many calendar days back a blank free float may take the last reported value from. Chosen at open for the same reason as the two above.	`DEFAULT_FREE_FLOAT_BACKFILL_DAYS`

Returns:

Name	Type	Description
`DataFetcher`	`DataFetcher`	Serving whatever the store holds.

Raises:

Type	Description
`StoreFormatError`	If the store was saved in a format this version no longer reads (see `outdated`).
`ConfigurationError`	If the path is not a readable store, or its rows contradict each other (see `beacon.data.kinds`).
`ValueError`	If fx_policy is not a known policy, or either day count is out of range.

Returns¶

Each name's daily total return in one currency, counting splits and cash on their ex-dates.

returns ¶

Each name's daily total return, in one currency.

returns = name_returns(fetcher, ["AAA", "BBB"], "2024-01-02", "2024-06-28",
                       currency="USD")

A day's return is what a holder of one share made, as a share of the previous close, in currency:

(close × split ratio + cash paid per share) ÷ previous close − 1

Splits (and reverse splits and stock dividends) multiply the shares held, so the close on the day is scaled by the ratio rather than read as a fall.
Cash (dividends, special dividends and returns of capital) counts on its ex-date, gross of any tax withheld, converted at that day's rate. That is when a holder is owed it, and how a backtest's NAV counts it.
Mergers, spin-offs and rights issues are not a scaling of one price, so no return accounts for them, as an adjusted close leaves them out.
A cancelled action never happened and is left out.
A day with no close is empty; the next close's return runs from the last close before the gap, so nothing is lost across it.

name_returns ¶

name_returns(
    source: _Source,
    identifiers: Iterable[str],
    start: str | None = None,
    end: str | None = None,
    currency: str | None = None,
    actions: DataFrame | None = None,
    closes: DataFrame | None = None,
) -> pd.DataFrame

Daily total returns by date and name, in currency.

Parameters:

Name	Type	Description	Default
`source`	`_Source`	The data, usually a `DataFetcher`.	required
`identifiers`	`Iterable[str]`	The names. One without prices in the window is left out.	required
`start`	`str \| None`	First date, inclusive. Its returns are empty, having no close before them in the window.	`None`
`end`	`str \| None`	Last date, inclusive.	`None`
`currency`	`str \| None`	The currency returns are measured in; each name's own when None.	`None`
`actions`	`DataFrame \| None`	The corporate actions, with `IDENTIFIER`, `TYPE`, `VALUE` and `EX_DATE`; read from source's `corporate_actions` when None.	`None`
`closes`	`DataFrame \| None`	The names' closes in their own currencies, dates by name, for a caller that has read them already; read from source when None.	`None`

Returns:

Type	Description
`DataFrame`	pd.DataFrame: Dates by name, NaN where a name has no close or no
`DataFrame`	close before it.

Data¶

data ¶

MarketData ¶

data property ¶

identifiers property ¶

identifier_set property ¶

columns property ¶

sessions property ¶

date_range property ¶

from_dataframe classmethod ¶

drop_non_positive_prices ¶

last_session_on_or_before ¶

session_columns ¶

session_rows ¶

codes_for ¶

session_values ¶

last_values ¶

get ¶

ReferenceData ¶

data property ¶

identifiers property ¶

columns property ¶

from_dataframe classmethod ¶

get ¶

DataFetcher ¶

identifiers property ¶

fx_pairs property ¶

instrument_identifiers property ¶

tradable_identifiers property ¶

market_columns property ¶

reference_identifiers property ¶

reference_columns property ¶

date_range property ¶

corporate_actions property ¶

features property ¶

market property ¶

source property ¶

store_path property ¶

reference property ¶

with_settings ¶

is_tradable ¶

short_rate ¶

fetch_short_rates ¶

short_rate_on ¶

resolve_session ¶

fetch_feature ¶

fetch_features ¶

replace_features ¶

feature_types ¶

feature_fields ¶

record_origin ¶

delisting_dates ¶

record_refresh ¶

last_refreshed ¶

age_seconds ¶

merge_market_data ¶

merge_reference_data ¶

fetch_corporate_actions ¶

fetch_trailing_dividend ¶

fetch_trailing_dividend_yield ¶

fetch_market_data ¶

warm_session ¶

fetch_shares_outstanding ¶

fetch_free_float_factor ¶

prices_on ¶

price_array_on ¶

last_market_values ¶

fetch_price ¶

last_prices ¶

fetch_prices ¶

fetch_fx_rates ¶

fx_route ¶

fx_rate_on ¶

latest_rows ¶

last_priced_on ¶

stale_identifiers ¶

fx_rates_on ¶

fetch_reference_data ¶

fetch_classification ¶

fetch_classifications ¶

data `property` ¶

identifiers `property` ¶

identifier_set `property` ¶

columns `property` ¶

sessions `property` ¶

date_range `property` ¶

from_dataframe `classmethod` ¶

data `property` ¶

identifiers `property` ¶

columns `property` ¶

from_dataframe `classmethod` ¶

identifiers `property` ¶

fx_pairs `property` ¶

instrument_identifiers `property` ¶

tradable_identifiers `property` ¶

market_columns `property` ¶

reference_identifiers `property` ¶

reference_columns `property` ¶

date_range `property` ¶

corporate_actions `property` ¶

features `property` ¶

market `property` ¶

source `property` ¶

store_path `property` ¶

reference `property` ¶

What `ADJ_CLOSE` means here¶

data `property` ¶

identifiers `property` ¶

identifier_set `property` ¶

columns `property` ¶

sessions `property` ¶

date_range `property` ¶

from_dataframe `classmethod` ¶

data `property` ¶

identifiers `property` ¶

columns `property` ¶

from_dataframe `classmethod` ¶

data `property` ¶

identifiers `property` ¶

is_empty `property` ¶

from_dataframe `classmethod` ¶

empty `classmethod` ¶

data `property` ¶

is_empty `property` ¶

identifiers `property` ¶

columns `property` ¶

types `property` ¶

from_dataframe `classmethod` ¶

empty `classmethod` ¶