beacon.data

Market and reference data access: MarketData/ReferenceData wrap tabular sources, and DataFetcher provides the unified query interface used throughout the calculation and backtest layers.

data

The init.py for the 'data' module.

This module handles fetching, parsing, and providing financial data.

MarketData

MarketData(file_path: str, date_format: str = '%Y-%m-%d')

Time-series data container backed by a MultiIndex DataFrame.

The source file must contain at least IDENTIFIER and DATE columns. After loading the DataFrame is indexed on (IDENTIFIER, DATE) and sorted, enabling fast .loc slicing by identifier or list of identifiers.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/base.py

def __init__(self,
             file_path: str,
             date_format: str = "%Y-%m-%d"):
    df = _read_file(file_path)
    self._df = self._prepare(df, date_format)

data property

data: DataFrame

Return a copy of the underlying DataFrame.

identifiers property

identifiers: list[str]

Unique identifiers present in the dataset.

Cached, because this is not the cheap property it reads as. It scans the whole MultiIndex, and fetch_fx_rates consults it on every call to decide whether a pair exists -- which the calculator makes once per foreign holding per day. Against a single-currency universe that never fired; against a global one it turned an O(rows) scan into an inner loop, and an index over eighty names took longer than the entire rest of the test suite.

Keyed on the frame's identity rather than a flag, so replacing _df invalidates it automatically instead of relying on every future mutation remembering to.

columns property

columns: list[str]

Non-index column names.

Cached on the frame's identity, like identifiers and for the same reason (BN-214). _market_scalar asks column not in market.columns on every price read -- 186,400 times over a 200-name three-year run -- and this built a fresh list of strings each time to answer a membership test.

sessions property

sessions: DatetimeIndex

The distinct dates the dataset carries, ascending.

Cached on the frame's identity, for the reason the identifiers above are (BN-190). Materialising the DATE level is an O(rows) take over the whole frame, and date_range and last_session_on_or_before each did it on every call — which resolve_session makes once per name while a methodology walks a universe. On a 1,600-name preview that was 59% of the runtime spent re-deriving a constant: every name on a given date resolves to the same session, and the frame does not move underneath them.

date_range property

date_range: tuple[Timestamp, Timestamp]

(earliest, latest) timestamps in the dataset.

from_dataframe classmethod

from_dataframe(
    df: DataFrame, date_format: str = "%Y-%m-%d"
) -> MarketData

Create a MarketData instance from an existing DataFrame.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/base.py

@classmethod
def from_dataframe(cls,
                   df: pd.DataFrame,
                   date_format: str = "%Y-%m-%d") -> "MarketData":
    """Create a MarketData instance from an existing DataFrame."""
    instance = object.__new__(cls)
    instance._df = cls._prepare(df.copy(), date_format)
    return instance

last_session_on_or_before

last_session_on_or_before(
    date: str | Timestamp,
) -> pd.Timestamp | None

The latest date the dataset carries at or before date.

A date inside the coverage that has no rows is an ordinary closed market, and this is the session that was in force through it. None when the dataset begins after date, since then there is no earlier session to be in force.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/base.py

def last_session_on_or_before(self,
                              date: str | pd.Timestamp) -> pd.Timestamp | None:
    """The latest date the dataset carries at or before *date*.

    A date inside the coverage that has no rows is an ordinary closed
    market, and this is the session that was in force through it. None
    when the dataset begins after *date*, since then there is no earlier
    session to be in force.
    """
    sessions = self.sessions
    position = as_of_position(sessions, date)

    return None if position is None else pd.Timestamp(sessions[position])

session_columns

session_columns(
    date: str | Timestamp,
) -> tuple[dict[str, dict[str, object]], set[str]]

Every column's values for one session, keyed by identifier (BN-213).

The read the daily loops make eight hundred times a run, done without touching pandas. get() filters the whole frame to find one day's rows -- the frame is indexed (IDENTIFIER, DATE), so a single day's rows are scattered through it rather than adjacent, and finding them costs what the frame costs. Measured over 156,400 rows: 13.5 ms a day, which is most of an index calculation.

This uses :meth:_date_index instead: the row positions for a date are already known, so the read is a gather and two dict builds.

Returns:

Name	Type	Description
tuple	dict[str, dict[str, object]]	({column: {identifier: value}}, identifiers) -- exactly
	set[str]	what a :class:~beacon.data.session.SessionPanel holds, so no
	tuple[dict[str, dict[str, object]], set[str]]	DataFrame is built on the way. Empty for a date the data has no
	tuple[dict[str, dict[str, object]], set[str]]	rows on.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/base.py

def session_columns(self,
                    date: str | pd.Timestamp) -> tuple[dict[str, dict[str, object]], set[str]]:
    """Every column's values for one session, keyed by identifier (BN-213).

    The read the daily loops make eight hundred times a run, done without
    touching pandas. `get()` filters the whole frame to find one day's rows
    -- the frame is indexed `(IDENTIFIER, DATE)`, so a single day's rows are
    scattered through it rather than adjacent, and finding them costs what
    the frame costs. Measured over 156,400 rows: 13.5 ms a day, which is
    most of an index calculation.

    This uses :meth:`_date_index` instead: the row positions for a date are
    already known, so the read is a gather and two dict builds.

    Returns:
        tuple: ``({column: {identifier: value}}, identifiers)`` -- exactly
        what a :class:`~beacon.data.session.SessionPanel` holds, so no
        DataFrame is built on the way. Empty for a date the data has no
        rows on.
    """
    order, bounds, identifiers, arrays = self._date_index()
    span = bounds.get(pd.Timestamp(date))

    if span is None:
        return {}, set()

    rows = order[span[0]:span[1]]

    # Reversed so that a repeated (identifier, date) keeps its FIRST row:
    # `dict` takes the last value written for a key, so feeding the pairs
    # backwards makes the earliest win. That is what `get()` followed by
    # `.iloc[0]` already did, and a panel that disagreed with the read it
    # replaces would be a quieter bug than the slowness it fixes.
    names = identifiers[rows][::-1]

    values = {column: dict(zip(names, array[rows][::-1], strict=True))
              for column, array in arrays.items()}

    return values, {str(name) for name in names}

get

get(
    identifier: str | list[str],
    start_date: str | None = None,
    end_date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Return data for one or more identifiers, optionally filtered.

Parameters:

Name	Type	Description	Default
identifier	str | list[str]	Single identifier or list of identifiers.	required
start_date	str | None	Date string to slice the start of the date range.	None
end_date	str | None	Date string to slice the end of the date range.	None
columns	list[str] | None	Subset of columns to return.	None

Returns:

Name	Type	Description
	DataFrame	pd.DataFrame: Single identifier: indexed by DATE. List of
identifiers	DataFrame	MultiIndexed by (IDENTIFIER, DATE). Empty
	DataFrame	DataFrame if no matching data is found.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/base.py

def get(self,
        identifier: str | list[str],
        start_date: str | None = None,
        end_date: str | None = None,
        columns: list[str] | None = None) -> pd.DataFrame:
    """Return data for one or more identifiers, optionally filtered.

    Args:
        identifier: Single identifier or list of identifiers.
        start_date: Date string to slice the start of the date range.
        end_date: Date string to slice the end of the date range.
        columns: Subset of columns to return.

    Returns:
        pd.DataFrame: Single identifier: indexed by ``DATE``. List of
        identifiers: MultiIndexed by ``(IDENTIFIER, DATE)``. Empty
        DataFrame if no matching data is found.
    """
    if isinstance(identifier, list):
        existing = self._df.index.get_level_values("IDENTIFIER")
        identifier = [i for i in identifier if i in existing]
        if not identifier:
            return pd.DataFrame()

    try:
        subset = self._df.loc[identifier]
    except KeyError:
        return pd.DataFrame()

    if start_date is not None or end_date is not None:
        if isinstance(identifier, list):
            dates = subset.index.get_level_values("DATE")
        else:
            dates = subset.index

        mask = pd.Series(True, index=subset.index)
        if start_date is not None:
            mask &= dates >= pd.Timestamp(start_date)
        if end_date is not None:
            mask &= dates <= pd.Timestamp(end_date)
        subset = subset.loc[mask]

    if columns is not None:
        subset = subset[columns]

    return subset

ReferenceData

ReferenceData(file_path: str)

Reference data container with validity ranges.

The source file must contain IDENTIFIER, DATE_FROM, and DATE_TO columns. DATE_TO may be NaT to indicate a currently-active record.

Indexed on IDENTIFIER (non-unique, since an identifier may have multiple validity periods).

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/base.py

def __init__(self,
             file_path: str):
    df = _read_file(file_path)
    self._df = self._prepare(df)

data property

data: DataFrame

Return a copy of the underlying DataFrame.

identifiers property

identifiers: list[str]

Unique identifiers present in the dataset. Cached, as above.

columns property

columns: list[str]

Column names (including DATE_FROM, DATE_TO).

from_dataframe classmethod

from_dataframe(df: DataFrame) -> ReferenceData

Create a ReferenceData instance from an existing DataFrame.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/base.py

@classmethod
def from_dataframe(cls,
                   df: pd.DataFrame) -> "ReferenceData":
    """Create a ReferenceData instance from an existing DataFrame."""
    instance = object.__new__(cls)
    instance._df = cls._prepare(df.copy())
    return instance

get

get(
    identifier: str | list[str],
    date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Return reference data for one or more identifiers.

Parameters:

Name	Type	Description	Default
identifier	str | list[str]	Single identifier or list of identifiers.	required
date	str | None	Point-in-time date. If provided, only rows where DATE_FROM <= date and (DATE_TO >= date or DATE_TO is NaT) are returned.	None
columns	list[str] | None	Subset of columns to return.	None

Returns:

Type	Description
DataFrame	pd.DataFrame: Indexed by IDENTIFIER. Empty DataFrame if no
DataFrame	match.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/base.py

def get(self,
        identifier: str | list[str],
        date: str | None = None,
        columns: list[str] | None = None) -> pd.DataFrame:
    """Return reference data for one or more identifiers.

    Args:
        identifier: Single identifier or list of identifiers.
        date: Point-in-time date. If provided, only rows where
            ``DATE_FROM <= date`` and (``DATE_TO >= date`` or
            ``DATE_TO`` is NaT) are returned.
        columns: Subset of columns to return.

    Returns:
        pd.DataFrame: Indexed by ``IDENTIFIER``. Empty DataFrame if no
        match.
    """
    if isinstance(identifier, list):
        existing = self._df.index
        identifier = [i for i in identifier if i in existing]
        if not identifier:
            return pd.DataFrame()

    try:
        subset = self._df.loc[identifier]
    except KeyError:
        return pd.DataFrame()

    # .loc on a non-unique index with a single str returns a Series
    # if exactly one row matches — normalize to DataFrame.
    if isinstance(subset, pd.Series):
        subset = subset.to_frame().T
        subset.index.name = "IDENTIFIER"

    if date is not None:
        ts = pd.Timestamp(date)
        mask = subset["DATE_FROM"] <= ts
        mask &= subset["DATE_TO"].isna() | (subset["DATE_TO"] >= ts)
        subset = subset.loc[mask]

    if columns is not None:
        subset = subset[columns]

    return subset

DataFetcher

DataFetcher(
    market_data: MarketData,
    reference_data: ReferenceData | None = None,
    corporate_actions: CorporateActions | None = None,
    features: FeatureData | None = None,
    fx_policy: str = DEFAULT_FX_POLICY,
    max_price_staleness_days: int | None = None,
    free_float_backfill_days: int = DEFAULT_FREE_FLOAT_BACKFILL_DAYS,
)

Unified query interface over MarketData and ReferenceData.

Parameters:

Name	Type	Description	Default
market_data	MarketData	Time-series market data container.	required
reference_data	ReferenceData | None	Reference data container.	None
corporate_actions	CorporateActions | None	Action history. Absent means an empty history rather than None, so callers never have to check before asking — "this instrument paid nothing" and "we hold no action data" give the same answer to every question this class can be asked.	None

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def __init__(self,
             market_data: MarketData,
             reference_data: ReferenceData | None = None,
             corporate_actions: CorporateActions | None = None,
             features: FeatureData | None = None,
             fx_policy: str = DEFAULT_FX_POLICY,
             max_price_staleness_days: int | None = None,
             free_float_backfill_days: int = DEFAULT_FREE_FLOAT_BACKFILL_DAYS):
    if fx_policy not in FX_POLICIES:
        raise ValueError(
            f"Unknown fx_policy: {fx_policy!r}. "
            f"Supported values: {list(FX_POLICIES)}.")

    # One assumption for every conversion in the library (BN-207). Held
    # here rather than passed per call because it is a property of how this
    # dataset is being read, not of any one question asked of it -- and
    # because a per-call default is how five conversion sites came to
    # disagree in the first place.
    self.fx_policy = fx_policy

    if (max_price_staleness_days is not None
            and max_price_staleness_days < 1):
        raise ValueError(
            f"max_price_staleness_days must be at least 1 day, got "
            f"{max_price_staleness_days!r}. Pass None to keep every name "
            f"regardless of when it last traded.")

    # How long a name may go without trading before it stops being worth
    # holding (BN-211). None keeps everything, which is what this library
    # always did -- so adopting the setting changes no index and no
    # backtest until somebody asks it to.
    #
    # A modelling choice rather than a data property, and global rather
    # than per-index on Karan's call: one answer for the whole
    # installation, reaching index construction and backtests alike.
    self.max_price_staleness_days = max_price_staleness_days

    # How many days a free float carries forward over blank cells
    # (BN-219). The third global setting, on the same terms as the two
    # above: it changes numbers, so it is chosen once and published. See
    # `beacon.data.free_float` for why 90 and why there is no unlimited.
    self.free_float_backfill_days = validated_window(free_float_backfill_days)

    self._market = market_data
    self._reference = reference_data
    self._actions = (corporate_actions if corporate_actions is not None
                     else CorporateActions.empty())
    # Empty rather than None, on the same terms as the actions above: a
    # dataset without features is still a dataset, and callers should be
    # able to ask it what it holds without checking for None first.
    self._features = (features if features is not None
                      else FeatureData.empty())

    # Loading is a refresh. Stamping construction rather than leaving this
    # empty is what makes an age meaningful from the first request: a
    # freshly started server holds data that is genuinely seconds old, and
    # reporting "unknown" until someone happens to sync would be less true,
    # not more careful.
    now = datetime.now(UTC)
    self._refreshed: dict[str, datetime | None] = {
        MARKET_DATASET: now,
        REFERENCE_DATASET: now if reference_data is not None else None,
        ACTIONS_DATASET: now if not self._actions.is_empty else None,
        FEATURES_DATASET: now if not self._features.is_empty else None,
        # The pairs are market rows, so they are as fresh as the market
        # data is. A separate stamp would drift from it for no reason.
        FX_DATASET: now,
    }

    # Where this data was loaded from, stamped by whatever built the
    # fetcher. None for one assembled in-process: saying "local" would
    # claim a provenance it does not have.
    self._source: str | None = None
    self._store_path: Path | None = None

    # Rate series, one per ordered pair, behind `fx_rate_on`. A run
    # converts every foreign name on every day and each fetch slices the
    # whole market frame, so an uncached lookup made an eighty-name global
    # index take longer than the rest of the suite put together. Cleared
    # whenever the market data underneath it is replaced.
    self._fx_series: dict[tuple[str, str], pd.Series] = {}
    # How each cached pair was found: direct, inverse, or a cross (BN-235).
    self._fx_routes: dict[tuple[str, str], str | None] = {}
    # Each name's observed free floats, blanks dropped, read once when a
    # blank day first needs carrying over (BN-219). Cleared with the FX
    # series on a merge, for the same reason.
    self._free_float_history: dict[tuple[str, str], pd.Series] = {}

    # The session a methodology is currently walking a universe over, read
    # in one slice. One panel rather than a growing map of them: the reads
    # that repeat are the ones inside a single rebalance -- a selection
    # rule prices every name and the weighting then prices the survivors
    # again -- and they are over at the moment the date moves on (BN-190).
    # Cleared whenever the market data underneath it is replaced.
    self._session_panel: SessionPanel | None = None

identifiers property

identifiers: list[str]

Unique identifiers present in market data.

fx_pairs property

fx_pairs: list[str]

Currency pairs held in the market data.

A pair is stored as an ordinary market identifier named f"{from}{to}" (BN-128), so nothing in the frame separates it from an instrument except what it carries: RATE is populated on a pair and null on everything else. That is the discriminator, rather than a name pattern -- an instrument legitimately called EURUSD would be misfiled by a six-letter rule, and a store may hold pairs for currencies its reference data never mentions.

instrument_identifiers property

instrument_identifiers: list[str]

Market identifiers that are instruments rather than currency pairs.

What a universe or a search should offer: a pair is a rate series, not something anybody holds.

market_columns property

market_columns: list[str]

Column names in the market data.

reference_identifiers property

reference_identifiers: list[str] | None

Unique identifiers in the reference data, or None if not loaded.

reference_columns property

reference_columns: list[str] | None

Column names in the reference data, or None if not loaded.

date_range property

date_range: tuple[Timestamp, Timestamp]

(earliest, latest) timestamps in the market data.

corporate_actions property

corporate_actions: CorporateActions

The action history. Empty rather than None when none was loaded.

features property

features: FeatureData

The feature table. Empty rather than None when none was loaded.

Exposed for persistence and for discovery, on the same terms as market. Point-in-time reads go through fetch_features (BN-135), not through this.

market property

market: MarketData

The market-data container itself.

Exposed for persistence (beacon.data.store): writing a fetcher to disk means reading back everything it holds, and the summarising properties above cannot reconstruct a frame. Query through fetch_market_data instead — this is the whole dataset, not an answer to a question.

source property

source: str | None

Where this data was loaded from, or None if nothing recorded it.

Describes the load, not every row: a later sync merges rows from somewhere else without changing where the store came from. Modelling mixed provenance would need a source per row, which nothing asks for.

store_path property

store_path: Path | None

The store this was loaded from, if it came from one.

reference property

reference: ReferenceData | None

The reference-data container, or None if none was loaded.

Exposed for persistence, on the same terms as :attr:market.

resolve_session

resolve_session(
    date: str | Timestamp,
) -> pd.Timestamp | None

The market session date resolves to, backfilling inside the data.

A date the data has no bar for but which sits inside its coverage is a day the market was shut. The last session on or before it is the one that was actually in force through the closure — reading it is not an approximation, it is what the day was. Past the last bar nothing is known, so that answers None rather than a stale print wearing a current date; the bound is the data's own coverage rather than a day count, because no day count can tell a long closure from the unknown future.

Parameters:

Name	Type	Description	Default
date	str | Timestamp	The date asked about.	required

Returns:

Type	Description
Timestamp | None	pd.Timestamp | None: The session, or None when date falls
Timestamp | None	outside the data's coverage on either side.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def resolve_session(self,
                    date: str | pd.Timestamp) -> pd.Timestamp | None:
    """The market session *date* resolves to, backfilling inside the data.

    A date the data has no bar for but which sits inside its coverage is a
    day the market was shut. The last session on or before it is the one
    that was actually in force through the closure — reading it is not an
    approximation, it is what the day was. Past the last bar nothing is
    known, so that answers None rather than a stale print wearing a
    current date; the bound is the data's own coverage rather than a day
    count, because no day count can tell a long closure from the unknown
    future.

    Args:
        date: The date asked about.

    Returns:
        pd.Timestamp | None: The session, or None when *date* falls
        outside the data's coverage on either side.
    """
    as_of = pd.Timestamp(date)
    first, last = self.date_range

    if pd.isna(first) or as_of < first or as_of > last:
        return None

    return self._market.last_session_on_or_before(as_of)

fetch_feature

fetch_feature(
    identifier: str,
    field: str,
    date: str | Timestamp | None = None,
    feature_type: str | None = None,
    max_age_days: int | None = MAX_AGE_DAYS,
) -> float | None

One feature value, as it was knowable on a date.

The point-in-time read. A value published after date is invisible, which is what keeps a backtest from screening on numbers nobody had.

Returns:

Type	Description
float | None	float | None: The value, or None when nothing is knowable. An
float | None	instrument with no coverage is an ordinary answer, not an error —
float | None	most datasets cover most names most of the time and not all of
float | None	them all of it.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_feature(self,
                  identifier: str,
                  field: str,
                  date: str | pd.Timestamp | None = None,
                  feature_type: str | None = None,
                  max_age_days: int | None = MAX_AGE_DAYS) -> float | None:
    """One feature value, as it was knowable on a date.

    The point-in-time read. A value published after `date` is invisible,
    which is what keeps a backtest from screening on numbers nobody had.

    Returns:
        float | None: The value, or None when nothing is knowable. An
        instrument with no coverage is an ordinary answer, not an error —
        most datasets cover most names most of the time and not all of
        them all of it.
    """
    return self._features.value_as_of(identifier, field, date,
                                      feature_type, max_age_days)

fetch_features

fetch_features(
    identifiers: list[str],
    fields: list[str],
    date: str | Timestamp | None = None,
    feature_type: str | None = None,
    max_age_days: int | None = MAX_AGE_DAYS,
) -> dict[str, dict[str, float | None]]

Several features for several instruments, on one date.

The batch form, on the same argument the reference batch endpoint made: a client that has to fan out per name will, and moving the fan-out inside the server only relocates the cost.

Returns:

Name	Type	Description
dict	dict[str, dict[str, float | None]]	identifier -> field -> value. Every requested pair is
	dict[str, dict[str, float | None]]	present, null where nothing is knowable, so a caller reads a value
	dict[str, dict[str, float | None]]	rather than testing for a key.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_features(self,
                   identifiers: list[str],
                   fields: list[str],
                   date: str | pd.Timestamp | None = None,
                   feature_type: str | None = None,
                   max_age_days: int | None = MAX_AGE_DAYS
                   ) -> dict[str, dict[str, float | None]]:
    """Several features for several instruments, on one date.

    The batch form, on the same argument the reference batch endpoint
    made: a client that has to fan out per name will, and moving the
    fan-out inside the server only relocates the cost.

    Returns:
        dict: identifier -> field -> value. Every requested pair is
        present, null where nothing is knowable, so a caller reads a value
        rather than testing for a key.
    """
    return {identifier: {field: self.fetch_feature(identifier, field,
                                                   date, feature_type,
                                                   max_age_days)
                         for field in fields}
            for identifier in identifiers}

replace_features

replace_features(features: FeatureData) -> None

Swap the feature table, stamping the refresh.

The only mutating method on a fetcher, and it exists because an import has to land somewhere the next request can see. It replaces rather than edits: merged_with builds the new table, so a read in flight keeps the frame it started with instead of watching rows appear under it.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def replace_features(self,
                     features: FeatureData) -> None:
    """Swap the feature table, stamping the refresh.

    The only mutating method on a fetcher, and it exists because an import
    has to land somewhere the next request can see. It replaces rather
    than edits: `merged_with` builds the new table, so a read in flight
    keeps the frame it started with instead of watching rows appear under
    it.
    """
    self._features = features
    self._refreshed[FEATURES_DATASET] = datetime.now(UTC)

feature_types

feature_types() -> list[str]

Datasets the loaded features carry, for discovery.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def feature_types(self) -> list[str]:
    """Datasets the loaded features carry, for discovery."""
    return self._features.types

feature_fields

feature_fields(
    feature_type: str | None = None,
) -> list[str]

Fields the loaded features carry, optionally within one dataset.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def feature_fields(self,
                   feature_type: str | None = None) -> list[str]:
    """Fields the loaded features carry, optionally within one dataset."""
    return self._features.fields(feature_type)

record_origin

record_origin(
    source: str, path: Path | None = None
) -> None

Note where this fetcher's data was loaded from.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def record_origin(self,
                  source: str,
                  path: Path | None = None) -> None:
    """Note where this fetcher's data was loaded from."""
    self._source = source
    self._store_path = path

delisting_dates

delisting_dates() -> dict[str, pd.Timestamp]

The last date each identifier is listed, for those whose life ends.

Resolved in one pass rather than per identifier per day: an index over five thousand names and ten years would otherwise make twelve million point-in-time lookups to find a few hundred delistings.

A name is treated as still listed if any of its records is open-ended, which is checked before taking the maximum -- max over a column containing NaT would silently ignore the open record and retire a name that never left.

Returns:

Name	Type	Description
dict	dict[str, Timestamp]	identifier -> last listed date. Names that never leave are
	dict[str, Timestamp]	absent, so an empty mapping means a constant universe and callers
	dict[str, Timestamp]	can skip the work entirely.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def delisting_dates(self) -> dict[str, pd.Timestamp]:
    """The last date each identifier is listed, for those whose life ends.

    Resolved in one pass rather than per identifier per day: an index over
    five thousand names and ten years would otherwise make twelve million
    point-in-time lookups to find a few hundred delistings.

    A name is treated as still listed if *any* of its records is
    open-ended, which is checked before taking the maximum -- `max` over a
    column containing NaT would silently ignore the open record and retire
    a name that never left.

    Returns:
        dict: identifier -> last listed date. Names that never leave are
        absent, so an empty mapping means a constant universe and callers
        can skip the work entirely.
    """
    if self._reference is None:
        return {}

    frame = self._reference.data.reset_index()

    if not {"IDENTIFIER", "DATE_TO"} <= set(frame.columns):
        return {}

    ends: dict[str, pd.Timestamp] = {}

    for identifier, values in frame.groupby("IDENTIFIER")["DATE_TO"]:
        if values.isna().any():
            continue

        ends[str(identifier)] = pd.Timestamp(values.max())

    return ends

record_refresh

record_refresh(
    dataset: str, when: datetime | None = None
) -> None

Note that a dataset has just been refreshed.

Parameters:

Name	Type	Description	Default
dataset	str	MARKET_DATASET or REFERENCE_DATASET.	required
when	datetime | None	The moment. None uses now, which is what a real sync wants; tests pass an explicit time so an age can be asserted rather than approximated.	None

Raises:

Type	Description
ValueError	If the dataset is not one this fetcher holds.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def record_refresh(self,
                   dataset: str,
                   when: datetime | None = None) -> None:
    """Note that a dataset has just been refreshed.

    Args:
        dataset: MARKET_DATASET or REFERENCE_DATASET.
        when: The moment. None uses now, which is what a real sync wants;
            tests pass an explicit time so an age can be asserted rather
            than approximated.

    Raises:
        ValueError: If the dataset is not one this fetcher holds.
    """
    if dataset not in DATASETS:
        raise ValueError(
            f"unknown dataset '{dataset}'. Known: {', '.join(DATASETS)}.")

    self._refreshed[dataset] = when if when is not None else datetime.now(UTC)

last_refreshed

last_refreshed(dataset: str) -> datetime | None

When a dataset was last loaded or synced.

Returns:

Type	Description
datetime | None	datetime or None: The moment, or None when the dataset is not
datetime | None	loaded at all — which is a different statement from "loaded and
datetime | None	never refreshed" and should not be collapsed into it.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def last_refreshed(self,
                   dataset: str) -> datetime | None:
    """When a dataset was last loaded or synced.

    Returns:
        datetime or None: The moment, or None when the dataset is not
        loaded at all — which is a different statement from "loaded and
        never refreshed" and should not be collapsed into it.
    """
    if dataset not in DATASETS:
        raise ValueError(
            f"unknown dataset '{dataset}'. Known: {', '.join(DATASETS)}.")

    return self._refreshed[dataset]

age_seconds

age_seconds(
    dataset: str, now: datetime | None = None
) -> float | None

How long ago a dataset was last refreshed, in seconds.

Parameters:

Name	Type	Description	Default
dataset	str	Which dataset.	required
now	datetime | None	The reference moment, for tests.	None

Returns:

Type	Description
float | None	float or None: The age, or None when the dataset is not loaded.
float | None	Never negative: a clock adjustment between the two readings would
float | None	otherwise report data refreshed in the future, which is noise
float | None	rather than information.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def age_seconds(self,
                dataset: str,
                now: datetime | None = None) -> float | None:
    """How long ago a dataset was last refreshed, in seconds.

    Args:
        dataset: Which dataset.
        now: The reference moment, for tests.

    Returns:
        float or None: The age, or None when the dataset is not loaded.
        Never negative: a clock adjustment between the two readings would
        otherwise report data refreshed in the future, which is noise
        rather than information.
    """
    stamped = self.last_refreshed(dataset)
    if stamped is None:
        return None

    elapsed = ((now if now is not None else datetime.now(UTC)) - stamped)

    return max(elapsed.total_seconds(), 0.0)

merge_market_data

merge_market_data(frame: DataFrame) -> int

Fold freshly ingested rows into the market data.

Newly fetched rows win where they overlap an existing identifier and date. A re-sync of a window is a correction — a restated close, a backfilled volume — so keeping the older value would make the sync pointless.

The swap at the end is a single assignment, so a reader either sees the whole old dataset or the whole new one. This process is single-threaded and cooperatively scheduled, so there is no torn state to guard against; a reader that started before the swap simply finishes against the data it began with.

Parameters:

Name	Type	Description	Default
frame	DataFrame	Long-form rows carrying IDENTIFIER and DATE.	required

Returns:

Name	Type	Description
int	int	Rows added, counting only genuinely new identifier/date pairs
	int	— a re-sync that restates existing rows returns 0, which is the
	int	truthful answer to "how much did this add".

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def merge_market_data(self,
                      frame: pd.DataFrame) -> int:
    """Fold freshly ingested rows into the market data.

    Newly fetched rows win where they overlap an existing identifier and
    date. A re-sync of a window is a correction — a restated close, a
    backfilled volume — so keeping the older value would make the sync
    pointless.

    The swap at the end is a single assignment, so a reader either sees the
    whole old dataset or the whole new one. This process is single-threaded
    and cooperatively scheduled, so there is no torn state to guard
    against; a reader that started before the swap simply finishes against
    the data it began with.

    Args:
        frame: Long-form rows carrying ``IDENTIFIER`` and ``DATE``.

    Returns:
        int: Rows added, counting only genuinely new identifier/date pairs
        — a re-sync that restates existing rows returns 0, which is the
        truthful answer to "how much did this add".
    """
    if frame.empty:
        return 0

    existing = self._market.data.reset_index()
    combined = pd.concat([existing, frame], ignore_index=True)
    combined["DATE"] = pd.to_datetime(combined["DATE"])

    before = len(existing)
    combined = combined.drop_duplicates(subset=["IDENTIFIER", "DATE"],
                                        keep="last")

    self._market = MarketData.from_dataframe(combined)
    # The pairs are market rows, so a merge can add or restate them; a
    # cache held over the swap would answer out of the old frame. The
    # session panel is the same story one day wide.
    self._fx_series.clear()
    self._fx_routes.clear()
    self._free_float_history.clear()
    self._session_panel = None
    self.record_refresh(MARKET_DATASET)

    return len(combined) - before

merge_reference_data

merge_reference_data(frame: DataFrame) -> int

Fold freshly ingested reference records in.

Parameters:

Name	Type	Description	Default
frame	DataFrame	Rows carrying IDENTIFIER and DATE_FROM.	required

Returns:

Name	Type	Description
int	int	Records added.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def merge_reference_data(self,
                         frame: pd.DataFrame) -> int:
    """Fold freshly ingested reference records in.

    Args:
        frame: Rows carrying ``IDENTIFIER`` and ``DATE_FROM``.

    Returns:
        int: Records added.
    """
    if frame.empty:
        return 0

    if self._reference is None:
        self._reference = ReferenceData.from_dataframe(frame)
        self.record_refresh(REFERENCE_DATASET)

        return len(frame)

    existing = self._reference.data.reset_index()
    combined = pd.concat([existing, frame], ignore_index=True)

    before = len(existing)
    combined = combined.drop_duplicates(subset=["IDENTIFIER", "DATE_FROM"],
                                        keep="last")

    self._reference = ReferenceData.from_dataframe(combined)
    self.record_refresh(REFERENCE_DATASET)

    return len(combined) - before

fetch_corporate_actions

fetch_corporate_actions(
    identifier: str,
    start_date: str | Timestamp | None = None,
    end_date: str | Timestamp | None = None,
    types: list[str] | None = None,
) -> pd.DataFrame

Corporate actions for one identifier over a window.

Parameters:

Name	Type	Description	Default
identifier	str	The instrument.	required
start_date	str | Timestamp | None	Earliest ex-date, inclusive.	None
end_date	str | Timestamp | None	Latest ex-date, inclusive.	None
types	list[str] | None	Restrict to these action types.	None

Returns:

Type	Description
DataFrame	pd.DataFrame: Matching actions, oldest first; empty when there are
DataFrame	none.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_corporate_actions(self,
                            identifier: str,
                            start_date: str | pd.Timestamp | None = None,
                            end_date: str | pd.Timestamp | None = None,
                            types: list[str] | None = None) -> pd.DataFrame:
    """Corporate actions for one identifier over a window.

    Args:
        identifier: The instrument.
        start_date: Earliest ex-date, inclusive.
        end_date: Latest ex-date, inclusive.
        types: Restrict to these action types.

    Returns:
        pd.DataFrame: Matching actions, oldest first; empty when there are
        none.
    """
    return self._actions.get(identifier, start_date, end_date, types)

fetch_trailing_dividend

fetch_trailing_dividend(
    identifier: str, as_of: str | Timestamp
) -> float

Ordinary dividends per share over the trailing twelve months.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_trailing_dividend(self,
                            identifier: str,
                            as_of: str | pd.Timestamp) -> float:
    """Ordinary dividends per share over the trailing twelve months."""
    return self._actions.trailing_dividend(identifier, as_of)

fetch_trailing_dividend_yield

fetch_trailing_dividend_yield(
    identifier: str,
    as_of: str | Timestamp,
    price: float | None = None,
) -> float | None

Trailing dividend yield, priced off the market data by default.

Parameters:

Name	Type	Description	Default
identifier	str	The instrument.	required
as_of	str | Timestamp	End of the trailing window.	required
price	float | None	Price to divide by. None reads the close on or before as_of from the market data.	None

Returns:

Type	Description
float | None	float or None: The yield, or None when no price is available — a
float | None	missing price is a reason to say nothing rather than to guess.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_trailing_dividend_yield(self,
                                  identifier: str,
                                  as_of: str | pd.Timestamp,
                                  price: float | None = None) -> float | None:
    """Trailing dividend yield, priced off the market data by default.

    Args:
        identifier: The instrument.
        as_of: End of the trailing window.
        price: Price to divide by. None reads the close on or before
            *as_of* from the market data.

    Returns:
        float or None: The yield, or None when no price is available — a
        missing price is a reason to say nothing rather than to guess.
    """
    if price is None:
        price = self._close_on_or_before(identifier, as_of)

    if price is None or price <= 0.0:
        return None

    return self._actions.trailing_dividend_yield(identifier, as_of, price)

fetch_market_data

fetch_market_data(
    identifier: str | list[str],
    start_date: str | None = None,
    end_date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Fetch time-series market data for one or more identifiers.

Parameters:

Name	Type	Description	Default
identifier	str | list[str]	One identifier or a list of identifiers.	required
start_date	str | None	Date string to filter the start of the date range.	None
end_date	str | None	Date string to filter the end of the date range.	None
columns	list[str] | None	Subset of columns to return.	None

Returns:

Name	Type	Description
	DataFrame	pd.DataFrame: Single identifier: indexed by DATE. Multiple
identifiers	DataFrame	MultiIndexed by (IDENTIFIER, DATE). Empty
	DataFrame	DataFrame if no matching data is found.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_market_data(self,
                      identifier: str | list[str],
                      start_date: str | None = None,
                      end_date: str | None = None,
                      columns: list[str] | None = None) -> pd.DataFrame:
    """Fetch time-series market data for one or more identifiers.

    Args:
        identifier: One identifier or a list of identifiers.
        start_date: Date string to filter the start of the date range.
        end_date: Date string to filter the end of the date range.
        columns: Subset of columns to return.

    Returns:
        pd.DataFrame: Single identifier: indexed by ``DATE``. Multiple
        identifiers: MultiIndexed by ``(IDENTIFIER, DATE)``. Empty
        DataFrame if no matching data is found.
    """
    return self._market.get(identifier, start_date, end_date, columns)

warm_session

warm_session(
    identifiers: list[str], date: str | Timestamp
) -> None

Read one session's rows for identifiers in a single slice.

A hint, not a contract: every read this serves answers identically without it, only slower. What it removes is the shape a methodology walking a universe otherwise has — one frame slice per name per column, each costing what the whole frame costs rather than what one row does, which is why a preview's cost per name climbed with the size of its universe (BN-190).

It also ends the duplication that made the same names priced twice in one rebalance. A selection rule prices every candidate; the weighting scheme then prices the survivors, the same names on the same day. The second warm is a subset of the first, so it keeps the panel rather than rebuilding it, and the reads that follow are free.

Nothing goes stale under it: the panel answers only for the identifiers it was built with, only on its own session, and a merge clears it. Calling this with a different session or a name it does not hold replaces it, so the caller never has to say when it is done.

Parameters:

Name	Type	Description	Default
identifiers	list[str]	The instruments about to be read one at a time.	required
date	str | Timestamp	The session they will be read on. Resolve it first — :meth:resolve_session — since a panel for a closed day holds nothing and every read would fall back to the frame.	required

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def warm_session(self,
                 identifiers: list[str],
                 date: str | pd.Timestamp) -> None:
    """Read one session's rows for *identifiers* in a single slice.

    A hint, not a contract: every read this serves answers identically
    without it, only slower. What it removes is the shape a methodology
    walking a universe otherwise has — one frame slice per name per column,
    each costing what the whole frame costs rather than what one row does,
    which is why a preview's cost per name climbed with the size of its
    universe (BN-190).

    It also ends the duplication that made the same names priced twice in
    one rebalance. A selection rule prices every candidate; the weighting
    scheme then prices the survivors, the same names on the same day. The
    second warm is a subset of the first, so it keeps the panel rather than
    rebuilding it, and the reads that follow are free.

    Nothing goes stale under it: the panel answers only for the identifiers
    it was built with, only on its own session, and a merge clears it.
    Calling this with a different session or a name it does not hold
    replaces it, so the caller never has to say when it is done.

    Args:
        identifiers: The instruments about to be read one at a time.
        date: The session they will be read on. Resolve it first —
            :meth:`resolve_session` — since a panel for a closed day holds
            nothing and every read would fall back to the frame.
    """
    session = pd.Timestamp(date)
    held = self._session_panel

    if held is not None and held.session == session and held.covers(identifiers):
        return

    # Straight from the column index, with no DataFrame built on the way
    # (BN-213). `get` filtered the whole frame to find one day's rows,
    # which is 13.5 ms against 0.017.
    values, present = self._market.session_columns(session)

    self._session_panel = SessionPanel.from_columns(session, values, present)

fetch_shares_outstanding

fetch_shares_outstanding(
    identifier: str,
    date: str,
    column: str = "SHARES_OUTSTANDING",
) -> float | None

Return shares outstanding for identifier on date.

Sourced from the column market-data field. Returns None if the column is not present or there is no value on that date.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_shares_outstanding(self,
                             identifier: str,
                             date: str,
                             column: str = "SHARES_OUTSTANDING") -> float | None:
    """Return shares outstanding for *identifier* on *date*.

    Sourced from the *column* market-data field. Returns ``None`` if the
    column is not present or there is no value on that date.
    """
    return self._market_scalar(identifier, date, column)

fetch_free_float_factor

fetch_free_float_factor(
    identifier: str, date: str, column: str = "FREE_FLOAT"
) -> float | None

Return the free-float factor in force for identifier on date.

That day's value when there is one. Otherwise the last value before it, if no older than free_float_backfill_days (BN-219): free float moves on corporate events and reviews, so a blank cell means nothing was reported, not that the float changed. Never a later value.

Returns None if the column is absent, or nothing was reported within the window. Callers refuse through :func:~beacon.data.free_float.require_free_float rather than choosing a fallback of their own.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_free_float_factor(self,
                            identifier: str,
                            date: str,
                            column: str = "FREE_FLOAT") -> float | None:
    """Return the free-float factor in force for *identifier* on *date*.

    That day's value when there is one. Otherwise the last value before
    it, if no older than `free_float_backfill_days` (BN-219): free float
    moves on corporate events and reviews, so a blank cell means nothing
    was reported, not that the float changed. Never a later value.

    Returns ``None`` if the column is absent, or nothing was reported
    within the window. Callers refuse through
    :func:`~beacon.data.free_float.require_free_float` rather than
    choosing a fallback of their own.
    """
    today = self._market_scalar(identifier, date, column)

    if today is not None or self.free_float_backfill_days == 0:
        return today

    return carried_forward(self._free_float_series(identifier, column),
                           pd.Timestamp(date),
                           self.free_float_backfill_days)

prices_on

prices_on(
    identifiers: list[str], date: str, column: str = "CLOSE"
) -> dict[str, float | None]

:meth:fetch_price for many names on one day, in one read.

The batch face of the same answer (BN-218): warm the day's page once, take the whole column from it, and convert NaN to None -- exactly what fetch_price returns name by name. It exists because the daily valuation asks the question 200 times a day, and every name paid for a chain of four calls to reach a dict lookup.

A name the page does not answer for goes through fetch_price rather than being assumed absent, so a panel that somehow missed it costs a slower read instead of a wrong one.

Returns:

Name	Type	Description
dict	dict[str, float | None]	identifier -> price, or None where there is none.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def prices_on(self,
              identifiers: list[str],
              date: str,
              column: str = "CLOSE") -> dict[str, float | None]:
    """:meth:`fetch_price` for many names on one day, in one read.

    The batch face of the same answer (BN-218): warm the day's page once,
    take the whole column from it, and convert NaN to None -- exactly what
    `fetch_price` returns name by name. It exists because the daily
    valuation asks the question 200 times a day, and every name paid for
    a chain of four calls to reach a dict lookup.

    A name the page does not answer for goes through `fetch_price` rather
    than being assumed absent, so a panel that somehow missed it costs a
    slower read instead of a wrong one.

    Returns:
        dict: identifier -> price, or None where there is none.
    """
    # Through `getattr`, like every other use of the hint: warming is an
    # optimisation a provider may lack, and a batch read must still answer
    # without it -- one name at a time, as `fetch_price` always did.
    warm = getattr(self, "warm_session", None)

    if callable(warm):
        warm(identifiers, date)

    panel = self._session_panel

    if panel is None or panel.stamp != date:
        return {name: self.fetch_price(name, date, column)
                for name in identifiers}

    stored = panel.column(column)
    prices: dict[str, float | None] = {}

    for name in identifiers:
        if not panel.answers(name, date):
            prices[name] = self.fetch_price(name, date, column)
            continue

        value = stored.get(name)
        prices[name] = None if value is None or pd.isna(value) else float(value)  # type: ignore[arg-type]

    return prices

fetch_price

fetch_price(
    identifier: str, date: str, column: str = "CLOSE"
) -> float | None

Return identifier's price on date, or None if it did not print.

The scalar form of the single-day, single-name fetch a methodology makes for every name in a universe. It reads the same value fetch_market_data(identifier, date, date) does — the same column of the same row — and returns it rather than a one-row frame to slice, which is what lets a warmed session serve it (BN-190).

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_price(self,
                identifier: str,
                date: str,
                column: str = "CLOSE") -> float | None:
    """Return *identifier*'s price on *date*, or None if it did not print.

    The scalar form of the single-day, single-name fetch a methodology
    makes for every name in a universe. It reads the same value
    ``fetch_market_data(identifier, date, date)`` does — the same column of
    the same row — and returns it rather than a one-row frame to slice,
    which is what lets a warmed session serve it (BN-190).
    """
    return self._market_scalar(identifier, date, column)

fetch_fx_rates

fetch_fx_rates(
    from_currency: str,
    to_currency: str,
    start_date: str | None = None,
    end_date: str | None = None,
    column: str = "RATE",
) -> pd.Series

Return the FX rate series converting from_currency into to_currency.

The pair is looked up as a market-data identifier named f"{from_currency}{to_currency}" (upper-cased). The column field is used if present, otherwise the first data column. Returns an empty Series if the pair is not found.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_fx_rates(self,
                   from_currency: str,
                   to_currency: str,
                   start_date: str | None = None,
                   end_date: str | None = None,
                   column: str = "RATE") -> pd.Series:
    """Return the FX rate series converting *from_currency* into *to_currency*.

    The pair is looked up as a market-data identifier named
    ``f"{from_currency}{to_currency}"`` (upper-cased). The *column* field is
    used if present, otherwise the first data column. Returns an empty
    Series if the pair is not found.
    """
    pair = f"{from_currency}{to_currency}".upper()
    if pair not in self._market.identifiers:
        return pd.Series(dtype=float)
    df = self._market.get(pair, start_date, end_date)
    if df.empty:
        return pd.Series(dtype=float)
    rate_col = column if column in df.columns else df.columns[0]
    return df[rate_col]

fx_route

fx_route(
    from_currency: str, to_currency: str
) -> str | None

How a rate for this pair is found: direct, inverse or a cross.

Returns:

Type	Description
str | None	str | None: "direct" for a stored pair, "inverse" for one over the
str | None	stored reverse pair, "cross via USD" for a rate built from two
str | None	legs, "same currency" when no conversion is needed, and None when
str | None	no rate can be found.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fx_route(self,
             from_currency: str,
             to_currency: str) -> str | None:
    """How a rate for this pair is found: direct, inverse or a cross.

    Returns:
        str | None: "direct" for a stored pair, "inverse" for one over the
        stored reverse pair, "cross via USD" for a rate built from two
        legs, "same currency" when no conversion is needed, and None when
        no rate can be found.
    """
    if from_currency.upper() == to_currency.upper():
        return "same currency"

    pair = (from_currency.upper(), to_currency.upper())
    self._rate_series(*pair)

    return self._fx_routes[pair]

fx_rate_on

fx_rate_on(
    from_currency: str,
    to_currency: str,
    date: str | Timestamp,
) -> float | None

The rate converting from_currency into to_currency on date.

The single currency conversion in the library (BN-188). There were three, and they disagreed: the index calculator carried a rate forward and refused when the pair was unknown, the reference endpoint substituted 1.0 and reported the local number under a dollar heading, and the market-cap weighting did not convert at all — which is how a yen name came to carry fifteen times the weight it should. One lookup means the number displayed and the number weighted by are the same quantity, which is the half of this that nothing was checking.

The series is fetched once per ordered pair and cached, because a run asks this on every foreign name on every day and each fetch slices the whole market frame.

Parameters:

Name	Type	Description	Default
from_currency	str	The currency being converted out of.	required
to_currency	str	The currency being converted into.	required
date	str | Timestamp	The date the rate is wanted on.	required

Returns:

Name	Type	Description
	float | None	float | None: The rate in force on date, carried forward over
	float | None	gaps, or None when the pair is unknown **or when its history
	float | None	begins after date**. Callers treat None as "cannot convert"
	float | None	rather than as a rate of one. Nothing here invents parity on a
	float | None	caller's behalf: a rate of 1.0 is a claim about two currencies,
	float | None	and the only one this makes is that a currency converts into
	float | None	itself.
	float | None	Carried forward only. A date before the series starts used to
	float | None	answer with the series' first rate — a rate dated after the day it
	float | None	was applied to, which is look-ahead (BN-204). resolve_session
	float | None	states the same rule for sessions and is where the wording comes
from	float | None	there is no earlier observation to be in force, so there is
	float | None	no answer rather than a substitute for one.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fx_rate_on(self,
               from_currency: str,
               to_currency: str,
               date: str | pd.Timestamp) -> float | None:
    """The rate converting *from_currency* into *to_currency* on *date*.

    The single currency conversion in the library (BN-188). There were
    three, and they disagreed: the index calculator carried a rate forward
    and refused when the pair was unknown, the reference endpoint
    substituted 1.0 and reported the local number under a dollar heading,
    and the market-cap weighting did not convert at all — which is how a
    yen name came to carry fifteen times the weight it should. One lookup
    means the number displayed and the number weighted by are the same
    quantity, which is the half of this that nothing was checking.

    The series is fetched once per ordered pair and cached, because a run
    asks this on every foreign name on every day and each fetch slices the
    whole market frame.

    Args:
        from_currency: The currency being converted out of.
        to_currency: The currency being converted into.
        date: The date the rate is wanted on.

    Returns:
        float | None: The rate in force on *date*, carried forward over
        gaps, or None when the pair is unknown **or when its history
        begins after *date***. Callers treat None as "cannot convert"
        rather than as a rate of one. Nothing here invents parity on a
        caller's behalf: a rate of 1.0 is a claim about two currencies,
        and the only one this makes is that a currency converts into
        itself.

        Carried **forward** only. A date before the series starts used to
        answer with the series' first rate — a rate dated after the day it
        was applied to, which is look-ahead (BN-204). `resolve_session`
        states the same rule for sessions and is where the wording comes
        from: there is no earlier observation to be in force, so there is
        no answer rather than a substitute for one.
    """
    if from_currency.upper() == to_currency.upper():
        return 1.0

    pair = (from_currency.upper(), to_currency.upper())

    series = self._rate_series(*pair)

    if series.empty:
        return None

    stamp = pd.Timestamp(date)

    if self.fx_policy == FX_EXACT_DAY:
        # No carry at all: the rate must have printed on this very day.
        # `.get` rather than a search, because "the rate for this date" is
        # a lookup under this policy rather than a question about ordering.
        value = series.get(stamp)

        return None if value is None or pd.isna(value) else float(value)

    # Through `as_of_position` rather than a local search (BN-208): it
    # returns None where the raw form returns -1, and -1 is a legal pandas
    # index meaning the *last* element. That collision is what put a March
    # rate on a January valuation here, twice.
    position = as_of_position(series.index, stamp)

    return None if position is None else float(series.iloc[position])

latest_rows

latest_rows(
    identifiers: list[str],
    as_of: Timestamp,
    columns: list[str] | None = None,
    recent_days: int = RECENT_DAYS,
) -> pd.DataFrame

One row per name: its most recent bar at or before as_of.

The batch "what is the last thing we know about these names" read, shared by the reference endpoint and the staleness gate rather than written twice (BN-211).

Two stages, because the obvious version is thirty times slower. Measured over 500 names and ten years of daily bars: reading the recent window costs 650 ms and reading the whole history costs 20.6 seconds. The fetch is not what differs -- identifier selection dominates it either way -- it is that every per-name slice afterwards then cuts a 1.37-million-row frame. So the recent window is read first and answers almost every name, and only the stragglers are read again without a lower bound.

Then the frame is reduced to one row per name once, with a grouped tail, rather than sliced per name downstream. That is what makes even an all-stale store cheap: 782 ms against 19.8 seconds.

Parameters:

Name	Type	Description	Default
identifiers	list[str]	Names to look up.	required
as_of	Timestamp	The date to look back from, inclusive.	required
columns	list[str] | None	Columns to read, or None for all of them.	None
recent_days	int	How far back the cheap first stage reaches.	RECENT_DAYS

Returns:

Type	Description
DataFrame	pd.DataFrame: MultiIndexed as the market data is, holding at most
DataFrame	one row per identifier. Names with no bar at or before as_of are
DataFrame	absent rather than present-and-empty.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def latest_rows(self,
                identifiers: list[str],
                as_of: pd.Timestamp,
                columns: list[str] | None = None,
                recent_days: int = RECENT_DAYS) -> pd.DataFrame:
    """One row per name: its most recent bar at or before *as_of*.

    The batch "what is the last thing we know about these names" read,
    shared by the reference endpoint and the staleness gate rather than
    written twice (BN-211).

    **Two stages, because the obvious version is thirty times slower.**
    Measured over 500 names and ten years of daily bars: reading the
    recent window costs 650 ms and reading the whole history costs 20.6
    seconds. The fetch is not what differs -- identifier selection
    dominates it either way -- it is that every per-name slice afterwards
    then cuts a 1.37-million-row frame. So the recent window is read
    first and answers almost every name, and only the stragglers are read
    again without a lower bound.

    Then the frame is reduced to one row per name **once**, with a grouped
    tail, rather than sliced per name downstream. That is what makes even
    an all-stale store cheap: 782 ms against 19.8 seconds.

    Args:
        identifiers: Names to look up.
        as_of: The date to look back from, inclusive.
        columns: Columns to read, or None for all of them.
        recent_days: How far back the cheap first stage reaches.

    Returns:
        pd.DataFrame: MultiIndexed as the market data is, holding at most
        one row per identifier. Names with no bar at or before *as_of* are
        absent rather than present-and-empty.
    """
    end_str = pd.Timestamp(as_of).strftime("%Y-%m-%d")
    recent = (pd.Timestamp(as_of)
              - pd.DateOffset(days=recent_days)).strftime("%Y-%m-%d")

    frame = _last_row_each(
        self.fetch_market_data(identifiers, recent, end_str, columns))
    seen = _identifiers_in(frame)
    missing = [name for name in identifiers if name not in seen]

    if not missing:
        return frame

    older = _last_row_each(
        self.fetch_market_data(missing, None, end_str, columns))

    if older.empty:
        return frame

    if frame.empty:
        return older

    return pd.concat([frame, older]).sort_index()

last_priced_on

last_priced_on(
    identifiers: list[str], as_of: Timestamp
) -> dict[str, pd.Timestamp]

When each name last printed a bar at or before as_of.

Parameters:

Name	Type	Description	Default
identifiers	list[str]	Names to look up.	required
as_of	Timestamp	The date to look back from.	required

Returns:

Name	Type	Description
dict	dict[str, Timestamp]	identifier -> the date of its last bar. A name with no bar
	dict[str, Timestamp]	at all is absent from the mapping, which is a different thing from
	dict[str, Timestamp]	one whose last bar is old.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def last_priced_on(self,
                   identifiers: list[str],
                   as_of: pd.Timestamp) -> dict[str, pd.Timestamp]:
    """When each name last printed a bar at or before *as_of*.

    Args:
        identifiers: Names to look up.
        as_of: The date to look back from.

    Returns:
        dict: identifier -> the date of its last bar. A name with no bar
        at all is absent from the mapping, which is a different thing from
        one whose last bar is old.
    """
    frame = self.latest_rows(identifiers, as_of)

    if frame.empty or not isinstance(frame.index, pd.MultiIndex):
        return {}

    names = frame.index.get_level_values("IDENTIFIER")
    dates = frame.index.get_level_values("DATE")

    return {str(name): pd.Timestamp(date)
            for name, date in zip(names, dates, strict=True)}

stale_identifiers

stale_identifiers(
    identifiers: list[str], as_of: Timestamp
) -> set[str]

Which names have not traded recently enough to be worth holding.

Empty when no threshold is set, which is the default: staleness is something an installation opts into, and until it does this costs one comparison and reads nothing (BN-211).

A name with no price at all is not reported here. That is a different condition with a different remedy -- the weighting already refuses it by name -- and folding the two together would quietly excuse a missing instrument as a quiet one.

Parameters:

Name	Type	Description	Default
identifiers	list[str]	Names to test.	required
as_of	Timestamp	The date staleness is measured from.	required

Returns:

Name	Type	Description
set	set[str]	Identifiers whose last bar is older than the threshold.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def stale_identifiers(self,
                      identifiers: list[str],
                      as_of: pd.Timestamp) -> set[str]:
    """Which names have not traded recently enough to be worth holding.

    Empty when no threshold is set, which is the default: staleness is
    something an installation opts into, and until it does this costs one
    comparison and reads nothing (BN-211).

    A name with **no** price at all is not reported here. That is a
    different condition with a different remedy -- the weighting already
    refuses it by name -- and folding the two together would quietly
    excuse a missing instrument as a quiet one.

    Args:
        identifiers: Names to test.
        as_of: The date staleness is measured from.

    Returns:
        set: Identifiers whose last bar is older than the threshold.
    """
    if self.max_price_staleness_days is None:
        return set()

    stamp = pd.Timestamp(as_of)
    priced = self.last_priced_on(identifiers, stamp)

    return {name for name, date in priced.items()
            if (stamp - date).days > self.max_price_staleness_days}

fx_rates_on

fx_rates_on(
    from_currency: str, to_currency: str, days: Index
) -> pd.Series | None

:meth:fx_rate_on over many days at once, as a Series.

The vectorised face of the same rule, for a caller that needs a rate for every day of a run rather than one date (BN-207). Chained levels want exactly that, and asking per day would be one search per day per currency where a single reindex answers the lot.

It exists so that sharing the rule does not force one call shape on every caller: the policy, the carry semantics and the meaning of "no rate" are decided here once, and the two methods differ only in how many answers they return. Two implementations of the lookup is how the library came to have five of them.

Parameters:

Name	Type	Description	Default
from_currency	str	The currency being converted out of.	required
to_currency	str	The currency being converted into.	required
days	Index	The dates wanted, ascending.	required

Returns:

Type	Description
Series | None	pd.Series | None: One rate per day, indexed by days, or None when
Series | None	the pair is unknown entirely. Individual days the policy cannot
Series | None	answer for are NaN — a day is missing, not the pair — which is the
Series | None	distinction _rate_series in chaining already depended on.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fx_rates_on(self,
                from_currency: str,
                to_currency: str,
                days: pd.Index) -> pd.Series | None:
    """:meth:`fx_rate_on` over many days at once, as a Series.

    The vectorised face of the same rule, for a caller that needs a rate
    for every day of a run rather than one date (BN-207). Chained levels
    want exactly that, and asking per day would be one search per day per
    currency where a single reindex answers the lot.

    It exists so that sharing the *rule* does not force one call shape on
    every caller: the policy, the carry semantics and the meaning of "no
    rate" are decided here once, and the two methods differ only in how
    many answers they return. Two implementations of the lookup is how the
    library came to have five of them.

    Args:
        from_currency: The currency being converted out of.
        to_currency: The currency being converted into.
        days: The dates wanted, ascending.

    Returns:
        pd.Series | None: One rate per day, indexed by *days*, or None when
        the pair is unknown entirely. Individual days the policy cannot
        answer for are NaN — a day is missing, not the pair — which is the
        distinction `_rate_series` in chaining already depended on.
    """
    if from_currency.upper() == to_currency.upper():
        return pd.Series(1.0, index=days)

    pair = (from_currency.upper(), to_currency.upper())

    series = self._rate_series(*pair)

    if series.empty:
        return None

    if self.fx_policy == FX_EXACT_DAY:
        return series.astype(float).reindex(days)

    # `ffill` is the carry, and it leaves NaN before the first rate rather
    # than back-filling it — which is the vectorised statement of BN-204:
    # forward over a gap, never backward into one.
    return series.astype(float).reindex(days, method="ffill")

fetch_reference_data

fetch_reference_data(
    identifier: str | list[str],
    date: str | None = None,
    columns: list[str] | None = None,
) -> pd.DataFrame

Fetch reference data for one or more identifiers.

Parameters:

Name	Type	Description	Default
identifier	str | list[str]	One identifier or a list of identifiers.	required
date	str | None	Point-in-time date. Only rows valid at this date are returned.	None
columns	list[str] | None	Subset of columns to return.	None

Returns:

Type	Description
DataFrame	pd.DataFrame: Indexed by IDENTIFIER. Empty DataFrame if no
DataFrame	reference data is loaded or identifier is not found.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_reference_data(self,
                         identifier: str | list[str],
                         date: str | None = None,
                         columns: list[str] | None = None) -> pd.DataFrame:
    """Fetch reference data for one or more identifiers.

    Args:
        identifier: One identifier or a list of identifiers.
        date: Point-in-time date. Only rows valid at this date are
            returned.
        columns: Subset of columns to return.

    Returns:
        pd.DataFrame: Indexed by ``IDENTIFIER``. Empty DataFrame if no
        reference data is loaded or identifier is not found.
    """
    if self._reference is None:
        return pd.DataFrame()

    return self._reference.get(identifier, date, columns)

fetch_classification

fetch_classification(
    identifier: str,
    date: str | Timestamp | None = None,
    scheme: str = DEFAULT_SCHEME,
) -> str | None

One instrument's classification as it stood on a date.

Reference data already carries validity ranges, so a name that moved from Industrials to Technology has two rows and this returns whichever was in force. That matters for anything historical: attributing a 2021 return to a sector the company only joined in 2023 is a real way to get a breakdown wrong.

Parameters:

Name	Type	Description	Default
identifier	str	The instrument.	required
date	str | Timestamp | None	The as-of date. None takes the currently-active record — the one with no end date — falling back to the latest start date if every record has been closed off.	None
scheme	str	Which column to read, e.g. "SECTOR", "INDUSTRY", "COUNTRY". Free-form, because which columns a client loads is its own business.	DEFAULT_SCHEME

Returns:

Type	Description
str | None	str or None: The classification, or None when it is unknown: no
str | None	reference data, no such instrument, no such column, or no record
str | None	valid on that date.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_classification(self,
                         identifier: str,
                         date: str | pd.Timestamp | None = None,
                         scheme: str = DEFAULT_SCHEME) -> str | None:
    """One instrument's classification as it stood on a date.

    Reference data already carries validity ranges, so a name that moved
    from Industrials to Technology has two rows and this returns whichever
    was in force. That matters for anything historical: attributing a 2021
    return to a sector the company only joined in 2023 is a real way to get
    a breakdown wrong.

    Args:
        identifier: The instrument.
        date: The as-of date. None takes the currently-active record — the
            one with no end date — falling back to the latest start date if
            every record has been closed off.
        scheme: Which column to read, e.g. ``"SECTOR"``, ``"INDUSTRY"``,
            ``"COUNTRY"``. Free-form, because which columns a client loads
            is its own business.

    Returns:
        str or None: The classification, or None when it is unknown: no
        reference data, no such instrument, no such column, or no record
        valid on that date.
    """
    if self._reference is None:
        return None

    frame = self._reference.get(identifier,
                                str(date) if date is not None else None)
    if frame.empty or scheme not in frame.columns:
        return None

    if date is None:
        frame = self._current_record(frame)

    value = frame[scheme].iloc[0]

    return None if pd.isna(value) else str(value)

fetch_classifications

fetch_classifications(
    identifiers: list[str],
    date: str | Timestamp | None = None,
    scheme: str = DEFAULT_SCHEME,
) -> dict[str, str | None]

Classifications for several instruments at once.

Every identifier appears, with None where the classification is unknown, so a caller can see what is missing rather than finding it silently absent.

Parameters:

Name	Type	Description	Default
identifiers	list[str]	The instruments.	required
date	str | Timestamp | None	As-of date, as for :meth:fetch_classification.	None
scheme	str	Which column to read.	DEFAULT_SCHEME

Returns:

Name	Type	Description
dict	dict[str, str | None]	Identifier to classification.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def fetch_classifications(self,
                          identifiers: list[str],
                          date: str | pd.Timestamp | None = None,
                          scheme: str = DEFAULT_SCHEME) -> dict[str, str | None]:
    """Classifications for several instruments at once.

    Every identifier appears, with None where the classification is
    unknown, so a caller can see what is missing rather than finding it
    silently absent.

    Args:
        identifiers: The instruments.
        date: As-of date, as for :meth:`fetch_classification`.
        scheme: Which column to read.

    Returns:
        dict: Identifier to classification.
    """
    return {identifier: self.fetch_classification(identifier, date, scheme)
            for identifier in identifiers}

group_by_classification

group_by_classification(
    identifiers: list[str],
    date: str | Timestamp | None = None,
    scheme: str = DEFAULT_SCHEME,
) -> dict[str, list[str]]

Instruments grouped by classification, ready for GroupBounds.

Unclassified instruments are collected under UNCLASSIFIED rather than dropped. A name missing from every bucket is how a constraint set quietly stops covering part of the universe.

Parameters:

Name	Type	Description	Default
identifiers	list[str]	The instruments.	required
date	str | Timestamp | None	As-of date.	None
scheme	str	Which column to read.	DEFAULT_SCHEME

Returns:

Name	Type	Description
dict	dict[str, list[str]]	Classification to the identifiers carrying it, each list in
	dict[str, list[str]]	the order the identifiers were given.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/fetcher.py

def group_by_classification(self,
                            identifiers: list[str],
                            date: str | pd.Timestamp | None = None,
                            scheme: str = DEFAULT_SCHEME) -> dict[str, list[str]]:
    """Instruments grouped by classification, ready for GroupBounds.

    Unclassified instruments are collected under UNCLASSIFIED rather than
    dropped. A name missing from every bucket is how a constraint set
    quietly stops covering part of the universe.

    Args:
        identifiers: The instruments.
        date: As-of date.
        scheme: Which column to read.

    Returns:
        dict: Classification to the identifiers carrying it, each list in
        the order the identifiers were given.
    """
    grouped: dict[str, list[str]] = {}

    for identifier in identifiers:
        label = self.fetch_classification(identifier, date, scheme)
        grouped.setdefault(label or UNCLASSIFIED, []).append(identifier)

    return grouped

load_data

load_data(env: Environment) -> DataFetcher

Read data files from the environment config and return a DataFetcher.

For each dataset, a DataFrame is checked first; if not provided, the file path is used instead. Raises ValueError if no market data is available from either source.

Source code in build/cache/py-beacon-2c9c3936c65abdb6b8403c50a023f58355be30ee/src/beacon/data/loader.py

def load_data(env: Environment) -> DataFetcher:
    """Read data files from the environment config and return a DataFetcher.

    For each dataset, a DataFrame is checked first; if not provided, the
    file path is used instead. Raises ValueError if no market data is
    available from either source.
    """
    if env.data_source.MARKET_DATA is not None:
        market = MarketData.from_dataframe(env.data_source.MARKET_DATA,
                                           date_format=env.data.DATE_FORMAT)
    elif env.data_source.MARKET_DATA_PATH is not None:
        market = MarketData(env.data_source.MARKET_DATA_PATH,
                            date_format=env.data.DATE_FORMAT)
    else:
        raise ValueError(
            "No market data provided. Set MARKET_DATA or "
            "MARKET_DATA_PATH on env.data_source."
        )

    reference = None
    if env.data_source.REFERENCE_DATA is not None:
        reference = ReferenceData.from_dataframe(env.data_source.REFERENCE_DATA)
    elif env.data_source.REFERENCE_DATA_PATH is not None:
        reference = ReferenceData(env.data_source.REFERENCE_DATA_PATH)

    return DataFetcher(market, reference)