Skip to content

Synthetic data

The synthetic market generator: a universe of invented companies whose prices behave like a market, written as a data store and extendable to a later date.

synthetic

Synthetic market data at demo scale.

Generates a universe of anonymised companies with prices that reproduce the stylized facts of equity returns (volatility clustering, fat tails, negative skew, and a factor structure that makes names co-move), together with the reference data, shares outstanding, free float and corporate actions that go with them.

from beacon.synthetic import SyntheticConfig, generate

dataset = generate(SyntheticConfig(assets=64, seed=1))
fetcher = dataset.fetcher()

Or from the command line, writing a store the server auto-loads:

python -m beacon.synthetic --assets 512 --start 2019-12-31 --seed 42

A generated store can later be extended to a new date without changing its history: extend(path), or python -m beacon.synthetic --extend PATH.

Nothing generated here resembles a real company: names are Company A … and every ticker carries a CMP prefix, which makes a collision with a real listing impossible rather than merely improbable.

This is not beacon.testing.dataset, which is a tiny frozen fixture whose exact values chart baselines depend on. See dataset.py for why the two must stay apart.

SyntheticConfig dataclass

SyntheticConfig(
    assets: int = DEFAULT_ASSETS,
    start: str = DEFAULT_START,
    end: str = DEFAULT_END,
    seed: int = DEFAULT_SEED,
    risk_free_rate: float = DEFAULT_RISK_FREE_RATE,
    equity_premium: float = DEFAULT_EQUITY_PREMIUM,
    currency: str = universe.DEFAULT_CURRENCY,
    delisting_rate: float = listings.ANNUAL_DELISTING_RATE,
    listing_rate: float = listings.ANNUAL_LISTING_RATE,
    features: bool = True,
    calendar: str = DEFAULT_CALENDAR,
)

What to generate.

Attributes:

Name Type Description
assets int

How many names.

start str

First date of the panel.

end str

Last date of the panel.

seed int

Everything random is drawn from this, so the same seed and the same dates produce the same dataset.

risk_free_rate float

Annualised, the base of the CAPM drift.

equity_premium float

Annualised excess return on a beta-one name.

currency str

Not used: each name's currency comes from its listing region (see beacon.synthetic.regions). Kept so existing callers that pass it still work.

delisting_rate float

Annualised hazard of a name leaving the universe. Zero gives a constant, survivorship-biased universe.

listing_rate float

Annualised hazard of a name having joined partway through rather than at the start.

features bool

Generate fundamental ratios and a little alternative data. On by default and cheap (about a tenth of the market panel), because a dataset with nothing to screen on cannot exercise a feature rule, and a store that silently lacks one is harder to diagnose than one that costs a few hundred thousand rows.

calendar str

The exchange MIC whose sessions the panel has bars on. Defaults to DEFAULT_CALENDAR ("XNYS"), the calendar an older stored index without one is migrated to, so generated data and an index on that calendar agree by construction rather than by coincidence.

One calendar for the whole dataset, including the names quoted in another currency: a universe whose venues genuinely disagree about sessions is a real question and not this one's. A name whose own exchange would have been shut still gets a bar, which is the lesser of the two wrongs: the alternative is a hole in the panel that every consumer would have to tell apart from missing data.

Raises:

Type Description
ValueError

If assets is below 1, end is not after start, or calendar is not a calendar this installation can use.

SyntheticDataset dataclass

SyntheticDataset(
    market: MarketData,
    reference: ReferenceData,
    actions: CorporateActions,
    universe: DataFrame,
    returns: DataFrame,
    features: FeatureData = FeatureData.empty(),
)

A generated universe and everything drawn from it.

Attributes:

Name Type Description
market MarketData

OHLCV, shares outstanding and free float.

reference ReferenceData

Names, classification, exchange and currency.

actions CorporateActions

Dividends and splits, matching the price path.

universe DataFrame

The per-name parameters behind the draw. Exposed because the targets are what a statistical check should compare against: the realised figures are a sample from them, not the same thing.

returns DataFrame

The economic total returns the prices were built from. Not recoverable from market alone without undoing the dividends and splits, which is precisely what a coherence test does.

features FeatureData

Fundamental ratios and alternative data, derived from the prices above so a valuation screen and a price screen agree.

fetcher

fetcher() -> DataFetcher

A DataFetcher over the generated data.

Extension dataclass

Extension(
    first: str | None,
    last: str | None,
    sessions: int,
    listed: int,
    delisted: int,
)

What an extension added.

Attributes:

Name Type Description
first str | None

The first new session, or None if there was nothing to add.

last str | None

The last new session, or None.

sessions int

How many sessions were added.

listed int

How many new names listed.

delisted int

How many names left.

generate

generate(
    config: SyntheticConfig | None = None,
    progress: Progress = _silent,
) -> SyntheticDataset

Generate a synthetic dataset.

Parameters:

Name Type Description Default
config SyntheticConfig | None

What to generate; defaults to 512 names over a fixed five-year window.

None
progress Progress

Called at each stage with the fraction done and what is happening. The fractions are rough shares of the time each stage takes at the default size, not exact.

_silent

Returns:

Name Type Description
SyntheticDataset SyntheticDataset

The panels and the parameters behind them.

Raises:

Type Description
ValueError

If the calendar has no sessions in the window.

write

write(
    config: SyntheticConfig,
    path: Path,
    progress: Progress = _silent,
) -> Path

Generate a dataset and write it as a data store.

The store also keeps what an extension needs to carry it on later (see beacon.synthetic.extend).

Parameters:

Name Type Description Default
config SyntheticConfig

What to generate.

required
path Path

Store directory, created if absent.

required
progress Progress

As for :func:generate, carried on through the write.

_silent

Returns:

Name Type Description
Path Path

The directory written.

dataset

Assembling a synthetic universe into the containers the rest of py-beacon reads.

This is the layer that turns four panels into a DataFetcher, and the one place that knows the whole dataset is meant to be mutually consistent: the reference data names the same identifiers the market data prices, the action history refers only to those identifiers, and the shares outstanding move on exactly the dates the split actions record.

This is not beacon.testing.dataset

They are deliberately different things and neither should grow into the other.

beacon.testing.dataset is a tiny frozen fixture: six names, three years of sessions, and price paths built from + and * alone so the numbers are bit-identical on every platform. Chart baselines and unit tests depend on those exact values, so it must never change.

This module is a generator: hundreds of names, years of history, drawn from a model whose parameters are meant to be tuned as the model improves. Nothing should assert on its exact values. Use the fixture when a test needs a known number, and this when something needs to look like a market.

SyntheticConfig dataclass

SyntheticConfig(
    assets: int = DEFAULT_ASSETS,
    start: str = DEFAULT_START,
    end: str = DEFAULT_END,
    seed: int = DEFAULT_SEED,
    risk_free_rate: float = DEFAULT_RISK_FREE_RATE,
    equity_premium: float = DEFAULT_EQUITY_PREMIUM,
    currency: str = universe.DEFAULT_CURRENCY,
    delisting_rate: float = listings.ANNUAL_DELISTING_RATE,
    listing_rate: float = listings.ANNUAL_LISTING_RATE,
    features: bool = True,
    calendar: str = DEFAULT_CALENDAR,
)

What to generate.

Attributes:

Name Type Description
assets int

How many names.

start str

First date of the panel.

end str

Last date of the panel.

seed int

Everything random is drawn from this, so the same seed and the same dates produce the same dataset.

risk_free_rate float

Annualised, the base of the CAPM drift.

equity_premium float

Annualised excess return on a beta-one name.

currency str

Not used: each name's currency comes from its listing region (see beacon.synthetic.regions). Kept so existing callers that pass it still work.

delisting_rate float

Annualised hazard of a name leaving the universe. Zero gives a constant, survivorship-biased universe.

listing_rate float

Annualised hazard of a name having joined partway through rather than at the start.

features bool

Generate fundamental ratios and a little alternative data. On by default and cheap (about a tenth of the market panel), because a dataset with nothing to screen on cannot exercise a feature rule, and a store that silently lacks one is harder to diagnose than one that costs a few hundred thousand rows.

calendar str

The exchange MIC whose sessions the panel has bars on. Defaults to DEFAULT_CALENDAR ("XNYS"), the calendar an older stored index without one is migrated to, so generated data and an index on that calendar agree by construction rather than by coincidence.

One calendar for the whole dataset, including the names quoted in another currency: a universe whose venues genuinely disagree about sessions is a real question and not this one's. A name whose own exchange would have been shut still gets a bar, which is the lesser of the two wrongs: the alternative is a hole in the panel that every consumer would have to tell apart from missing data.

Raises:

Type Description
ValueError

If assets is below 1, end is not after start, or calendar is not a calendar this installation can use.

SyntheticDataset dataclass

SyntheticDataset(
    market: MarketData,
    reference: ReferenceData,
    actions: CorporateActions,
    universe: DataFrame,
    returns: DataFrame,
    features: FeatureData = FeatureData.empty(),
)

A generated universe and everything drawn from it.

Attributes:

Name Type Description
market MarketData

OHLCV, shares outstanding and free float.

reference ReferenceData

Names, classification, exchange and currency.

actions CorporateActions

Dividends and splits, matching the price path.

universe DataFrame

The per-name parameters behind the draw. Exposed because the targets are what a statistical check should compare against: the realised figures are a sample from them, not the same thing.

returns DataFrame

The economic total returns the prices were built from. Not recoverable from market alone without undoing the dividends and splits, which is precisely what a coherence test does.

features FeatureData

Fundamental ratios and alternative data, derived from the prices above so a valuation screen and a price screen agree.

fetcher
fetcher() -> DataFetcher

A DataFetcher over the generated data.

generate

generate(
    config: SyntheticConfig | None = None,
    progress: Progress = _silent,
) -> SyntheticDataset

Generate a synthetic dataset.

Parameters:

Name Type Description Default
config SyntheticConfig | None

What to generate; defaults to 512 names over a fixed five-year window.

None
progress Progress

Called at each stage with the fraction done and what is happening. The fractions are rough shares of the time each stage takes at the default size, not exact.

_silent

Returns:

Name Type Description
SyntheticDataset SyntheticDataset

The panels and the parameters behind them.

Raises:

Type Description
ValueError

If the calendar has no sessions in the window.

write

write(
    config: SyntheticConfig,
    path: Path,
    progress: Progress = _silent,
) -> Path

Generate a dataset and write it as a data store.

The store also keeps what an extension needs to carry it on later (see beacon.synthetic.extend).

Parameters:

Name Type Description Default
config SyntheticConfig

What to generate.

required
path Path

Store directory, created if absent.

required
progress Progress

As for :func:generate, carried on through the write.

_silent

Returns:

Name Type Description
Path Path

The directory written.

settings_of

settings_of(config: SyntheticConfig) -> state.Settings

The generator state a store made with config starts from.

extend

Extend a generated store to a later date, without changing its history.

from beacon.synthetic.extend import extend

extend(Path("my-store"))                    # up to today
extend(Path("my-store"), end="2026-06-30")

or from the command line, python -m beacon.synthetic --extend my-store.

Why not regenerate with a later end date

The generator draws every number from one random stream sized by the whole date range. A later end date changes every past price, and with it every index and backtest saved against the store.

What an extension does

It carries the market on from where the store stops:

  • every listed name from its last close and share count, with the volatility, factor loadings, alpha and dividend yield it was generated with (saved beside the data; see beacon.synthetic.state);
  • each FX pair from its last rate;
  • new listings and delistings at the store's rates, timed by the same crisis intensity as before, with new names continuing the ticker sequence;
  • dividends and split reviews on the same calendar rules, without a second ex-date in a month that already had one;
  • features, including quarters that ended before the extension but are reported inside it.

The new days draw from a random stream derived from the store's seed and the dates of the new sessions, so the same store extended to the same date always gives the same data.

What changes and what does not

Market and feature rows already in the store are never touched: the new rows are appended to the files, and the bytes already there stay as they were. Three kinds of record do change, because they describe a state rather than a day: a name that delists gets its end date and status, a dividend whose pay date arrives becomes paid, and each listed name's next earnings date moves forward.

Crises are the real, dated episodes in beacon.synthetic.regimes, so days after the last of them are calm, as they would be in a store generated over the same dates.

Extension dataclass

Extension(
    first: str | None,
    last: str | None,
    sessions: int,
    listed: int,
    delisted: int,
)

What an extension added.

Attributes:

Name Type Description
first str | None

The first new session, or None if there was nothing to add.

last str | None

The last new session, or None.

sessions int

How many sessions were added.

listed int

How many new names listed.

delisted int

How many names left.

extend

extend(
    path: Path,
    end: str | date | None = None,
    progress: Progress = _silent,
) -> Extension

Extend a generated store to end, keeping everything already in it.

Parameters:

Name Type Description Default
path Path

The store folder.

required
end str | date | None

The last date to extend to; today if omitted. A date with no new session (a weekend, or a date the store already reaches) leaves the store as it was.

None
progress Progress

Called at each stage with the fraction done and what is happening.

_silent

Returns:

Name Type Description
Extension Extension

What was added.

Raises:

Type Description
ValueError

If the store holds no generator state, which is the case for anything not generated by py-beacon 0.2.0 or later.

features

Synthetic features: fundamental ratios and a little alternative data.

A small set, deliberately. The point is to have something to screen on and something to exercise the point-in-time path with, not to simulate a data vendor. Four fundamental ratios and two alternative series is enough for a universe filter, an index rule and a look-ahead test, and cheap enough that every generated store can carry it.

Coherent with the price series, not drawn beside it

The trap this module exists to avoid.

A price-to-earnings ratio drawn independently of the prices in the same dataset contradicts them: screen on pe_ratio < 15 and screen on price, and the two disagree about the same company. Worse, the disagreement is invisible until somebody checks, and by then it has been believed.

So the ratios are derived from the generated prices:

eps        = close / pe_ratio          (so pe x eps == price, exactly)
book_value = close / pb_ratio
d/e        = drawn per name, by sector: the one genuinely independent of
             price, because leverage is a balance-sheet fact

The multiple is what gets drawn, sector by sector, and the per-share figure follows from it. That ordering is what makes them consistent by construction rather than by luck.

Sentiment and page views are anchored the same way: sentiment tracks recent returns, because it does, and page views scale with size and spike on large moves. A name nobody has heard of does not trend on a quiet day.

The announcement lag varies, and that is not decoration

DATE holds when a value became knowable (beacon.data.features), so a fundamental for the quarter ending 31 March is published somewhere in the following weeks. A constant lag would make every look-ahead test pass whether or not the accessor was correct: with all values 45 days late, any off-by-one still lands in the same gap. The lag is drawn per name per quarter, so a test standing on a given date sees a genuinely ragged edge.

Coverage is deliberately incomplete

Real fundamentals are missing for some names and some quarters, and alternative datasets cover a fraction of a universe, mostly the large, visible names. Generating a complete grid would make the missing-coverage behaviour in FeatureRule untestable against this data, and would overstate what an alternative vendor sells.

build

build(
    universe: DataFrame,
    prices: DataFrame,
    returns: DataFrame,
    rng: Generator,
    fundamentals: bool = True,
    alternative: bool = True,
) -> pd.DataFrame

Generate the feature table for a panel.

Parameters:

Name Type Description Default
universe DataFrame

Output of universe.build, for sector and size.

required
prices DataFrame

Wide close prices, dates by identifier.

required
returns DataFrame

Wide daily returns, for the sentiment anchor.

required
rng Generator

Seeded generator.

required
fundamentals bool

Include the ratio set.

True
alternative bool

Include sentiment and page views.

True

Returns:

Type Description
DataFrame

pd.DataFrame: Long-form feature rows, ready for FeatureData.

carry_on

carry_on(
    stored: DataFrame,
    universe: DataFrame,
    joined: Index,
    closes: DataFrame,
    after: Timestamp,
    rng: Generator,
) -> pd.DataFrame

Feature rows that become known after after, for an extension.

A name keeps the multiples and coverage its stored rows show. A quarter that ended before after but had not been reported by then is reported now, with its lag drawn from what remains of the usual range, so no row lands on a date the store has already passed.

Parameters:

Name Type Description Default
stored DataFrame

The store's feature rows.

required
universe DataFrame

Every name that can report, with SECTOR and market_cap.

required
joined Index

The names listing in this extension, which draw their coverage and multiples as the originals did.

required
closes DataFrame

Wide closes from LOOKBACK_DAYS before after to the extension's last date.

required
after Timestamp

The last date already in the store.

required
rng Generator

Seeded generator.

required

fx

Exchange rates, generated as market data so the FX paths are actually exercised.

DataFetcher.fetch_fx_rates looks a pair up as an ordinary market-data identifier named f"{from}{to}": EURUSD converts euros into dollars. Nothing special: a currency pair is a row set like any other, which is why the calculator and the corporate-action handler can convert without knowing where the rate came from.

Why a rate is not a price

Two differences, and both matter to what the data is for.

Volatility is far lower. A major pair realises 7-11% annualised against 25-35% for a single equity. A generator that gave currencies equity-like volatility would make every unhedged exposure look like the dominant risk in a global portfolio, which is the opposite of true.

Not everything floats. The Hong Kong dollar runs inside a band the monetary authority defends, and realises well under 1%. Modelling it like the others would manufacture a diversification benefit that does not exist, and an optimiser told it can hedge HKD risk would allocate to a trade nobody makes.

Flight to quality

Crises are not currency-neutral. Money moves into dollars when volatility spikes, so every pair here drifts down against USD in proportion to how far the regime has lifted correlations. It is the same intensity series the return process uses, so the currency move lines up with the equity drawdown rather than wandering off on its own, which is what makes a hedged-versus-unhedged comparison over a crisis show anything at all.

build

build(
    dates: DatetimeIndex,
    rng: Generator,
    regimes: tuple[Regime, ...] = CRISES,
    start_from: dict[str, float] | None = None,
) -> pd.DataFrame

Generate one rate series per non-base currency.

Parameters:

Name Type Description Default
dates DatetimeIndex

The panel's business days.

required
rng Generator

Seeded generator.

required
regimes tuple[Regime, ...]

Dated episodes, for the flight-to-quality drift.

CRISES
start_from dict[str, float] | None

The rate each pair starts from, where known. An extension passes the last stored rates, so each path carries on from where it stopped rather than from the pair's usual level.

None

Returns:

Type Description
DataFrame

pd.DataFrame: Long-form, with IDENTIFIER/DATE and the rate in both

DataFrame

RATE and CLOSE, ready to be concatenated onto the equity market data.

listings

When a company joins the universe and when it leaves it.

A generated name may list partway through the panel and may leave before it ends. A universe where every name lists on day one and never leaves is the definition of a survivorship-biased dataset, and it makes a whole class of behaviour unreachable: additions and deletions, the divisor adjustment each one needs, a backtest that has to dispose of a holding that stopped trading, and point-in-time universe resolution ("who was in the index then", answered from history rather than from today's list).

It also makes the bias itself unmeasurable, which is the point. An index built only from the names that survived to the end of the panel outperforms one built as it went along, because the losers were quietly removed from the sample before the question was asked. A dataset where that difference is zero cannot demonstrate the single most common error in backtesting.

The rates are real, the events are not

Roughly 3% of a large-cap index leaves per year and roughly 3% joins: acquisitions, failures, and index-committee decisions all together. Over ten years that is about a quarter of the universe turning over, which is enough to matter and not so much that the panel becomes unrecognisable.

Failures cluster, and that is the whole point

A constant hazard would put delistings evenly across the panel, and the bias a point-in-time index suffers would be a slow drip. Real companies fail together, in exactly the windows where the index is already falling, which is what makes survivorship bias large rather than merely present. The hazard here is scaled by the same regime intensity the return process uses, so delistings pile into 2001, 2008 and 2020.

Listings do the opposite: nobody floats a company into a collapsing market, so new issuance is suppressed when the intensity is high. That asymmetry is why a crisis shrinks a universe rather than churning it.

draw

draw(
    count: int,
    dates: DatetimeIndex,
    rng: Generator,
    delisting_rate: float = ANNUAL_DELISTING_RATE,
    listing_rate: float = ANNUAL_LISTING_RATE,
    regimes: tuple[Regime, ...] = CRISES,
    alpha: ndarray | None = None,
) -> pd.DataFrame

Draw a listed life for every name.

Parameters:

Name Type Description Default
count int

How many names.

required
dates DatetimeIndex

The panel's business days.

required
rng Generator

Seeded generator.

required
delisting_rate float

Annualised hazard of leaving. Zero keeps every name to the end of the panel.

ANNUAL_DELISTING_RATE
listing_rate float

Annualised hazard of having joined partway through. Zero lists every name from the start.

ANNUAL_LISTING_RATE
regimes tuple[Regime, ...]

Dated episodes, for the clustering.

CRISES
alpha ndarray | None

Each name's alpha. When given, delistings tilt towards the weakest names without changing how many there are.

None

Returns:

Type Description
DataFrame

pd.DataFrame: listed_from and listed_to per name, both

DataFrame

Timestamps; listed_to is NaT for a name still listed at the end.

joining

joining(
    count: int,
    dates: DatetimeIndex,
    rng: Generator,
    listing_rate: float = ANNUAL_LISTING_RATE,
    regimes: tuple[Regime, ...] = CRISES,
) -> pd.DatetimeIndex

When each of count new names lists, over an extension's dates.

Timed like the original listings: suppressed in a crisis, so issuance dries up when the market falls.

leaving

leaving(
    alpha: ndarray,
    dates: DatetimeIndex,
    rng: Generator,
    delisting_rate: float = ANNUAL_DELISTING_RATE,
    regimes: tuple[Regime, ...] = CRISES,
) -> pd.DatetimeIndex

When each listed name leaves over an extension's dates, NaT if it stays.

The same hazard as the original delistings, clustered in crises and tilted towards the weakest names by their alpha.

prices

Turning a return panel into the market data a client actually reads: OHLCV, shares outstanding and free float, with dividends and splits folded into the price path rather than bolted on beside it.

The price is not the return path

A return series compounds into a total-return index. A stored CLOSE is neither that nor a clean geometric path: it drops on an ex-dividend date and it halves on a split, and every downstream calculation (trailing yield, the divisor adjustment, a split-adjusted chart) exists to undo one of those two things. Generating prices that never do either would produce a dataset on which none of that code could be exercised, which is the opposite of the point.

So the stored close is built as

close = initial · Π (1 + r) · (1 - q) / cumulative_split

where q is the ex-date drop as a fraction of price, and the corporate-action history records exactly the dividends and splits that appear in it. The two are generated together, so they cannot disagree.

Reconstructing the economic path

Multiplying CLOSE by the cumulative split ratio and adding dividends back recovers the return panel this module was handed. A test asserts it, because "coherent" is otherwise a claim rather than a property.

OHLC

Open comes off the previous close through an overnight gap; high and low are pushed out from whichever of open and close is the extreme. Constructing them that way makes H >= max(O, C) and L <= min(O, C) true by arithmetic rather than by a repair pass afterwards: a clamp that fixes violations after the fact is a clamp somebody eventually has to trust.

Splits are applied to all four prices at once, so the intraday relationships survive a split day intact.

Volume

Log-normal, scaled by market capitalisation so a mega-cap trades more than a small-cap, and pushed up on days when the move was large: the well-documented volume/volatility relationship. Without the second part, ADV would be a constant with noise on it and a liquidity screen built on it would never bind.

dividend_dates

dividend_dates(dates: DatetimeIndex) -> pd.DatetimeIndex

The ex-dividend dates inside a panel's date range.

The first business day on or after the 15th of each dividend month, so every ex-date is a day the panel actually has a price for.

review_dates

review_dates(dates: DatetimeIndex) -> pd.DatetimeIndex

The annual split-review dates: the first business day of each year.

The first year is skipped: a name cannot split before the panel starts, and reviewing on day one would split names purely for having been drawn an expensive opening price.

build

build(
    universe: DataFrame,
    returns: DataFrame,
    rng: Generator,
    block_size: int = BLOCK_SIZE,
    calendar: DatetimeIndex | None = None,
) -> tuple[pd.DataFrame, pd.DataFrame]

Build the market-data panel and the corporate-action history.

Names are processed in blocks. This is where the memory actually goes: thirteen full (days x names) panels are alive between the return input and the long-form output (the dividend drops, the multiplicative path, the split factor, the pre-drop price, the dividend amounts, the close, the share count, and five OHLCV frames). Blocking bounds all of them at the block, leaving the output frame as the floor.

Nothing here couples one name to another: dividend dates come from the calendar, and a split is triggered by that name's own quoted price. So a block is the same computation on fewer columns, not an approximation.

Parameters:

Name Type Description Default
universe DataFrame

Output of universe.build.

required
returns DataFrame

Date-indexed total returns, one column per name.

required
rng Generator

Seeded generator.

required
block_size int

How many names to process at a time. Affects peak memory and nothing else.

BLOCK_SIZE
calendar DatetimeIndex | None

The sessions the dividend and split-review schedules are read from. Defaults to the panel's own dates. An extension passes sessions reaching back before its first date, so a month or year the existing data already began is not given a second ex-date or review.

None

Returns:

Name Type Description
tuple DataFrame

Long-form market data with IDENTIFIER/DATE and OHLCV plus

DataFrame

SHARES_OUTSTANDING and FREE_FLOAT, and a long-form action history.

profiles

The reference fields a profile view expects: identifiers, classification depth, and corporate facts.

The universe generator produces what the engine needs to calculate: sector, currency, exchange, share count. A client showing an instrument's profile needs a good deal more, and every field it cannot fill renders as a dash. This generates the rest.

Coherent with what is already generated

The same rule the synthetic features follow. A generated fact that contradicts another generated fact is worse than an absent one, because the contradiction is invisible until somebody checks and by then it has been believed.

So:

  • trading_status follows the listings model: a name delisted in 2021 is not "Active", and a screen on status has to agree with one on dates
  • dividend_frequency follows the dividend yield actually generated. A name that pays nothing is not "Quarterly"; saying so would make a dividend-frequency screen select names with no dividends
  • ipo_date is the listing date, not a second date drawn beside it
  • the GICS levels nest: sub-industry inside industry inside industry group inside sector. A four-level hierarchy whose levels do not contain each other is decoration, and any roll-up computed from it would be wrong
The identifiers are structurally valid

isin, cusip and sedol carry their real check digits, and figi its real shape. Not for authenticity (nothing here is a real security), but because a client that validates an identifier before using it would otherwise reject the whole store, and an ISIN is the field somebody is most likely to parse.

A wrong check digit is also the kind of thing that works until the day something checks it, which is the failure mode worth spending twenty lines to avoid.

build

build(
    universe: DataFrame, rng: Generator, as_of: Timestamp
) -> pd.DataFrame

Generate the profile columns for a universe.

Parameters:

Name Type Description Default
universe DataFrame

Output of universe.build, indexed by identifier.

required
rng Generator

Seeded generator.

required
as_of Timestamp

The panel's last date, for next_earnings and status.

required

Returns:

Type Description
DataFrame

pd.DataFrame: Indexed by identifier, one column per profile field.

next_earnings

next_earnings(
    as_of: Timestamp, count: int, rng: Generator
) -> list[pd.Timestamp]

The next reporting date, forward of the panel's end.

regimes

Market regimes: the crises a stationary model cannot produce.

The return process in returns.py is stationary by construction. Its volatility clusters, its tails are fat and its correlations are stable: all true of markets on average, and all wrong about the periods that matter most.

Twenty-five years of equity history is not a draw from one distribution. It contains three or four episodes where volatility trebles, drift turns sharply negative, and (the part that matters) correlations rise toward one. Diversification stops working precisely when it is most wanted, and a model without that overstates how much a spread portfolio protects you.

What a regime does

Each regime scales three things over a dated window:

  • volatility: a multiplier on the market factor, so every name inherits it through its own beta rather than being scaled directly
  • drift: an annualised amount added to the market's return over the window
  • correlation: the market factor's share of total variance rises, which is what makes names move together

The third is the one worth stating twice. Raising volatility alone produces a big drawdown that a diversified portfolio still cushions. Raising the market's share of variance is what removes the cushion.

Shape, not a step function

A crisis does not begin at midnight. Each window is ramped in and out with a raised-cosine taper over a fraction of its length, so volatility builds and subsides. A rectangular window produces a discontinuity in realised volatility that shows up as an obviously artificial jump on any chart.

The dates are real, the paths are not

The windows below are the actual episodes. What happens inside them is still generated: this is not a replay of 2008, and nothing here reproduces any real security's price. It is a synthetic market that has crises where the real one did, which is what makes a backtest over it exercise the code paths a calm market never reaches.

Regime dataclass

Regime(
    name: str,
    start: str,
    end: str,
    volatility: float = 1.0,
    drift: float = 0.0,
    correlation: float = 0.0,
    ramp: float = 0.25,
)

One dated market episode.

Attributes:

Name Type Description
name str

What it was.

start str

First date of the episode.

end str

Last date.

volatility float

Multiplier on the market factor's volatility at peak intensity. 1.0 is calm.

drift float

Annualised return added to the market over the window, at peak. Negative for a crisis.

correlation float

How far the market's share of total variance moves toward one, from 0 (unchanged) to 1 (everything is the market). This is the diversification failure.

ramp float

Fraction of the window spent easing in and out.

intensity_series

intensity_series(
    dates: DatetimeIndex,
    regimes: tuple[Regime, ...] = CRISES,
) -> pd.DataFrame

Per-date intensity for every regime touching a panel.

Parameters:

Name Type Description Default
dates DatetimeIndex

The panel's business days.

required
regimes tuple[Regime, ...]

Episodes to apply.

CRISES

Returns:

Type Description
DataFrame

pd.DataFrame: Date-indexed, one column per regime, values in [0, 1].

DataFrame

Regimes that fall entirely outside the panel are omitted, so a

DataFrame

window that touches none of them carries no columns and behaves as a

DataFrame

stationary market.

market_multipliers

market_multipliers(
    dates: DatetimeIndex,
    regimes: tuple[Regime, ...] = CRISES,
) -> tuple[np.ndarray, np.ndarray, np.ndarray]

How the market factor is scaled on each date.

Overlapping regimes take the most severe value rather than compounding. The 2008 panic is the peak of the financial crisis, not a second shock on top of it, and multiplying the two gave a 5.5x volatility multiplier and a combined -112% annual drift: a market that fell further in one quarter than it has in any real decade.

Parameters:

Name Type Description Default
dates DatetimeIndex

The panel's business days.

required
regimes tuple[Regime, ...]

Episodes to apply.

CRISES

Returns:

Name Type Description
tuple ndarray

Volatility multiplier, daily drift adjustment, and the share of

ndarray

variance moved toward the market factor, each per date.

describe

describe(
    dates: DatetimeIndex,
    regimes: tuple[Regime, ...] = CRISES,
) -> list[str]

One line per regime the panel covers, for a caller to log or print.

regions

Listing venues: where a generated company trades, and in what currency.

A single-currency universe cannot exercise most of what py-beacon does with currency. The calculator converts every market value through fetch_fx_rates, corporate actions convert their cash amounts the same way, and both paths are dead code against a universe that is entirely USD: they run, they multiply by 1.0, and nothing they could get wrong would show up.

What a region is here

A listing venue with one currency, one exchange and a share of the world. Deliberately not a country: "Europe" prices in EUR and lists on XETR, which is true of enough of the continent to be useful and wrong about Switzerland. Modelling countries properly would mean a currency per country and a correlation structure between them, which is a bigger claim than generated data should make.

The weights are roughly MSCI ACWI

The United States really is around 60% of global equity market capitalisation. A synthetic global universe that splits evenly across regions would make every FX exposure look far more material than it is, and a "global" index built on it would behave nothing like one built on the real thing.

Prices are quoted in local currency

A name's initial_price is in its own currency, and its market cap is converted from the USD-scale draw so the size distribution stays global while the quoted price stays local. Without that conversion a 5 billion company would mean five billion of whatever it happens to be quoted in, and the biggest companies in the universe would be an artefact of the exchange rate.

The price range is deliberately not localised. Real JPY quotes run to thousands of yen, but the split rule, the tick size and the four-decimal rounding are all calibrated to the 12-480 band, and localising the range would mean localising all three. The prices are local-currency amounts drawn from a common band, which is a limitation worth stating rather than hiding.

Region dataclass

Region(
    name: str,
    currency: str,
    exchange: str,
    country: str,
    weight: float,
    rate: float,
    volatility: float,
)

One listing venue.

Attributes:

Name Type Description
name str

What the reference data reports as REGION.

currency str

ISO code every name listed here is quoted in.

exchange str

MIC the names carry.

country str

ISO 3166-1 alpha-2 of the listing venue. Distinct from name: "Europe" is a region and DE is a country, and a filter asking "listed in Germany" cannot be answered by the first.

weight float

Share of the universe, roughly MSCI ACWI.

rate float

Units of currency per one US dollar at the start of the panel. Quoted this way round (rather than as the market convention, which differs per pair) because it is the direction the conversion needs and a single convention cannot be misread.

volatility float

Annualised volatility of the exchange rate.

assign

assign(count: int, rng: Generator) -> np.ndarray

Which region each name lists in.

Allocated by quota rather than drawn independently, so a 20-name universe still contains a non-US name and the realised weights match the targets at any size. An independent draw at 0.02 leaves Australia absent from most small universes, which is the size a test uses.

Parameters:

Name Type Description Default
count int

How many names.

required
rng Generator

Seeded generator, used only to shuffle the assignment so region does not correlate with position, and therefore not with sector, which is assigned round-robin.

required

Returns:

Type Description
ndarray

np.ndarray: One region index per name.

pairs

pairs() -> list[tuple[str, float, float]]

The FX pairs a generated panel needs, as (identifier, rate, volatility).

One per non-base currency, named the way fetch_fx_rates looks it up: f"{from}{to}", so converting EUR into USD reads EURUSD. The rate is inverted from the region's, which stores units per dollar.

domiciles

domiciles(assigned: ndarray, rng: Generator) -> list[str]

Where each name is incorporated.

Usually the listing country. A minority are incorporated elsewhere, drawn from the destinations that listing venue actually uses.

Parameters:

Name Type Description Default
assigned ndarray

One region index per name.

required
rng Generator

Seeded generator.

required

Returns:

Name Type Description
list list[str]

ISO 3166-1 alpha-2 per name.

frame

frame(assigned: ndarray) -> pd.DataFrame

Region, currency and exchange per name, given the assignment.

returns

The return process: a factor model with GJR-GARCH volatility and fat tails.

Gaussian random walks are the obvious way to make fake prices and they are wrong in every way that matters to what this data is for. They have no volatility clustering, so a drawdown chart shows nothing recognisable; no fat tails, so a risk model estimated from them is never stressed; and no cross-correlation, so an optimiser sees a diversification opportunity that no market offers and every constraint binds strangely. Each stylized fact below is here because leaving it out breaks a view somebody has to look at.

The model

For name i on day t:

r[i,t] = mu[i] + b[i]·f_market[t] + g[i]·f_sector(i)[t]
               + h[i]·f_region(i)[t] + e[i,t]

Four kinds of independent series, each a GJR-GARCH(1,1) process with standardised Student-t innovations:

  • The market factor: one series everything loads on. This is what makes names co-move, and giving it its own GARCH is what makes them co-move more in a crisis, which is when correlation matters.
  • Sector factors: one per GICS sector, so two banks resemble each other more than a bank resembles a utility.
  • Region factors: one per listing venue, so two names listed in Tokyo move together for reasons that have nothing to do with their industry.
  • Idiosyncratic noise: one per name.

Loadings are set from a variance budget rather than drawn directly, so a name's total volatility is a target that is hit rather than an outcome to be discovered. With the market at ~34% of variance, the sector at ~16% and the region at ~8%, same-sector pairs correlate near 0.50 and cross-sector pairs near 0.35, and the average across the universe lands near 0.39.

Why GJR rather than plain GARCH
sigma2[t] = omega + (alpha + lam·1[e<0])·e[t-1]^2 + beta·sigma2[t-1]

The lam term makes a fall raise tomorrow's volatility more than a rise of the same size does. That asymmetry is the leverage effect, and it is what produces negative skew; without it, simulated returns are symmetric and a drawdown looks like an upswing turned upside down.

Persistence is alpha + lam/2 + beta, drawn in 0.94-0.99: high enough that quiet and turbulent periods last for months rather than days, below one so the process is stationary and the unconditional variance is defined.

What is deterministic here, and what is not

Given a seed, this module produces the same numbers on the same machine and the same numpy. It does not guarantee bit-identical output across operating systems: standard_t and the exponentials behind it run through the platform's libm, which is free to differ in the last bit. That is a deliberate limit: avoiding transcendentals entirely would mean abandoning Student-t innovations and log-normal volume, which is most of what makes this data worth generating. Reproducibility is tested per-platform; the statistical acceptance checks are tolerance-based and hold anywhere.

standardised_t

standardised_t(
    rng: Generator,
    degrees: ndarray | float,
    shape: tuple[int, ...],
) -> np.ndarray

Negatively skewed Student-t draws, rescaled to unit variance.

A raw t has variance nu/(nu-2), so feeding it into a GARCH recursion unscaled would inflate every variance by that factor and make the target volatility wrong by 40% at four degrees of freedom.

The two-piece step stretches the downside and compresses the upside, then the whole thing is re-standardised: the stretch changes both the mean and the variance, and leaving either uncorrected would show up as a spurious drift and a missed volatility target.

simulate_gjr

simulate_gjr(
    steps: int,
    target_variance: ndarray,
    persistence: ndarray,
    innovations: ndarray,
) -> np.ndarray

Run the GJR-GARCH(1,1) recursion over pre-drawn innovations.

Vectorised across series and looped over time, which is the only way round: each day's variance depends on the day before, but every series can take its step together.

The innovations are an argument rather than drawn here, and that is what makes block generation possible. Every series in the recursion is independent of every other (the arithmetic is elementwise throughout), so running it over a slice of the universe gives bit-identical results to running it over all of it, provided each series sees the same innovations.

Parameters:

Name Type Description Default
steps int

Days to return, after burn-in.

required
target_variance ndarray

Unconditional daily variance per series.

required
persistence ndarray

alpha + lam/2 + beta per series, strictly below 1.

required
innovations ndarray

Shape (steps + BURN_IN, len(target_variance)), standardised to unit variance.

required

Returns:

Type Description
ndarray

np.ndarray: Shape (steps, len(target_variance)), mean zero, with

ndarray

unconditional variance equal to target_variance.

pin_realised_variance

pin_realised_variance(
    series: ndarray, target_variance: ndarray
) -> np.ndarray

Rescale a factor so its realised variance equals its target.

Applied to the shared factors only, and it is not cosmetic. A GARCH series at persistence 0.98 with t(4.5) innovations has enormous dispersion in its sample variance: over five years one draw can realise nearly twice its unconditional level. For an idiosyncratic series that is a fact about one name. For the market factor, which every name loads on, it rescales the entire universe: a generated dataset could come out with every volatility at 43% and an average correlation of 0.73 instead of 29% and 0.40.

Only the overall level is pinned. The clustering, the fat tails and the leverage asymmetry are properties of the shape of the path and survive a single multiplicative rescaling untouched.

Idiosyncratic series are deliberately left alone, so a name's realised volatility still varies around its target the way a real one does.

simulate

simulate(
    universe: DataFrame,
    dates: DatetimeIndex,
    rng: Generator,
    risk_free_rate: float,
    equity_premium: float,
    regimes: tuple[Regime, ...] = CRISES,
    block_size: int = BLOCK_SIZE,
) -> pd.DataFrame

Simulate the total-return panel.

Parameters:

Name Type Description Default
universe DataFrame

Output of universe.build, carrying the volatility target and variance shares each loading is derived from.

required
dates DatetimeIndex

Business days to simulate.

required
rng Generator

Seeded generator.

required
risk_free_rate float

Annualised, the base of the CAPM expectation.

required
equity_premium float

Annualised excess return on a beta-one name.

required
regimes tuple[Regime, ...]

Dated crisis episodes to overlay. Empty for a stationary market.

CRISES
block_size int

How many names to simulate at a time. Affects peak memory and nothing else: the panel is identical at any block size, and a test holds that.

BLOCK_SIZE

Returns:

Type Description
DataFrame

pd.DataFrame: Date-indexed daily total returns, one column per name.

state

What a generated store keeps so it can be extended later.

The stored prices say where every name ended up, but not the parameters it was drawn from: its target volatility, how its variance splits between the market, its sector and its region, its alpha, its dividend yield. Continuing a name without those would mean re-estimating them from a few years of one noisy path, and the extension would behave like a different market. So a generated store also saves them, in a synthetic folder beside the data:

  • settings.json: the seed, the rates and the calendar it was made with, and every extension since.
  • universe.csv.gz: one row per name, with the parameters above and its listed life.

Nothing else reads these files. A store without them loads as normal; it just cannot be extended.

Settings dataclass

Settings(
    seed: int,
    assets: int,
    start: str,
    end: str,
    calendar: str,
    risk_free_rate: float,
    equity_premium: float,
    delisting_rate: float,
    listing_rate: float,
    features: bool,
    extensions: list[dict[str, str]] = list(),
    version: int = STATE_VERSION,
)

How a store was generated, and how far it has been extended.

Attributes:

Name Type Description
seed int

The seed it was generated with. Extensions derive their own random streams from it.

assets int

The size of the universe it was generated with. New listings are sized against it.

start str

The first date asked for.

end str

The last date asked for, updated by each extension.

calendar str

The exchange whose sessions it has bars on.

risk_free_rate float

Annualised.

equity_premium float

Annualised.

delisting_rate float

Annualised hazard of a name leaving.

listing_rate float

Annualised hazard of a name joining.

features bool

Whether it carries features.

extensions list[dict[str, str]]

Each extension, as the first and last new session.

directory

directory(path: Path) -> Path

Where a store's state lives.

exists

exists(path: Path) -> bool

Whether a store has the state an extension needs.

save

save(
    path: Path, settings: Settings, universe: DataFrame
) -> None

Write a store's state beside its data.

load

load(path: Path) -> tuple[Settings, pd.DataFrame]

Read a store's state.

Raises:

Type Description
ValueError

If the store has none, or it was written by a newer py-beacon.

universe

The static half of a synthetic universe: who the companies are.

Everything here is decided once per run and does not vary by date: names, tickers, classification, and the per-name parameters the return process is driven by. Splitting it out from the time series keeps one question separate from the other: this module answers "what is in the universe", returns and prices answer "what did it do".

Nothing here resembles a real company

Names are Company A … Company Z, then Company AA … in the spreadsheet-column order, with tickers CMPA, CMPB, … CMPAA. The CMP prefix is what makes a collision with a real listing impossible rather than merely unlikely: a three-letter prefix on every symbol means no generated ticker can ever equal a real one, whatever the universe grows to. A test asserts it against a blocklist of well-known symbols anyway, because the guarantee is only as good as the naming function that provides it.

Sector names are the eleven GICS sectors, which are a public taxonomy rather than anybody's property. Sub-industries are not: real GICS sub-industry names would imply a classification this data does not have, so they are generic segments within each sector.

The parameters, and where they come from

The figures below are approximate long-run statistics for US large- and mid-cap equity. They are targets for a generator, not estimates: the point is that a chart drawn from this data looks like a chart drawn from a market, so somebody reviewing a layout is reviewing it against realistic numbers.

ticker_suffix

ticker_suffix(position: int) -> str

The spreadsheet-column label for a position: A, B, ... Z, AA, AB, ...

Parameters:

Name Type Description Default
position int

Zero-based index into the universe.

required

Returns:

Name Type Description
str str

The label, always at least one character.

identifiers

identifiers(count: int) -> list[str]

The tickers for a universe of a given size.

One function, used by every dataset in this package, so a name and its ticker cannot disagree between the market data and the reference data.

company_name

company_name(identifier: str) -> str

The display name matching a generated ticker.

build

build(
    count: int,
    rng: Generator,
    dates: DatetimeIndex | None = None,
    currency: str = DEFAULT_CURRENCY,
    delisting_rate: float = listings.ANNUAL_DELISTING_RATE,
    listing_rate: float = listings.ANNUAL_LISTING_RATE,
) -> pd.DataFrame

Draw the static universe.

Parameters:

Name Type Description Default
count int

How many names.

required
rng Generator

Seeded generator; every draw here comes from it, so the universe is a function of the seed alone.

required
dates DatetimeIndex | None

The panel's business days. When given, each name draws a listed life over them; when omitted every name is listed for the whole panel, which is what callers that only want the static fields get.

None
currency str

Not used. A name's currency comes from its listing region, because a universe forced into one currency is the case the FX paths never exercise. Kept for callers that pass it.

DEFAULT_CURRENCY

Returns:

Type Description
DataFrame

pd.DataFrame: One row per name, indexed by identifier, carrying both

DataFrame

the reference fields and the parameters the return process needs.

newcomers

newcomers(
    first_position: int,
    count: int,
    rng: Generator,
    universe_size: int,
) -> pd.DataFrame

Names that join after a store was generated, drawn like the originals.

Their tickers carry on the sequence (a universe of 512 names continues at position 512), and each takes a size rank drawn uniformly from the original universe, so newcomers are sized like the names already there rather than all small or all large.

Parameters:

Name Type Description Default
first_position int

The position of the first newcomer.

required
count int

How many.

required
rng Generator

Seeded generator.

required
universe_size int

The size of the universe the store was generated with, which sets the size profile.

required

Returns:

Type Description
DataFrame

pd.DataFrame: As :func:build without listed lives, which the caller

DataFrame

draws.

reference_frame

reference_frame(
    universe: DataFrame,
    valid_from: str,
    profile: DataFrame | None = None,
) -> pd.DataFrame

The reference dataset, long-form and ready for ReferenceData.

Parameters:

Name Type Description Default
universe DataFrame

Output of :func:build.

required
valid_from str

DATE_FROM for names listed since the panel began. A generated universe has no history of reclassification, so a name that was there at the start gets one record valid from the start rather than pretending to a change it never had.

required
profile DataFrame | None

Optional profile columns from profiles.build, joined on identifier. Absent leaves the profile columns out.

None

Returns:

Type Description
DataFrame

pd.DataFrame: One row per name, carrying the listed life. DATE_TO is

DataFrame

NaT for a name still listed at the end of the panel, which is what

DataFrame

ReferenceData.get reads as "still valid", so point-in-time

DataFrame

resolution drops a delisted name automatically.