Synthetic data¶
The synthetic market generator: a universe of invented companies whose prices behave like a market, written as a data store and extendable to a later date.
synthetic ¶
Synthetic market data at demo scale.
Generates a universe of anonymised companies with prices that reproduce the stylized facts of equity returns (volatility clustering, fat tails, negative skew, and a factor structure that makes names co-move), together with the reference data, shares outstanding, free float and corporate actions that go with them.
from beacon.synthetic import SyntheticConfig, generate
dataset = generate(SyntheticConfig(assets=64, seed=1))
fetcher = dataset.fetcher()
Or from the command line, writing a store the server auto-loads:
python -m beacon.synthetic --assets 512 --start 2019-12-31 --seed 42
A generated store can later be extended to a new date without changing its
history: extend(path), or python -m beacon.synthetic --extend PATH.
Nothing generated here resembles a real company: names are Company A …
and every ticker carries a CMP prefix, which makes a collision with a real
listing impossible rather than merely improbable.
This is not beacon.testing.dataset, which is a tiny frozen fixture whose
exact values chart baselines depend on. See dataset.py for why the two must
stay apart.
SyntheticConfig
dataclass
¶
SyntheticConfig(
assets: int = DEFAULT_ASSETS,
start: str = DEFAULT_START,
end: str = DEFAULT_END,
seed: int = DEFAULT_SEED,
risk_free_rate: float = DEFAULT_RISK_FREE_RATE,
equity_premium: float = DEFAULT_EQUITY_PREMIUM,
currency: str = universe.DEFAULT_CURRENCY,
delisting_rate: float = listings.ANNUAL_DELISTING_RATE,
listing_rate: float = listings.ANNUAL_LISTING_RATE,
features: bool = True,
calendar: str = DEFAULT_CALENDAR,
)
What to generate.
Attributes:
| Name | Type | Description |
|---|---|---|
assets |
int
|
How many names. |
start |
str
|
First date of the panel. |
end |
str
|
Last date of the panel. |
seed |
int
|
Everything random is drawn from this, so the same seed and the same dates produce the same dataset. |
risk_free_rate |
float
|
Annualised, the base of the CAPM drift. |
equity_premium |
float
|
Annualised excess return on a beta-one name. |
currency |
str
|
Not used: each name's currency comes from its listing
region (see |
delisting_rate |
float
|
Annualised hazard of a name leaving the universe. Zero gives a constant, survivorship-biased universe. |
listing_rate |
float
|
Annualised hazard of a name having joined partway through rather than at the start. |
features |
bool
|
Generate fundamental ratios and a little alternative data. On by default and cheap (about a tenth of the market panel), because a dataset with nothing to screen on cannot exercise a feature rule, and a store that silently lacks one is harder to diagnose than one that costs a few hundred thousand rows. |
calendar |
str
|
The exchange MIC whose sessions the panel has bars on.
Defaults to One calendar for the whole dataset, including the names quoted in another currency: a universe whose venues genuinely disagree about sessions is a real question and not this one's. A name whose own exchange would have been shut still gets a bar, which is the lesser of the two wrongs: the alternative is a hole in the panel that every consumer would have to tell apart from missing data. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
SyntheticDataset
dataclass
¶
SyntheticDataset(
market: MarketData,
reference: ReferenceData,
actions: CorporateActions,
universe: DataFrame,
returns: DataFrame,
features: FeatureData = FeatureData.empty(),
)
A generated universe and everything drawn from it.
Attributes:
| Name | Type | Description |
|---|---|---|
market |
MarketData
|
OHLCV, shares outstanding and free float. |
reference |
ReferenceData
|
Names, classification, exchange and currency. |
actions |
CorporateActions
|
Dividends and splits, matching the price path. |
universe |
DataFrame
|
The per-name parameters behind the draw. Exposed because the targets are what a statistical check should compare against: the realised figures are a sample from them, not the same thing. |
returns |
DataFrame
|
The economic total returns the prices were built from. Not
recoverable from |
features |
FeatureData
|
Fundamental ratios and alternative data, derived from the prices above so a valuation screen and a price screen agree. |
Extension
dataclass
¶
What an extension added.
Attributes:
| Name | Type | Description |
|---|---|---|
first |
str | None
|
The first new session, or None if there was nothing to add. |
last |
str | None
|
The last new session, or None. |
sessions |
int
|
How many sessions were added. |
listed |
int
|
How many new names listed. |
delisted |
int
|
How many names left. |
generate ¶
generate(
config: SyntheticConfig | None = None,
progress: Progress = _silent,
) -> SyntheticDataset
Generate a synthetic dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
SyntheticConfig | None
|
What to generate; defaults to 512 names over a fixed five-year window. |
None
|
progress
|
Progress
|
Called at each stage with the fraction done and what is happening. The fractions are rough shares of the time each stage takes at the default size, not exact. |
_silent
|
Returns:
| Name | Type | Description |
|---|---|---|
SyntheticDataset |
SyntheticDataset
|
The panels and the parameters behind them. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the calendar has no sessions in the window. |
write ¶
Generate a dataset and write it as a data store.
The store also keeps what an extension needs to carry it on later (see
beacon.synthetic.extend).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
SyntheticConfig
|
What to generate. |
required |
path
|
Path
|
Store directory, created if absent. |
required |
progress
|
Progress
|
As for :func: |
_silent
|
Returns:
| Name | Type | Description |
|---|---|---|
Path |
Path
|
The directory written. |
dataset ¶
Assembling a synthetic universe into the containers the rest of py-beacon reads.
This is the layer that turns four panels into a DataFetcher, and the one
place that knows the whole dataset is meant to be mutually consistent: the
reference data names the same identifiers the market data prices, the action
history refers only to those identifiers, and the shares outstanding move on
exactly the dates the split actions record.
This is not beacon.testing.dataset¶
They are deliberately different things and neither should grow into the other.
beacon.testing.dataset is a tiny frozen fixture: six names, three years of
sessions, and price paths built from + and * alone so the numbers are
bit-identical on every platform. Chart baselines and unit tests depend on those
exact values,
so it must never change.
This module is a generator: hundreds of names, years of history, drawn from a model whose parameters are meant to be tuned as the model improves. Nothing should assert on its exact values. Use the fixture when a test needs a known number, and this when something needs to look like a market.
SyntheticConfig
dataclass
¶
SyntheticConfig(
assets: int = DEFAULT_ASSETS,
start: str = DEFAULT_START,
end: str = DEFAULT_END,
seed: int = DEFAULT_SEED,
risk_free_rate: float = DEFAULT_RISK_FREE_RATE,
equity_premium: float = DEFAULT_EQUITY_PREMIUM,
currency: str = universe.DEFAULT_CURRENCY,
delisting_rate: float = listings.ANNUAL_DELISTING_RATE,
listing_rate: float = listings.ANNUAL_LISTING_RATE,
features: bool = True,
calendar: str = DEFAULT_CALENDAR,
)
What to generate.
Attributes:
| Name | Type | Description |
|---|---|---|
assets |
int
|
How many names. |
start |
str
|
First date of the panel. |
end |
str
|
Last date of the panel. |
seed |
int
|
Everything random is drawn from this, so the same seed and the same dates produce the same dataset. |
risk_free_rate |
float
|
Annualised, the base of the CAPM drift. |
equity_premium |
float
|
Annualised excess return on a beta-one name. |
currency |
str
|
Not used: each name's currency comes from its listing
region (see |
delisting_rate |
float
|
Annualised hazard of a name leaving the universe. Zero gives a constant, survivorship-biased universe. |
listing_rate |
float
|
Annualised hazard of a name having joined partway through rather than at the start. |
features |
bool
|
Generate fundamental ratios and a little alternative data. On by default and cheap (about a tenth of the market panel), because a dataset with nothing to screen on cannot exercise a feature rule, and a store that silently lacks one is harder to diagnose than one that costs a few hundred thousand rows. |
calendar |
str
|
The exchange MIC whose sessions the panel has bars on.
Defaults to One calendar for the whole dataset, including the names quoted in another currency: a universe whose venues genuinely disagree about sessions is a real question and not this one's. A name whose own exchange would have been shut still gets a bar, which is the lesser of the two wrongs: the alternative is a hole in the panel that every consumer would have to tell apart from missing data. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
SyntheticDataset
dataclass
¶
SyntheticDataset(
market: MarketData,
reference: ReferenceData,
actions: CorporateActions,
universe: DataFrame,
returns: DataFrame,
features: FeatureData = FeatureData.empty(),
)
A generated universe and everything drawn from it.
Attributes:
| Name | Type | Description |
|---|---|---|
market |
MarketData
|
OHLCV, shares outstanding and free float. |
reference |
ReferenceData
|
Names, classification, exchange and currency. |
actions |
CorporateActions
|
Dividends and splits, matching the price path. |
universe |
DataFrame
|
The per-name parameters behind the draw. Exposed because the targets are what a statistical check should compare against: the realised figures are a sample from them, not the same thing. |
returns |
DataFrame
|
The economic total returns the prices were built from. Not
recoverable from |
features |
FeatureData
|
Fundamental ratios and alternative data, derived from the prices above so a valuation screen and a price screen agree. |
generate ¶
generate(
config: SyntheticConfig | None = None,
progress: Progress = _silent,
) -> SyntheticDataset
Generate a synthetic dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
SyntheticConfig | None
|
What to generate; defaults to 512 names over a fixed five-year window. |
None
|
progress
|
Progress
|
Called at each stage with the fraction done and what is happening. The fractions are rough shares of the time each stage takes at the default size, not exact. |
_silent
|
Returns:
| Name | Type | Description |
|---|---|---|
SyntheticDataset |
SyntheticDataset
|
The panels and the parameters behind them. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the calendar has no sessions in the window. |
write ¶
Generate a dataset and write it as a data store.
The store also keeps what an extension needs to carry it on later (see
beacon.synthetic.extend).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
SyntheticConfig
|
What to generate. |
required |
path
|
Path
|
Store directory, created if absent. |
required |
progress
|
Progress
|
As for :func: |
_silent
|
Returns:
| Name | Type | Description |
|---|---|---|
Path |
Path
|
The directory written. |
settings_of ¶
The generator state a store made with config starts from.
extend ¶
Extend a generated store to a later date, without changing its history.
from beacon.synthetic.extend import extend
extend(Path("my-store")) # up to today
extend(Path("my-store"), end="2026-06-30")
or from the command line, python -m beacon.synthetic --extend my-store.
Why not regenerate with a later end date¶
The generator draws every number from one random stream sized by the whole date range. A later end date changes every past price, and with it every index and backtest saved against the store.
What an extension does¶
It carries the market on from where the store stops:
- every listed name from its last close and share count, with the
volatility, factor loadings, alpha and dividend yield it was generated with
(saved beside the data; see
beacon.synthetic.state); - each FX pair from its last rate;
- new listings and delistings at the store's rates, timed by the same crisis intensity as before, with new names continuing the ticker sequence;
- dividends and split reviews on the same calendar rules, without a second ex-date in a month that already had one;
- features, including quarters that ended before the extension but are reported inside it.
The new days draw from a random stream derived from the store's seed and the dates of the new sessions, so the same store extended to the same date always gives the same data.
What changes and what does not¶
Market and feature rows already in the store are never touched: the new rows are appended to the files, and the bytes already there stay as they were. Three kinds of record do change, because they describe a state rather than a day: a name that delists gets its end date and status, a dividend whose pay date arrives becomes paid, and each listed name's next earnings date moves forward.
Crises are the real, dated episodes in beacon.synthetic.regimes, so days
after the last of them are calm, as they would be in a store generated over
the same dates.
Extension
dataclass
¶
What an extension added.
Attributes:
| Name | Type | Description |
|---|---|---|
first |
str | None
|
The first new session, or None if there was nothing to add. |
last |
str | None
|
The last new session, or None. |
sessions |
int
|
How many sessions were added. |
listed |
int
|
How many new names listed. |
delisted |
int
|
How many names left. |
extend ¶
Extend a generated store to end, keeping everything already in it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Path
|
The store folder. |
required |
end
|
str | date | None
|
The last date to extend to; today if omitted. A date with no new session (a weekend, or a date the store already reaches) leaves the store as it was. |
None
|
progress
|
Progress
|
Called at each stage with the fraction done and what is happening. |
_silent
|
Returns:
| Name | Type | Description |
|---|---|---|
Extension |
Extension
|
What was added. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the store holds no generator state, which is the case for anything not generated by py-beacon 0.2.0 or later. |
features ¶
Synthetic features: fundamental ratios and a little alternative data.
A small set, deliberately. The point is to have something to screen on and something to exercise the point-in-time path with, not to simulate a data vendor. Four fundamental ratios and two alternative series is enough for a universe filter, an index rule and a look-ahead test, and cheap enough that every generated store can carry it.
Coherent with the price series, not drawn beside it¶
The trap this module exists to avoid.
A price-to-earnings ratio drawn independently of the prices in the same
dataset contradicts them: screen on pe_ratio < 15 and screen on price, and
the two disagree about the same company. Worse, the disagreement is invisible
until somebody checks, and by then it has been believed.
So the ratios are derived from the generated prices:
eps = close / pe_ratio (so pe x eps == price, exactly)
book_value = close / pb_ratio
d/e = drawn per name, by sector: the one genuinely independent of
price, because leverage is a balance-sheet fact
The multiple is what gets drawn, sector by sector, and the per-share figure follows from it. That ordering is what makes them consistent by construction rather than by luck.
Sentiment and page views are anchored the same way: sentiment tracks recent returns, because it does, and page views scale with size and spike on large moves. A name nobody has heard of does not trend on a quiet day.
The announcement lag varies, and that is not decoration¶
DATE holds when a value became knowable (beacon.data.features), so a
fundamental for the quarter ending 31 March is published somewhere in the
following weeks. A constant lag would make every look-ahead test pass
whether or not the accessor was correct: with all values 45 days late, any
off-by-one still lands in the same gap. The lag is drawn per name per quarter,
so a test standing on a given date sees a genuinely ragged edge.
Coverage is deliberately incomplete¶
Real fundamentals are missing for some names and some quarters, and
alternative datasets cover a fraction of a universe, mostly the large,
visible names. Generating a complete grid would make the missing-coverage
behaviour in FeatureRule untestable against this data, and would overstate
what an alternative vendor sells.
build ¶
build(
universe: DataFrame,
prices: DataFrame,
returns: DataFrame,
rng: Generator,
fundamentals: bool = True,
alternative: bool = True,
) -> pd.DataFrame
Generate the feature table for a panel.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
universe
|
DataFrame
|
Output of |
required |
prices
|
DataFrame
|
Wide close prices, dates by identifier. |
required |
returns
|
DataFrame
|
Wide daily returns, for the sentiment anchor. |
required |
rng
|
Generator
|
Seeded generator. |
required |
fundamentals
|
bool
|
Include the ratio set. |
True
|
alternative
|
bool
|
Include sentiment and page views. |
True
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: Long-form feature rows, ready for |
carry_on ¶
carry_on(
stored: DataFrame,
universe: DataFrame,
joined: Index,
closes: DataFrame,
after: Timestamp,
rng: Generator,
) -> pd.DataFrame
Feature rows that become known after after, for an extension.
A name keeps the multiples and coverage its stored rows show. A quarter that ended before after but had not been reported by then is reported now, with its lag drawn from what remains of the usual range, so no row lands on a date the store has already passed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
stored
|
DataFrame
|
The store's feature rows. |
required |
universe
|
DataFrame
|
Every name that can report, with SECTOR and market_cap. |
required |
joined
|
Index
|
The names listing in this extension, which draw their coverage and multiples as the originals did. |
required |
closes
|
DataFrame
|
Wide closes from |
required |
after
|
Timestamp
|
The last date already in the store. |
required |
rng
|
Generator
|
Seeded generator. |
required |
fx ¶
Exchange rates, generated as market data so the FX paths are actually exercised.
DataFetcher.fetch_fx_rates looks a pair up as an ordinary market-data
identifier named f"{from}{to}": EURUSD converts euros into dollars.
Nothing special: a currency pair is a row set like any other, which is why the
calculator and the corporate-action handler can convert without knowing where
the rate came from.
Why a rate is not a price¶
Two differences, and both matter to what the data is for.
Volatility is far lower. A major pair realises 7-11% annualised against 25-35% for a single equity. A generator that gave currencies equity-like volatility would make every unhedged exposure look like the dominant risk in a global portfolio, which is the opposite of true.
Not everything floats. The Hong Kong dollar runs inside a band the monetary authority defends, and realises well under 1%. Modelling it like the others would manufacture a diversification benefit that does not exist, and an optimiser told it can hedge HKD risk would allocate to a trade nobody makes.
Flight to quality¶
Crises are not currency-neutral. Money moves into dollars when volatility spikes, so every pair here drifts down against USD in proportion to how far the regime has lifted correlations. It is the same intensity series the return process uses, so the currency move lines up with the equity drawdown rather than wandering off on its own, which is what makes a hedged-versus-unhedged comparison over a crisis show anything at all.
build ¶
build(
dates: DatetimeIndex,
rng: Generator,
regimes: tuple[Regime, ...] = CRISES,
start_from: dict[str, float] | None = None,
) -> pd.DataFrame
Generate one rate series per non-base currency.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dates
|
DatetimeIndex
|
The panel's business days. |
required |
rng
|
Generator
|
Seeded generator. |
required |
regimes
|
tuple[Regime, ...]
|
Dated episodes, for the flight-to-quality drift. |
CRISES
|
start_from
|
dict[str, float] | None
|
The rate each pair starts from, where known. An extension passes the last stored rates, so each path carries on from where it stopped rather than from the pair's usual level. |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: Long-form, with IDENTIFIER/DATE and the rate in both |
DataFrame
|
RATE and CLOSE, ready to be concatenated onto the equity market data. |
listings ¶
When a company joins the universe and when it leaves it.
A generated name may list partway through the panel and may leave before it ends. A universe where every name lists on day one and never leaves is the definition of a survivorship-biased dataset, and it makes a whole class of behaviour unreachable: additions and deletions, the divisor adjustment each one needs, a backtest that has to dispose of a holding that stopped trading, and point-in-time universe resolution ("who was in the index then", answered from history rather than from today's list).
It also makes the bias itself unmeasurable, which is the point. An index built only from the names that survived to the end of the panel outperforms one built as it went along, because the losers were quietly removed from the sample before the question was asked. A dataset where that difference is zero cannot demonstrate the single most common error in backtesting.
The rates are real, the events are not¶
Roughly 3% of a large-cap index leaves per year and roughly 3% joins: acquisitions, failures, and index-committee decisions all together. Over ten years that is about a quarter of the universe turning over, which is enough to matter and not so much that the panel becomes unrecognisable.
Failures cluster, and that is the whole point¶
A constant hazard would put delistings evenly across the panel, and the bias a point-in-time index suffers would be a slow drip. Real companies fail together, in exactly the windows where the index is already falling, which is what makes survivorship bias large rather than merely present. The hazard here is scaled by the same regime intensity the return process uses, so delistings pile into 2001, 2008 and 2020.
Listings do the opposite: nobody floats a company into a collapsing market, so new issuance is suppressed when the intensity is high. That asymmetry is why a crisis shrinks a universe rather than churning it.
draw ¶
draw(
count: int,
dates: DatetimeIndex,
rng: Generator,
delisting_rate: float = ANNUAL_DELISTING_RATE,
listing_rate: float = ANNUAL_LISTING_RATE,
regimes: tuple[Regime, ...] = CRISES,
alpha: ndarray | None = None,
) -> pd.DataFrame
Draw a listed life for every name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
count
|
int
|
How many names. |
required |
dates
|
DatetimeIndex
|
The panel's business days. |
required |
rng
|
Generator
|
Seeded generator. |
required |
delisting_rate
|
float
|
Annualised hazard of leaving. Zero keeps every name to the end of the panel. |
ANNUAL_DELISTING_RATE
|
listing_rate
|
float
|
Annualised hazard of having joined partway through. Zero lists every name from the start. |
ANNUAL_LISTING_RATE
|
regimes
|
tuple[Regime, ...]
|
Dated episodes, for the clustering. |
CRISES
|
alpha
|
ndarray | None
|
Each name's alpha. When given, delistings tilt towards the weakest names without changing how many there are. |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: |
DataFrame
|
Timestamps; |
joining ¶
joining(
count: int,
dates: DatetimeIndex,
rng: Generator,
listing_rate: float = ANNUAL_LISTING_RATE,
regimes: tuple[Regime, ...] = CRISES,
) -> pd.DatetimeIndex
When each of count new names lists, over an extension's dates.
Timed like the original listings: suppressed in a crisis, so issuance dries up when the market falls.
leaving ¶
leaving(
alpha: ndarray,
dates: DatetimeIndex,
rng: Generator,
delisting_rate: float = ANNUAL_DELISTING_RATE,
regimes: tuple[Regime, ...] = CRISES,
) -> pd.DatetimeIndex
When each listed name leaves over an extension's dates, NaT if it stays.
The same hazard as the original delistings, clustered in crises and tilted towards the weakest names by their alpha.
prices ¶
Turning a return panel into the market data a client actually reads: OHLCV, shares outstanding and free float, with dividends and splits folded into the price path rather than bolted on beside it.
The price is not the return path¶
A return series compounds into a total-return index. A stored CLOSE is
neither that nor a clean geometric path: it drops on an ex-dividend date and it
halves on a split, and every downstream calculation (trailing yield, the
divisor adjustment, a split-adjusted chart) exists to undo one of those two
things. Generating prices that never do either would produce a dataset on which
none of that code could be exercised, which is the opposite of the point.
So the stored close is built as
close = initial · Π (1 + r) · (1 - q) / cumulative_split
where q is the ex-date drop as a fraction of price, and the corporate-action
history records exactly the dividends and splits that appear in it. The two are
generated together, so they cannot disagree.
Reconstructing the economic path¶
Multiplying CLOSE by the cumulative split ratio and adding dividends back
recovers the return panel this module was handed. A test asserts it, because
"coherent" is otherwise a claim rather than a property.
OHLC¶
Open comes off the previous close through an overnight gap; high and low are
pushed out from whichever of open and close is the extreme. Constructing them
that way makes H >= max(O, C) and L <= min(O, C) true by arithmetic
rather than by a repair pass afterwards: a clamp that fixes violations after
the fact is a clamp somebody eventually has to trust.
Splits are applied to all four prices at once, so the intraday relationships survive a split day intact.
Volume¶
Log-normal, scaled by market capitalisation so a mega-cap trades more than a small-cap, and pushed up on days when the move was large: the well-documented volume/volatility relationship. Without the second part, ADV would be a constant with noise on it and a liquidity screen built on it would never bind.
dividend_dates ¶
The ex-dividend dates inside a panel's date range.
The first business day on or after the 15th of each dividend month, so every ex-date is a day the panel actually has a price for.
review_dates ¶
The annual split-review dates: the first business day of each year.
The first year is skipped: a name cannot split before the panel starts, and reviewing on day one would split names purely for having been drawn an expensive opening price.
build ¶
build(
universe: DataFrame,
returns: DataFrame,
rng: Generator,
block_size: int = BLOCK_SIZE,
calendar: DatetimeIndex | None = None,
) -> tuple[pd.DataFrame, pd.DataFrame]
Build the market-data panel and the corporate-action history.
Names are processed in blocks. This is where the memory actually goes:
thirteen full (days x names) panels are alive between the return input
and the long-form output (the dividend drops, the multiplicative path,
the split factor, the pre-drop price, the dividend amounts, the close, the
share count, and five OHLCV frames). Blocking bounds all of them at the
block, leaving the output frame as the floor.
Nothing here couples one name to another: dividend dates come from the calendar, and a split is triggered by that name's own quoted price. So a block is the same computation on fewer columns, not an approximation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
universe
|
DataFrame
|
Output of |
required |
returns
|
DataFrame
|
Date-indexed total returns, one column per name. |
required |
rng
|
Generator
|
Seeded generator. |
required |
block_size
|
int
|
How many names to process at a time. Affects peak memory and nothing else. |
BLOCK_SIZE
|
calendar
|
DatetimeIndex | None
|
The sessions the dividend and split-review schedules are read from. Defaults to the panel's own dates. An extension passes sessions reaching back before its first date, so a month or year the existing data already began is not given a second ex-date or review. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
tuple |
DataFrame
|
Long-form market data with IDENTIFIER/DATE and OHLCV plus |
DataFrame
|
SHARES_OUTSTANDING and FREE_FLOAT, and a long-form action history. |
profiles ¶
The reference fields a profile view expects: identifiers, classification depth, and corporate facts.
The universe generator produces what the engine needs to calculate: sector, currency, exchange, share count. A client showing an instrument's profile needs a good deal more, and every field it cannot fill renders as a dash. This generates the rest.
Coherent with what is already generated¶
The same rule the synthetic features follow. A generated fact that contradicts another generated fact is worse than an absent one, because the contradiction is invisible until somebody checks and by then it has been believed.
So:
trading_statusfollows the listings model: a name delisted in 2021 is not "Active", and a screen on status has to agree with one on datesdividend_frequencyfollows the dividend yield actually generated. A name that pays nothing is not "Quarterly"; saying so would make a dividend-frequency screen select names with no dividendsipo_dateis the listing date, not a second date drawn beside it- the GICS levels nest: sub-industry inside industry inside industry group inside sector. A four-level hierarchy whose levels do not contain each other is decoration, and any roll-up computed from it would be wrong
The identifiers are structurally valid¶
isin, cusip and sedol carry their real check digits, and figi its real
shape. Not for authenticity (nothing here is a real security), but because a
client that validates an identifier before using it would otherwise reject the
whole store, and an ISIN is the field somebody is most likely to parse.
A wrong check digit is also the kind of thing that works until the day something checks it, which is the failure mode worth spending twenty lines to avoid.
build ¶
Generate the profile columns for a universe.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
universe
|
DataFrame
|
Output of |
required |
rng
|
Generator
|
Seeded generator. |
required |
as_of
|
Timestamp
|
The panel's last date, for |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: Indexed by identifier, one column per profile field. |
next_earnings ¶
The next reporting date, forward of the panel's end.
regimes ¶
Market regimes: the crises a stationary model cannot produce.
The return process in returns.py is stationary by construction. Its
volatility clusters, its tails are fat and its correlations are stable: all
true of markets on average, and all wrong about the periods that matter most.
Twenty-five years of equity history is not a draw from one distribution. It contains three or four episodes where volatility trebles, drift turns sharply negative, and (the part that matters) correlations rise toward one. Diversification stops working precisely when it is most wanted, and a model without that overstates how much a spread portfolio protects you.
What a regime does¶
Each regime scales three things over a dated window:
- volatility: a multiplier on the market factor, so every name inherits it through its own beta rather than being scaled directly
- drift: an annualised amount added to the market's return over the window
- correlation: the market factor's share of total variance rises, which is what makes names move together
The third is the one worth stating twice. Raising volatility alone produces a big drawdown that a diversified portfolio still cushions. Raising the market's share of variance is what removes the cushion.
Shape, not a step function¶
A crisis does not begin at midnight. Each window is ramped in and out with a raised-cosine taper over a fraction of its length, so volatility builds and subsides. A rectangular window produces a discontinuity in realised volatility that shows up as an obviously artificial jump on any chart.
The dates are real, the paths are not¶
The windows below are the actual episodes. What happens inside them is still generated: this is not a replay of 2008, and nothing here reproduces any real security's price. It is a synthetic market that has crises where the real one did, which is what makes a backtest over it exercise the code paths a calm market never reaches.
Regime
dataclass
¶
Regime(
name: str,
start: str,
end: str,
volatility: float = 1.0,
drift: float = 0.0,
correlation: float = 0.0,
ramp: float = 0.25,
)
One dated market episode.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
What it was. |
start |
str
|
First date of the episode. |
end |
str
|
Last date. |
volatility |
float
|
Multiplier on the market factor's volatility at peak intensity. 1.0 is calm. |
drift |
float
|
Annualised return added to the market over the window, at peak. Negative for a crisis. |
correlation |
float
|
How far the market's share of total variance moves toward one, from 0 (unchanged) to 1 (everything is the market). This is the diversification failure. |
ramp |
float
|
Fraction of the window spent easing in and out. |
intensity_series ¶
Per-date intensity for every regime touching a panel.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dates
|
DatetimeIndex
|
The panel's business days. |
required |
regimes
|
tuple[Regime, ...]
|
Episodes to apply. |
CRISES
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: Date-indexed, one column per regime, values in [0, 1]. |
DataFrame
|
Regimes that fall entirely outside the panel are omitted, so a |
DataFrame
|
window that touches none of them carries no columns and behaves as a |
DataFrame
|
stationary market. |
market_multipliers ¶
market_multipliers(
dates: DatetimeIndex,
regimes: tuple[Regime, ...] = CRISES,
) -> tuple[np.ndarray, np.ndarray, np.ndarray]
How the market factor is scaled on each date.
Overlapping regimes take the most severe value rather than compounding. The 2008 panic is the peak of the financial crisis, not a second shock on top of it, and multiplying the two gave a 5.5x volatility multiplier and a combined -112% annual drift: a market that fell further in one quarter than it has in any real decade.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dates
|
DatetimeIndex
|
The panel's business days. |
required |
regimes
|
tuple[Regime, ...]
|
Episodes to apply. |
CRISES
|
Returns:
| Name | Type | Description |
|---|---|---|
tuple |
ndarray
|
Volatility multiplier, daily drift adjustment, and the share of |
ndarray
|
variance moved toward the market factor, each per date. |
describe ¶
One line per regime the panel covers, for a caller to log or print.
regions ¶
Listing venues: where a generated company trades, and in what currency.
A single-currency universe cannot exercise most of what py-beacon does with
currency. The calculator converts every market value through
fetch_fx_rates, corporate actions convert their cash amounts the same way,
and both paths are dead code against a universe that is entirely USD: they
run, they multiply by 1.0, and nothing they could get wrong would show up.
What a region is here¶
A listing venue with one currency, one exchange and a share of the world. Deliberately not a country: "Europe" prices in EUR and lists on XETR, which is true of enough of the continent to be useful and wrong about Switzerland. Modelling countries properly would mean a currency per country and a correlation structure between them, which is a bigger claim than generated data should make.
The weights are roughly MSCI ACWI¶
The United States really is around 60% of global equity market capitalisation. A synthetic global universe that splits evenly across regions would make every FX exposure look far more material than it is, and a "global" index built on it would behave nothing like one built on the real thing.
Prices are quoted in local currency¶
A name's initial_price is in its own currency, and its market cap is
converted from the USD-scale draw so the size distribution stays global
while the quoted price stays local. Without that conversion a 5 billion
company would mean five billion of whatever it happens to be quoted in, and
the biggest companies in the universe would be an artefact of the exchange
rate.
The price range is deliberately not localised. Real JPY quotes run to thousands of yen, but the split rule, the tick size and the four-decimal rounding are all calibrated to the 12-480 band, and localising the range would mean localising all three. The prices are local-currency amounts drawn from a common band, which is a limitation worth stating rather than hiding.
Region
dataclass
¶
Region(
name: str,
currency: str,
exchange: str,
country: str,
weight: float,
rate: float,
volatility: float,
)
One listing venue.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
What the reference data reports as REGION. |
currency |
str
|
ISO code every name listed here is quoted in. |
exchange |
str
|
MIC the names carry. |
country |
str
|
ISO 3166-1 alpha-2 of the listing venue. Distinct from
|
weight |
float
|
Share of the universe, roughly MSCI ACWI. |
rate |
float
|
Units of |
volatility |
float
|
Annualised volatility of the exchange rate. |
assign ¶
Which region each name lists in.
Allocated by quota rather than drawn independently, so a 20-name universe still contains a non-US name and the realised weights match the targets at any size. An independent draw at 0.02 leaves Australia absent from most small universes, which is the size a test uses.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
count
|
int
|
How many names. |
required |
rng
|
Generator
|
Seeded generator, used only to shuffle the assignment so region does not correlate with position, and therefore not with sector, which is assigned round-robin. |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
np.ndarray: One region index per name. |
pairs ¶
The FX pairs a generated panel needs, as (identifier, rate, volatility).
One per non-base currency, named the way fetch_fx_rates looks it up:
f"{from}{to}", so converting EUR into USD reads EURUSD. The rate
is inverted from the region's, which stores units per dollar.
domiciles ¶
Where each name is incorporated.
Usually the listing country. A minority are incorporated elsewhere, drawn from the destinations that listing venue actually uses.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
assigned
|
ndarray
|
One region index per name. |
required |
rng
|
Generator
|
Seeded generator. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
list |
list[str]
|
ISO 3166-1 alpha-2 per name. |
frame ¶
Region, currency and exchange per name, given the assignment.
returns ¶
The return process: a factor model with GJR-GARCH volatility and fat tails.
Gaussian random walks are the obvious way to make fake prices and they are wrong in every way that matters to what this data is for. They have no volatility clustering, so a drawdown chart shows nothing recognisable; no fat tails, so a risk model estimated from them is never stressed; and no cross-correlation, so an optimiser sees a diversification opportunity that no market offers and every constraint binds strangely. Each stylized fact below is here because leaving it out breaks a view somebody has to look at.
The model¶
For name i on day t:
r[i,t] = mu[i] + b[i]·f_market[t] + g[i]·f_sector(i)[t]
+ h[i]·f_region(i)[t] + e[i,t]
Four kinds of independent series, each a GJR-GARCH(1,1) process with standardised Student-t innovations:
- The market factor: one series everything loads on. This is what makes names co-move, and giving it its own GARCH is what makes them co-move more in a crisis, which is when correlation matters.
- Sector factors: one per GICS sector, so two banks resemble each other more than a bank resembles a utility.
- Region factors: one per listing venue, so two names listed in Tokyo move together for reasons that have nothing to do with their industry.
- Idiosyncratic noise: one per name.
Loadings are set from a variance budget rather than drawn directly, so a name's total volatility is a target that is hit rather than an outcome to be discovered. With the market at ~34% of variance, the sector at ~16% and the region at ~8%, same-sector pairs correlate near 0.50 and cross-sector pairs near 0.35, and the average across the universe lands near 0.39.
Why GJR rather than plain GARCH¶
sigma2[t] = omega + (alpha + lam·1[e<0])·e[t-1]^2 + beta·sigma2[t-1]
The lam term makes a fall raise tomorrow's volatility more than a rise of the
same size does. That asymmetry is the leverage effect, and it is what produces
negative skew; without it, simulated returns are symmetric and a drawdown
looks like an upswing turned upside down.
Persistence is alpha + lam/2 + beta, drawn in 0.94-0.99: high enough that
quiet and turbulent periods last for months rather than days, below one so the
process is stationary and the unconditional variance is defined.
What is deterministic here, and what is not¶
Given a seed, this module produces the same numbers on the same machine and
the same numpy. It does not guarantee bit-identical output across operating
systems: standard_t and the exponentials behind it run through the platform's
libm, which is free to differ in the last bit. That is a deliberate limit:
avoiding transcendentals entirely would mean abandoning Student-t innovations
and log-normal volume, which is most of what makes this data worth generating.
Reproducibility is tested per-platform; the statistical acceptance checks are
tolerance-based and hold anywhere.
standardised_t ¶
Negatively skewed Student-t draws, rescaled to unit variance.
A raw t has variance nu/(nu-2), so feeding it into a GARCH recursion
unscaled would inflate every variance by that factor and make the target
volatility wrong by 40% at four degrees of freedom.
The two-piece step stretches the downside and compresses the upside, then the whole thing is re-standardised: the stretch changes both the mean and the variance, and leaving either uncorrected would show up as a spurious drift and a missed volatility target.
simulate_gjr ¶
simulate_gjr(
steps: int,
target_variance: ndarray,
persistence: ndarray,
innovations: ndarray,
) -> np.ndarray
Run the GJR-GARCH(1,1) recursion over pre-drawn innovations.
Vectorised across series and looped over time, which is the only way round: each day's variance depends on the day before, but every series can take its step together.
The innovations are an argument rather than drawn here, and that is what makes block generation possible. Every series in the recursion is independent of every other (the arithmetic is elementwise throughout), so running it over a slice of the universe gives bit-identical results to running it over all of it, provided each series sees the same innovations.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
steps
|
int
|
Days to return, after burn-in. |
required |
target_variance
|
ndarray
|
Unconditional daily variance per series. |
required |
persistence
|
ndarray
|
|
required |
innovations
|
ndarray
|
Shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
np.ndarray: Shape |
ndarray
|
unconditional variance equal to |
pin_realised_variance ¶
Rescale a factor so its realised variance equals its target.
Applied to the shared factors only, and it is not cosmetic. A GARCH series at persistence 0.98 with t(4.5) innovations has enormous dispersion in its sample variance: over five years one draw can realise nearly twice its unconditional level. For an idiosyncratic series that is a fact about one name. For the market factor, which every name loads on, it rescales the entire universe: a generated dataset could come out with every volatility at 43% and an average correlation of 0.73 instead of 29% and 0.40.
Only the overall level is pinned. The clustering, the fat tails and the leverage asymmetry are properties of the shape of the path and survive a single multiplicative rescaling untouched.
Idiosyncratic series are deliberately left alone, so a name's realised volatility still varies around its target the way a real one does.
simulate ¶
simulate(
universe: DataFrame,
dates: DatetimeIndex,
rng: Generator,
risk_free_rate: float,
equity_premium: float,
regimes: tuple[Regime, ...] = CRISES,
block_size: int = BLOCK_SIZE,
) -> pd.DataFrame
Simulate the total-return panel.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
universe
|
DataFrame
|
Output of |
required |
dates
|
DatetimeIndex
|
Business days to simulate. |
required |
rng
|
Generator
|
Seeded generator. |
required |
risk_free_rate
|
float
|
Annualised, the base of the CAPM expectation. |
required |
equity_premium
|
float
|
Annualised excess return on a beta-one name. |
required |
regimes
|
tuple[Regime, ...]
|
Dated crisis episodes to overlay. Empty for a stationary market. |
CRISES
|
block_size
|
int
|
How many names to simulate at a time. Affects peak memory and nothing else: the panel is identical at any block size, and a test holds that. |
BLOCK_SIZE
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: Date-indexed daily total returns, one column per name. |
state ¶
What a generated store keeps so it can be extended later.
The stored prices say where every name ended up, but not the parameters it was
drawn from: its target volatility, how its variance splits between the market,
its sector and its region, its alpha, its dividend yield. Continuing a name
without those would mean re-estimating them from a few years of one noisy
path, and the extension would behave like a different market. So a generated
store also saves them, in a synthetic folder beside the data:
settings.json: the seed, the rates and the calendar it was made with, and every extension since.universe.csv.gz: one row per name, with the parameters above and its listed life.
Nothing else reads these files. A store without them loads as normal; it just cannot be extended.
Settings
dataclass
¶
Settings(
seed: int,
assets: int,
start: str,
end: str,
calendar: str,
risk_free_rate: float,
equity_premium: float,
delisting_rate: float,
listing_rate: float,
features: bool,
extensions: list[dict[str, str]] = list(),
version: int = STATE_VERSION,
)
How a store was generated, and how far it has been extended.
Attributes:
| Name | Type | Description |
|---|---|---|
seed |
int
|
The seed it was generated with. Extensions derive their own random streams from it. |
assets |
int
|
The size of the universe it was generated with. New listings are sized against it. |
start |
str
|
The first date asked for. |
end |
str
|
The last date asked for, updated by each extension. |
calendar |
str
|
The exchange whose sessions it has bars on. |
risk_free_rate |
float
|
Annualised. |
equity_premium |
float
|
Annualised. |
delisting_rate |
float
|
Annualised hazard of a name leaving. |
listing_rate |
float
|
Annualised hazard of a name joining. |
features |
bool
|
Whether it carries features. |
extensions |
list[dict[str, str]]
|
Each extension, as the first and last new session. |
save ¶
Write a store's state beside its data.
load ¶
Read a store's state.
Raises:
| Type | Description |
|---|---|
ValueError
|
If the store has none, or it was written by a newer py-beacon. |
universe ¶
The static half of a synthetic universe: who the companies are.
Everything here is decided once per run and does not vary by date: names,
tickers, classification, and the per-name parameters the return process is
driven by. Splitting it out from the time series keeps one question separate
from the other: this module answers "what is in the universe", returns and
prices answer "what did it do".
Nothing here resembles a real company¶
Names are Company A … Company Z, then Company AA … in the
spreadsheet-column order, with tickers CMPA, CMPB, … CMPAA. The
CMP prefix is what makes a collision with a real listing impossible rather
than merely unlikely: a three-letter prefix on every symbol means no generated
ticker can ever equal a real one, whatever the universe grows to. A test
asserts it against a blocklist of well-known symbols anyway, because the
guarantee is only as good as the naming function that provides it.
Sector names are the eleven GICS sectors, which are a public taxonomy rather than anybody's property. Sub-industries are not: real GICS sub-industry names would imply a classification this data does not have, so they are generic segments within each sector.
The parameters, and where they come from¶
The figures below are approximate long-run statistics for US large- and mid-cap equity. They are targets for a generator, not estimates: the point is that a chart drawn from this data looks like a chart drawn from a market, so somebody reviewing a layout is reviewing it against realistic numbers.
ticker_suffix ¶
The spreadsheet-column label for a position: A, B, ... Z, AA, AB, ...
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
position
|
int
|
Zero-based index into the universe. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
The label, always at least one character. |
identifiers ¶
The tickers for a universe of a given size.
One function, used by every dataset in this package, so a name and its ticker cannot disagree between the market data and the reference data.
build ¶
build(
count: int,
rng: Generator,
dates: DatetimeIndex | None = None,
currency: str = DEFAULT_CURRENCY,
delisting_rate: float = listings.ANNUAL_DELISTING_RATE,
listing_rate: float = listings.ANNUAL_LISTING_RATE,
) -> pd.DataFrame
Draw the static universe.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
count
|
int
|
How many names. |
required |
rng
|
Generator
|
Seeded generator; every draw here comes from it, so the universe is a function of the seed alone. |
required |
dates
|
DatetimeIndex | None
|
The panel's business days. When given, each name draws a listed life over them; when omitted every name is listed for the whole panel, which is what callers that only want the static fields get. |
None
|
currency
|
str
|
Not used. A name's currency comes from its listing region, because a universe forced into one currency is the case the FX paths never exercise. Kept for callers that pass it. |
DEFAULT_CURRENCY
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: One row per name, indexed by identifier, carrying both |
DataFrame
|
the reference fields and the parameters the return process needs. |
newcomers ¶
Names that join after a store was generated, drawn like the originals.
Their tickers carry on the sequence (a universe of 512 names continues at position 512), and each takes a size rank drawn uniformly from the original universe, so newcomers are sized like the names already there rather than all small or all large.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
first_position
|
int
|
The position of the first newcomer. |
required |
count
|
int
|
How many. |
required |
rng
|
Generator
|
Seeded generator. |
required |
universe_size
|
int
|
The size of the universe the store was generated with, which sets the size profile. |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: As :func: |
DataFrame
|
draws. |
reference_frame ¶
reference_frame(
universe: DataFrame,
valid_from: str,
profile: DataFrame | None = None,
) -> pd.DataFrame
The reference dataset, long-form and ready for ReferenceData.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
universe
|
DataFrame
|
Output of :func: |
required |
valid_from
|
str
|
DATE_FROM for names listed since the panel began. A generated universe has no history of reclassification, so a name that was there at the start gets one record valid from the start rather than pretending to a change it never had. |
required |
profile
|
DataFrame | None
|
Optional profile columns from |
None
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
pd.DataFrame: One row per name, carrying the listed life. |
DataFrame
|
NaT for a name still listed at the end of the panel, which is what |
DataFrame
|
|
DataFrame
|
resolution drops a delisted name automatically. |