Expressions¶
Typed references to a datapoint, such as data.market.close, and the conditions built from them, used by expression rules, feature rules, filtered universes and backtest screens.
expressions ¶
Expressions: a typed, autocompleting way to refer to a datapoint.
from beacon.expressions import data
data.reference.sector == "Financials"
data.market.adv_3m > 1_000_000
An expression is a tree, not a value. It serialises into the params of a
stored rule, so a screen written in Python and one built in the client are the
same document.
All ¶
Bases: _Group
Every operand must pass.
Any_ ¶
Bases: _Group
At least one operand must pass.
Named with a trailing underscore so it does not shadow typing.Any, which
this module also uses.
Comparison ¶
Expression ¶
Base for everything that can be composed and serialised.
Subclasses supply to_dict; composition and the truth-value guard are
shared, so a new node type cannot forget either.
Field ¶
Bases: Expression
A named datapoint in a namespace.
namespace is the surface it came from (reference, market,
features, actions), and dataset narrows a feature to one TYPE, so
two vendors may both ship a field called revenue without collision.
A Field is an Expression so it composes, but on its own it says
nothing: comparing it is what produces something screenable.
key
property
¶
The datapoint this field names, as a plain hashable tuple.
What to use as a dict key or set member. A Field cannot safely serve
as one for two distinct-but-equal instances (see __hash__ above),
and quietly raising deep inside a resolver is the worst place to find
that out.
is_in ¶
One of a set of values.
Named is_in rather than in, which is a keyword, and deliberately
not spelled with __contains__: Python coerces that to a bool, so
x in field could never build a tree.
same_as ¶
Whether two fields name the same datapoint.
The plain == a field cannot offer, because == builds a tree.
Data ¶
The root: data.
Deliberately a small fixed set of namespaces. Unlike the fields inside
them, the datasets Beacon holds are a closed contract: a typo like
data.refrence.sector should fail at the attribute rather than build a
field in a namespace nothing will ever resolve.
Features ¶
The feature namespace, which nests by dataset type.
data.features.fundamentals.revenue, not data.features.revenue.
TYPE is what separates datasets sharing one table: revenue from a
vendor and revenue from a user's own model are different series.
Flattening them here would leave the API unable to say which it meant at
exactly the moment the user is choosing between them.
FeatureType ¶
Bases: Namespace
One feature dataset, such as data.features.fundamentals.
Wholly open: the fields a dataset carries are whatever was loaded, and a fixed list here would be wrong the first time somebody imported their own.
Namespace ¶
One dataset's fields, reached by attribute.
Declared names complete and are documented; anything else still resolves,
because a store may carry columns this list has never heard of. What is
not allowed is a private or dunder name, which would otherwise turn a
typo like data.market.__deepcopy__ into a field.
distinct_fields_in ¶
Every field an expression mentions, deduplicated, in the order written.
Deduplicated by Field.key rather than by putting the fields in a set:
two distinct Field objects naming the same datapoint have equal hashes,
so a set would compare them with __eq__, get a tree back, and raise.
fields_in ¶
Every field an expression mentions, in the order written.
What a caller needs to check coverage before running a screen, or to show which datapoints a saved definition depends on.
from_dict ¶
Rebuild an expression from to_dict output.
The inverse has to be exact rather than close: a stored definition is reloaded and re-run, and a screen that resolves differently after a round trip makes a backtest irreproducible in a way nothing would flag.
column_for ¶
The stored column a field names.
Upper case for market, reference and action columns, which are stored that
way; feature fields keep their case, because a feature's FIELD is
whatever the loader wrote and upper-casing it would stop matching.
catalogue ¶
Every datapoint an expression can name, listed from the loaded store.
GET /data/features/catalogue publishes the feature fields. This module lists
the rest (market, reference and corporate-action fields) alongside them, so a
client builds one field picker rather than one per dataset, and the picker
and the expression API cannot disagree about what exists.
Fields are read from the store, not only from the declarations in
namespaces.py: a store may carry columns the declarations do not know about,
and those are the columns a user loaded themselves. The declarations still
decide two things the data cannot say: which market fields are derived
(computed per request, so stored nowhere), and which action fields exist
(kind and status are computed when the API returns an action, not stored).
describe_fields ¶
Every field a client can offer, in picker order.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fetcher
|
DataFetcher
|
The loaded store. |
required |
Returns:
| Type | Description |
|---|---|
list[dict[str, Any]]
|
list[dict]: One entry per datapoint, each naming its namespace, its |
list[dict[str, Any]]
|
path, and whether it is derived. |
core ¶
The expression tree: fields, comparisons, and boolean composition.
A typed way to name a datapoint and say something about it:
data.reference.sector == "Financials"
data.market.adv_3m > 1_000_000
(data.features.fundamentals.pe_ratio < 15) & (data.market.adv_3m > 1e6)
Nothing here evaluates. Building an expression produces a tree, which is then either serialised into a stored index definition or resolved against one instrument on one date. That indirection is the whole point: an index definition is a document, and a screen that cannot be written down cannot be saved, reloaded, or sent to a client.
It compiles into the rule envelope¶
A rule in a stored pipeline is {"id", "type", "params"}. An expression does
not sit beside that: it becomes the params of a rule type (ExpressionRule),
so there is one representation of a pipeline rather than two that drift.
to_dict and from_dict here make that possible, and the round trip is exact
rather than approximate: a definition saved and reloaded screens identically,
so a backtest is reproducible.
Two hazards¶
__eq__ does not return a bool. That is what lets sector == "Financials"
build a tree, and it breaks three things Python assumes:
assert expression == xin a test would pass silently, asserting nothingexpression in [...]compares by__eq__and then takes a truth value- an object defining
__eq__loses__hash__unless it declares one
The first two are handled by __bool__ raising rather than returning a value,
which turns a silent wrong answer into an error that names the problem.
The third is handled by declaring __hash__, but only partly. CPython checks
is before == when looking up a set or dict entry, so reusing the same
field object in a set or an in [...] test works and never reaches __eq__.
Two distinct objects naming the same datapoint hash alike, fall through to
__eq__, get a tree back and raise. A Field is therefore a safe dict key only
when the same instance is reused. Field.key is the plain tuple to use
instead, and distinct_fields_in deduplicates with it rather than with a set of
fields.
and and or cannot be overloaded. Python evaluates a and b by taking
bool(a) and returning one operand or the other; there is no hook. So
(a == 1) and (b > 2) would quietly discard half the expression (the same trap
as in pandas). Use & and | to compose. __bool__ raises an error that says
exactly what to type instead.
Expression ¶
Base for everything that can be composed and serialised.
Subclasses supply to_dict; composition and the truth-value guard are
shared, so a new node type cannot forget either.
Field ¶
Bases: Expression
A named datapoint in a namespace.
namespace is the surface it came from (reference, market,
features, actions), and dataset narrows a feature to one TYPE, so
two vendors may both ship a field called revenue without collision.
A Field is an Expression so it composes, but on its own it says
nothing: comparing it is what produces something screenable.
key
property
¶
The datapoint this field names, as a plain hashable tuple.
What to use as a dict key or set member. A Field cannot safely serve
as one for two distinct-but-equal instances (see __hash__ above),
and quietly raising deep inside a resolver is the worst place to find
that out.
is_in ¶
One of a set of values.
Named is_in rather than in, which is a keyword, and deliberately
not spelled with __contains__: Python coerces that to a bool, so
x in field could never build a tree.
same_as ¶
Whether two fields name the same datapoint.
The plain == a field cannot offer, because == builds a tree.
Comparison ¶
All ¶
Bases: _Group
Every operand must pass.
Any_ ¶
Bases: _Group
At least one operand must pass.
Named with a trailing underscore so it does not shadow typing.Any, which
this module also uses.
from_dict ¶
Rebuild an expression from to_dict output.
The inverse has to be exact rather than close: a stored definition is reloaded and re-run, and a screen that resolves differently after a round trip makes a backtest irreproducible in a way nothing would flag.
distinct_fields_in ¶
Every field an expression mentions, deduplicated, in the order written.
Deduplicated by Field.key rather than by putting the fields in a set:
two distinct Field objects naming the same datapoint have equal hashes,
so a set would compare them with __eq__, get a tree back, and raise.
fields_in ¶
Every field an expression mentions, in the order written.
What a caller needs to check coverage before running a screen, or to show which datapoints a saved definition depends on.
namespaces ¶
Where data.market.close and data.features.fundamentals.revenue come from.
from beacon.expressions import data
data.reference.sector == "Financials"
data.market.market_cap > 1e9
data.features.fundamentals.revenue > 1e9
Imported from beacon.expressions, not from beacon¶
from beacon import data would be the obvious spelling and it cannot work:
beacon.data is already the data package, and importing any of its
submodules rebinds that name on the parent, so an expression root living there
would be whichever won the import race. beacon.data.market,
beacon.data.reference and beacon.data.actions raise an error pointing to
beacon.expressions, since those are where the mistake is plausible.
data is a description, not a dataset. It is a module-level symbol bound
to nothing, because an expression is written before there is anything to
evaluate it against: in a script, in a saved definition, in a client. Binding
it to a loaded store would make the import order matter and the same screen
mean different things in two processes.
Declared where there is a contract, open where there is not¶
The split is what makes autocomplete possible without a generation step.
Market, reference and action columns are declared. They are a documented
contract (beacon.data), so they are listed here and complete everywhere (in
Jupyter, in an IDE, in dir()), and they cannot drift, because this list is
the contract rather than a copy of it.
Feature types and fields are open. Somebody loads satellite_imagery
tomorrow and it has to work with no code change, so that half accepts any
attribute and is checked against the loaded data instead.
Declared does not mean closed. A store carrying an extra reference column resolves too: the declaration is what is known in advance, not what is allowed.
Lower case here, upper case in storage¶
data.reference.sector resolves to the SECTOR column. The API reads like
Python and the store reads like a data feed; column_for maps between them.
Derived fields resolve like stored ones¶
adv_3m, market_cap and free_float_market_cap are computed per request
rather than stored. A user should not have to know which side of that line a
datapoint falls on, so they live in the market namespace beside the stored
columns and carry a flag saying they are derived.
Namespace ¶
One dataset's fields, reached by attribute.
Declared names complete and are documented; anything else still resolves,
because a store may carry columns this list has never heard of. What is
not allowed is a private or dunder name, which would otherwise turn a
typo like data.market.__deepcopy__ into a field.
FeatureType ¶
Bases: Namespace
One feature dataset, such as data.features.fundamentals.
Wholly open: the fields a dataset carries are whatever was loaded, and a fixed list here would be wrong the first time somebody imported their own.
Features ¶
The feature namespace, which nests by dataset type.
data.features.fundamentals.revenue, not data.features.revenue.
TYPE is what separates datasets sharing one table: revenue from a
vendor and revenue from a user's own model are different series.
Flattening them here would leave the API unable to say which it meant at
exactly the moment the user is choosing between them.
Data ¶
The root: data.
Deliberately a small fixed set of namespaces. Unlike the fields inside
them, the datasets Beacon holds are a closed contract: a typo like
data.refrence.sector should fail at the attribute rather than build a
field in a namespace nothing will ever resolve.
column_for ¶
The stored column a field names.
Upper case for market, reference and action columns, which are stored that
way; feature fields keep their case, because a feature's FIELD is
whatever the loader wrote and upper-casing it would stop matching.
market_columns_for ¶
The stored market columns a set of fields reads.
Stored market fields name their column directly. Derived ones are expanded
through :data:DERIVED_REQUIRES into what they are computed from.
Reference, action and feature fields read other tables and contribute
nothing here, since a market-column check has nothing to say about them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fields
|
list[Field]
|
Typically |
required |
Returns:
| Name | Type | Description |
|---|---|---|
frozenset |
frozenset[str]
|
Upper-case market column names. |
resolve ¶
Turning an expression into an answer for one instrument on one date.
An expression is a description; resolving it needs an instrument and a date, and in an index that date is the rebalance.
Point in time¶
A screen on data.features.fundamentals.revenue at a rebalance on 1 April
must see what was published by 1 April. Q1 revenue announced in mid-May is
invisible on that date, however completely the quarter had ended.
Every feature read here goes through DataFetcher.fetch_feature, which is the
accessor that enforces this. Reading the table directly would put look-ahead
back in, and the resulting backtest would look better and be wrong: the
failure nobody catches, because a better number is not a symptom anybody
investigates.
Market and reference data are read as of the same date for the same reason. A market column takes its last value on or before the date (looking back at most 10 days), so a rebalance on a day the instrument did not trade still sees its latest value.
Missing is not zero¶
A name with no value for a field yields None, and on_missing decides what
the comparison answers. This is deliberately distinct from a value that is
legitimately zero: zero fails a > 0 test honestly, where missing has nothing
to compare at all. Collapsing the two would let a screen for "revenue above a
billion" quietly admit every company the dataset has never heard of.
resolve ¶
resolve(
expression: Expression,
identifier: str,
date: Timestamp,
fetcher: DataFetcher,
on_missing: bool = False,
max_age_days: int | None = MAX_AGE_DAYS,
) -> bool
Whether an instrument passes an expression on a date.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
expression
|
Expression
|
The tree to evaluate. |
required |
identifier
|
str
|
The instrument. |
required |
date
|
Timestamp
|
The date to stand on (in an index, a rebalance date). |
required |
fetcher
|
DataFetcher
|
The data. |
required |
on_missing
|
bool
|
What a comparison answers when the field has no value. |
False
|
max_age_days
|
int | None
|
How stale a feature may be and still count. |
MAX_AGE_DAYS
|
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
The answer for this instrument on this date. |
value_of ¶
value_of(
field: Field,
identifier: str,
date: Timestamp,
fetcher: DataFetcher,
max_age_days: int | None = MAX_AGE_DAYS,
) -> Any
One field's value for one instrument, as of a date.
Returns:
| Type | Description |
|---|---|
Any
|
The value, or None when the instrument has none knowable by |
stubs ¶
Generating a .pyi so a static analyser can complete the open namespaces.
__dir__ is enough for Jupyter and IPython, which ask a live object what it
has. Pylance and mypy do not run code (they read declarations), so the half of
the namespace that is open by design is invisible to them. This writes that
half down.
python -m beacon.expressions.stubs --out beacon-data.pyi
--store picks the store directory (default: the one the server auto-loads)
and --out the file to write (default: beacon-data.pyi). Feature fields
that are not valid Python names are left out of the stub with a warning; they
still resolve at runtime.
Opt-in, and regenerable¶
The answer depends on what is loaded, so there is no correct file to ship. A user who wants completions for their own feature datasets generates one; a user who does not, does not, and everything still works.
A stale stub is safe¶
A stub kept from an older store will autocomplete a field the data no longer
carries. That is fine, because it then fails validation (validation.py) with
a finding naming the field.
The generated file is a convenience; the loaded data is the authority. Treating the stub as the contract and validating against it would let a stale file silently authorise a screen the data cannot answer, which is the wrong failure: it would select nothing and say nothing.
generate ¶
The stub source for a loaded store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fetcher
|
DataFetcher
|
The data whose feature datasets should complete. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
|
validation ¶
Checking an expression against the data it will run on.
data.reference.sectr must not silently select nothing. A screen that quietly
matches no instruments produces an empty index and no explanation, the same
failure universe member validation exists to prevent: the result looks like a
legitimate answer, so nobody investigates.
Findings, not exceptions¶
Validation returns a list rather than raising on the first problem, in the
same shape as pipeline and universe validation. A user fixing a screen wants
every mistake at once, and a client needs to point at the offending rule
rather than show a message with no anchor. Each Finding carries a stable
code (UNKNOWN_FIELD, UNKNOWN_FEATURE_TYPE or UNKNOWN_NAMESPACE) so a
client can branch on the kind of problem.
An expression with no findings is valid. errors_in is the same list filtered
to what actually blocks, and is_valid is true when that list is empty.
"Did you mean"¶
sectr is one edit from sector, so the finding says so: an unknown name
comes with up to three close matches from what is loaded, or, when nothing is
close, the first few names that are available.
The loaded data is the authority¶
Not the declaration in namespaces, and not a generated stub. A store may
carry reference columns nobody declared, so an undeclared name is checked
against what is actually loaded before it is called wrong. And a stub kept
from an older store autocompletes a field the data no longer has, which is
safe because validation runs against the data and produces a finding rather
than a wrong selection. Two exceptions: derived market fields (adv_3m,
market_cap, free_float_market_cap) are always accepted because they are
computed per request, and action fields are checked against the declared list
because kind and status are computed rather than stored.
Finding
dataclass
¶
One problem with an expression.
Deliberately a plain dataclass rather than the server's pydantic Finding:
expressions are a library concern and must not depend on the server, and
as_dict produces exactly the shape the API already returns.
validate ¶
Every problem with an expression, checked against loaded data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
expression
|
Expression
|
The tree to check. |
required |
fetcher
|
DataFetcher
|
The data it will be resolved against. |
required |
Returns:
| Type | Description |
|---|---|
list[Finding]
|
list[Finding]: Empty when the expression is valid. |
errors_in ¶
Only the findings that block.
is_valid ¶
Whether an expression can run against this data.