Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Group-Hash Indifferentiability

The Zcash security arguments model the Pasta group hashes as random oracles into the curve groups. This page explains what justifies that modelling, and what the formalization does and does not establish. It is written for a reader who has not seen an indifferentiability proof before; no prior acquaintance with the notion is assumed.

The counting that the argument rests on lives in CompElliptic (Hashing/TwoTermUniformity.lean, Hashing/PastaSSWU.lean); the probabilistic argument on this page lives under Zcash/Security/GroupHash/.

The deployed hash

Let be an elliptic curve group, its field of definition (base field), and a convenient input domain. The deployed group hash is

where sends a field element to a curve point, and the sum is the group law.

We model as a random oracle: an idealized hash whose output on each new input is a fresh uniform pair. The question is whether itself may then be modelled as a random oracle into the group . But first, we'll try to explain why a simpler construction does not suffice.

Mapping to a curve

How can we map from a field to an elliptic curve group? In the case of short Weierstrass curves, each non-identity point has coordinates satisfying the curve equation:

An obvious candidate for a map from a field element to a curve element would be to choose one of the points with using a deterministic square root function , i.e. .

Sapling used twisted Edwards curves which have a different equation, but that is essentially what it did — pick one of the coordinates, and then the equation in the other is quadratic. It's easy to construct a hash into the field with low bias using a conventional hash function with a large enough output size, by taking its output as an integer modulo the field size. Then the mapping above is bijective, so its output points will be approximately evenly distributed, although only among half of the curve points — the half chosen by the deterministic .

The problem is that then not all -coordinates, and therefore not all inputs to the hash, map to a point. For each -coordinate, we have 0, 1, or 2 solutions for depending on the number of square roots of . Heuristically, roughly half of the -coordinates should have no solutions for , roughly half of them should have two solutions, and a negligible proportion (only the case , which may not happen at all for a particular curve) have one solution. That is in fact what happens in practice. If the curve has points, and is odd —as it is for Pallas and Vesta— then the number of -coordinates that correspond to a point on the curve is exactly . The proportion that correspond to a point is , writing for the number of elements of . The Hasse bound, , makes the heuristic precise: is within of .

For fixed generators, having a group hash that is not a total function is not so much of a problem: we can extend it to a total function by repeated hashing with an index. Since there are only a fixed set of generators and they are found off-line, non-constant timing due to the variable number of iterations is not an issue. But Sapling had introduced diversified addresses, which require on-line use of the group hash in order to derive an address from a diversifier. We avoided timing attacks in Sapling by not doing repeated hashing, and accepting that only half of all diversifiers would be valid. But we had encountered complications in the application protocol (ZIP 32 and its usage) due to this abstraction leak from the underlying cryptography. We wanted to avoid that when designing Orchard.

Fortunately, an Informational RFC for deterministic, constant-time Hashing to Elliptic Curves was close enough to ready (it was in a late draft, and in fact did not change significantly before the final version, RFC 9380). The scheme we analyse here is that standard, specialized to Pallas and Vesta.

What 𝑓 looks like

So how does , or as RFC 9380 calls it map_to_curve, work? A naive approach would be to try to "fill in" the other half of the curve points that were missed by the deterministic square root, using the other half of the -coordinates. But there is no known way of doing so (in fact, if there were then it would indicate undesired structure and potential cryptographic weaknesses in the curve).

The basic idea of having two different cases depending on whether a given input yields solutions for a square root, however, is exactly what RFC 9380's "Simplified SWU" construction does. For now we will ignore a complication that arises for short Weierstrass curves with , like Pallas and Vesta; we'll get to that in its own section. Then, ignoring negligible cases we have:

The Simplified SWU construction arranges that the two candidate curve-equation values differ by a nonsquare factor, so exactly one of the two branches is available for each input . The precise formulas —including how is transformed before it becomes an -coordinate— are in CompElliptic's Hashing/SimplifiedSWU.lean, the formalization of the construction.

The actual construction also fixes the sign at the end: the output's -coordinate is negated if necessary so that its sign matches the sign of the input, in the convention that RFC 9380 calls sgn0. This makes odd, that is, for . Oddness carries weight below: it is what splits an input pair across a point and its negation, and the character-sum analysis relies on it too.

The images of and are not disjoint; for Simplified SWU they in fact coincide, apart from a negligible proportion of exceptional points. To see why, fix a target point . Whether any input reaches via comes down to a quadratic equation in ; the equation depends only on the -coordinate of , which shares with . A solution yields inputs precisely when is a square —that is, when really is for some input — and then, since is odd, the input pair has one member mapping to and the other to . So each realizable solution contributes exactly one preimage of . Reaching via comes down to a second quadratic in , in the same way. Now, two facts connect the branches:

  • solves the -equation exactly when solves the -equation;
  • is a square iff is, because their product is the square .

So input reaching via corresponds to the inputs reaching via and vice versa. Hence is reached via iff it is reached via .

This coexists with the exact halves above because those partition the inputs, not the outputs. The correspondence carries the -half of the inputs into the -half and back, preserving the point reached. About of the output space is reached —with 2 or 4 preimages per reached point excluding exceptional cases— and the remaining by neither map. (These proportions are heuristic; we confirmed them by exact computation on small curves, and they can be proven with error by counting points on the branch varieties — Lang–Weil, "Number of Points of Varieties in Finite Fields", Amer. J. Math. 76(4), 1954, doi:10.2307/2372655. A modern exposition of that paper is Tao, The Lang-Weil bound, 2012.)

Where the ⅜ comes from

Fix a target point and consider the quadratic in deciding whether it is reached — the branch- one, say. (The branch- one behaves identically under .) Two coin flips decide the outcome.

  • The quadratic has two roots when its discriminant is a square: probability about .
  • Given a split, each root yields an input pair exactly when is a square. These two events are perfectly correlated, because the product is fixed by the quadratic's coefficients: writing the quadratic character as on nonzero squares and on nonsquares, we have . That sign is about half the time, in which case exactly one root yields inputs. It is otherwise — then both roots yield inputs or neither does, each about half the time.

By oddness, each input pair contributes one preimage to the target point and one to its negation. So the point is reached from preimages (one per branch) with probability , and from preimages (two per branch) with probability ; otherwise it is unreached. The reach probability is , and reached points have preimages on average.

It turns out, for the Pasta curves, that we cannot do much better than this coverage by mapping directly from a single field element (or at least, trying to do so would not lead to a less complicated scheme overall, given other constraints like the desire for a constant-time group hash).

Particular application protocols might actually be perfectly fine with this kind of non-uniform mapping. However, it can easily be distinguished from a uniform one, and each of our security arguments would then need to take the non-uniformity into account separately. That need does not go away entirely; what we can do is pay for it once, with a concrete figure. So we would like a mapping that, applied to outputs, can be distinguished from a uniform mapping onto the whole group only with a concretely bounded advantage. As we will see, modelling as a random oracle, that advantage is at most after queries for the mapping we chose.

The detour through an isogenous curve

Now for the complication we deferred. The formulas of Simplified SWU require the curve coefficients to satisfy and . Pallas and Vesta both have the curve equation , i.e. . This is not by coincidence; the same Complex Multiplication structure that allows us to find a cycle of curves is what blocks Simplified SWU from working.

One symmetry, two effects

The short Weierstrass form with corresponds to curves with -invariant , that is, with Complex Multiplication by and an automorphism group of order : there are exactly six invertible mappings from the curve to itself that preserve the group structure, namely . (These stay on the curve because appears only cubed, and .)

Daira-Emma Hopwood's ZK Study Club talk "Optimizing Halo and Constructing Graphs of Elliptic Curves" (part 1, bonus session, slides) explains why the Pasta curves have this form: the two curves of a 2-cycle necessarily share their CM discriminant, and with known methods a cycle can only feasibly be found when that discriminant is tiny (the Pasta search fixed the smallest, , which is exactly the case ). Slides 8 and 9 give a nice visual form of the argument.

Simplified SWU, for its part, obtains its branch pair by solving for the -coordinate at which the scaling defect

vanishes, where . The -coefficient is proportional to , so on a curve there is nothing to solve for: every -scaling is an isomorphism onto a sextic twist, making the defect constant in , and it vanishes only when the scaling is one of the extra automorphisms — which , a nonsquare, never is.

RFC 9380 (section 6.6.3) resolves this with a detour: run Simplified SWU on an auxiliary curve with that is isogenous to the target. An isogeny is a mapping from one curve to another, given by rational maps on the coordinates, that preserves the identity point. In general it need not be invertible; over the algebraic closure, a degree- isogeny is -to-. For Pallas and Vesta, the auxiliary curves are the ones that the protocol specification and the pasta_curves crate call iso-Pallas and iso-Vesta respectively. Having used Simplified SWU to obtain a point on the auxiliary curve, we apply the isogeny (here of degree 3; the Pasta curves were chosen to make the degree as low as possible), in order to land on the intended curve.

For the analysis on this page the detour is short, at least conceptually. An isogeny is always a group homomorphism, and for these particular curve pairs it is a bijection on the rational points. (Isogenous curves have equally many rational points, and the kernels of these particular isogenies contain no rational point other than the identity.) A bijective relabelling of the outputs neither merges nor splits fibres, so the branch structure, the preimage counts, and the oddness that the character-sum analysis below relies on, all transport across unchanged. The formalization defines (mapToCurve) as the composition and states the counting theorems directly on that mapping.

Because the isogeny is a homomorphism, there are two equivalent ways to compute : either by adding the two Simplified SWU outputs on the auxiliary curve and applying the isogeny once, or by mapping each point across the isogeny and then adding. The former method is used by RFC 9380 and hashtocurve.sage; the latter by pasta_curves. The two orders agree exactly (mapHashOutputsToCurve_eq), so nothing depends on the choice.

Although a correctly constructed isogeny is always a homomorphism (Silverman, The Arithmetic of Elliptic Curves, Theorem III.4.8), Mathlib does not prove that or have the necessary machinery to do so in general. Instead we prove that the particular rational maps given in the protocol specification (§5.4.9.8) and hashtocurve.sage are bijective (iso_map_bijective) and are homomorphisms (iso_map_add). The latter turns out to be quite involved, requiring a careful choice of coordinates to make it feasible to prove the necessary identities using Mathlib's linear_combination tactic. The details are explained in Homomorphism.lean.

We've now described the deployed construction in full, and established the motivation for using instead of a mapping from a single field element. The rest of this page is about why that construction works, specifically why it can reasonably be modelled as a random oracle.

Uniformity is not enough

A first guess is that it would suffice for 's outputs to be close to uniform on . We will see from the regularity analysis below that this holds. It does not suffice, because is not a black box. The function is public: anyone can compute the intermediate pair and check that really equals . A security argument that replaces by an ideal random oracle must survive an adversary that does exactly that. So the question is not "do 's outputs look uniform?" but "can the pair of oracles be faked consistently, given only ?".

Indifferentiability

Indifferentiability (Maurer–Renner–Holenstein, Indifferentiability, Impossibility Results on Reductions, and Applications to the Random Oracle Methodology) makes that question precise. A simulator is given oracle access to the ideal random oracle , and must answer queries. A distinguisher talks to two oracles and tries to tell which of two worlds it is in:

  • the real world — the genuine intermediate oracle and the genuine construction built on top of it;
  • the ideal world — the ideal random oracle into the group, and the simulator faking the intermediate hash consistently with it.

The construction is -indifferentiable if some simulator makes every distinguisher's advantage at most after queries. The point of establishing this is the Maurer–Renner–Holenstein composition theorem: any protocol proven secure with an ideal in place of the group hash stays secure with the real — provided one is content to model as a random oracle. So indifferentiability is what lets the rest of the security development treat the group hash as a random oracle without having to reason about again.

A heuristic, not an assumption

Modelling as a random oracle is a heuristic, not a falsifiable hardness assumption. Non-instantiability results (Canetti–Goldreich–Halevi, The Random Oracle Methodology, Revisited) show that a scheme can be provably secure in the random-oracle model yet insecure under every concrete instantiation. So an indifferentiability proof does not guarantee real-world security on its own; it restricts attention to adversaries that treat as a black box, which is where analytical effort is most useful to spend. The Security Models page develops this framing.

† The "any" has a shape requirement: the protocol's security game —challenger, adversary, and win condition together— must fold into a single distinguisher talking to the two oracles, as the games in this development do. Composition can genuinely fail for definitions that restrict the state shared between the stages of an adversary (Ristenpart–Shacham–Shrimpton, Careful with Composition: Limitations of Indifferentiability and Universal Composability). The boundary is made precise, as a restriction on the memory available to the simulator, in Demay–Gaži–Hirt–Maurer, Resource-Restricted Indifferentiability.

The simulator is forced

The consistency check above pins down what the simulator must do. On a query it learns , a uniform group element, and it must return a pair with

because the distinguisher can and will check that equation. Moreover the pair must look like a fresh output, i.e. uniform — so the simulator must return a preimage of that is close enough to uniform under the two-term sum. Following the proof of Theorem 1 of Brier–Coron–Icart–Madore–Randriam–Tibouchi (Efficient Indifferentiable Hashing into Ordinary Elliptic Curves), specialized to this construction, two ingredients make this possible.

The first ingredient: regularity

For uniform , the distribution of is close to uniform on . CompElliptic's TwoTermUniformity proves this from a Weil bound on the character sums of .

A character of is a homomorphism into the nonzero complex numbers: it turns the group operation into ordinary multiplication, , and its values lie on the unit circle. The character sum of at is

the character added up over all outputs of . The trivial character gives ; a Weil bound bounds the absolute value at the nontrivial characters, from which such character-sum bounds follow. The name "Weil bound" is from André Weil's proof of the Riemann hypothesis for algebraic curves over finite fields (Sur les courbes algébriques et les variétés qui s'en déduisent, 1948). A modern presentation of the elliptic-curve case is Kohel–Shparlinski, On Exponential Sums and Group Generators for Elliptic Curves over Finite Fields, ANTS-IV, LNCS 1838, 2000.

Character sums measure uniformity because a distribution on is uniform exactly when all its nontrivial character sums vanish — so small nontrivial character sums mean close to uniform. That is what lets a Weil bound control the regularity distance

where counts the pairs with — the size of the fibre of . Dividing by turns the count into the probability that the two-term sum lands on , so the sum is the distance between that output distribution and the uniform distribution on . The distance between two distributions and on a finite set is , the total of the absolute differences of the probabilities they assign. will be calculated in the next section.

Characters, for readers who know the DFT

The DFT analyses a signal on against the reference waves , one per frequency . What makes those waves work is not anything analytic about the exponential — it is the identity , which turns addition of signal positions into multiplication of wave values. A character keeps exactly that property and discards the rest. For the characters are precisely the reference waves of the DFT; for a general finite abelian group there are exactly as many characters as group elements, and they support the same Fourier toolkit — in particular orthogonality (a nontrivial wave sums to zero over a full period) and Parseval (total energy is the same in the signal and frequency domains). Curve points under point addition are a finite abelian group, so all of this applies to them directly; no geometry enters.

The regularity proof is then the standard DFT pipeline for a convolution: the distribution of for independent uniform is the convolution of two copies of the distribution of , and convolution in the signal domain is multiplication in the frequency domain, so the transform of at frequency is the square — just as convolving a signal with itself squares its spectrum. The Weil bound says every nontrivial frequency is small; squaring, Parseval, and Cauchy–Schwarz then yield the regularity distance.

Calculating the Weil constant

The regularity distance is proved relative to the named hypothesis WeilBounded. That hypothesis is parameterized: it asserts a constant with every nontrivial character sum of the zero-repaired mapping at most , and the final advantage scales with .

The formalization (sum_abs_prob_dev_le) bounds the regularity distance of the previous section by any budget whose square dominates that is, any just above .

That expression is the aside's pipeline, made quantitative. The two-term spectrum at is , so each of the nontrivial frequencies has spectral energy at most . Parseval turns total spectral energy into the summed squared deviation of the pair counts, divided by ; Cauchy–Schwarz bounds the square of an sum by times the sum of squares, cancelling the quotient; and normalizing counts to probabilities divides by — leaving .

At the deployed sizes and (see below), yielding .

The Weil bound places a bound on character sums along covering curves of the encoding, once is calculated via a per-encoding genus computation. Proving this result in general requires machinery that is not yet in Mathlib, which is why the hypothesis is named rather than discharged; that is where the deep number theory lives.

The calculation of the constant for a specific encoding and curves, on the other hand, is relatively straightforward. For example, Farashahi–Fouque–Shparlinski–Tibouchi–Voloch carry out this calculation for a sibling of the deployed encoding —simplified SWU with , over fields of size , with a quadratic-residue sign rule— and obtain from genus-8 coverings.

The deployed variant differs in all three parameters. The Weil bound for both Pallas and Vesta has been calculated as from genus-6 coverings (see zcash/pasta's weilbound.sage). This is where the deployed comes from: the hypothesis wants , and the extra half over the absorbs the trailing . In square-root-free form this is , which holds at the deployed sizes with margin about .

The calculation of is proven on paper in CompElliptic's design/weil-constant-derivation.md, modulo results cited as established mathematics. It is also formalized, down to Weil's theorem at the two branch covers, in CompElliptic's Hashing/BranchCovers.lean and Hashing/WeilInstance.lean. The per-cover inputs are — the analogous sums over the rational points of the two branch coverings, stated in square-root-free form. Everything between those inputs and the deployed WeilBounded instances is machine-checked. The paper proof's own checkable inputs are also machine-checked (CompElliptic's Hashing/WeilSupport.lean), and the design doc cites each proven fact at its point of use, with CI keeping the references exact. Weil's theorem itself stays the cited input: even stating it needs vocabulary (genus, places, covers of curves) that Mathlib does not yet have. That vocabulary is tracked at CompElliptic#30.

The second ingredient: preimage sampling

For each , the simulator must sample a pair uniformly from the fibre . Sampling one coordinate is easy: draw uniformly. Then the second coordinate must satisfy , so ranges over the preimages of under the single map . That single-term fibre has at most a constant number of elements — we saw in the "Where the ⅜ comes from" note above that each point has at most nonzero preimages under , and CompElliptic's card_mapToCurve_fibre_le proves the weaker but sufficient bound of , again counting nonzero preimages.

Care is needed to make the pair uniform on the fibre. Drawing uniformly from the preimages of would over-weight the pairs whose preimage set is small: the pair's probability would be with the size of its preimage set, and varies across the fibre. So the simulator instead fixes a bound on the preimage counts and draws a slot index uniformly, alongside . If the preimage set of has an element with index , the round accepts the pair with that element; otherwise it rejects, and the simulator redraws both and . In particular an empty preimage set always rejects. Now every pair of the fibre consistent with is accepted in a round with the same probability , whatever the size of its preimage set, so conditional on acceptance the pair is exactly uniform on the fibre. The bound also controls the cost: a round accepts with probability , about for typical , so few rounds are needed. This is the rejection sampler whose costs and output law Simulator.lean proves, instantiated at the deployed mappings at the constant deployedFibreBound = 11 — the bound of for nonzero preimages, plus one for the input .

The single-query bias, in detail

This is the part the formalization currently establishes, in Zcash/Security/GroupHash/Sampler.lean, and it is the technical heart of the argument. It compares the two worlds on a single fresh query, before worrying about how queries compose.

Two per-query laws

On a fresh query, the distinguisher observes a pair in (from which the group element is a fixed function). Each world draws that pair from a distribution:

  • real: the pair is uniform on — this is answering honestly (PMF.uniformOfFintype);
  • ideal: draw a uniform group element , then draw a pair uniformly from the fibre of (idealLaw, the bind of the uniform law on with the fibre sampler).

The fibre sampler and its fallback

fibreSampler f Q samples a pair uniformly from the fibre of . One subtlety: the two-term sum need not be surjective, so some have an empty fibre, with no pair to return. On those, the sampler falls back to a uniform pair on , which keeps it a genuine distribution. The fallback's only effect is on the bias, where it is accounted for exactly.

The bias reduces to the regularity distance

The claim, in each direction, is that the law in each world overshoots that of the other world by at most : for every test valued in , . This one-sided form (PMFWeightedBiasLE) is what the query-composition step needs.

To bound it, regroup the per-pair difference by the group element . Take a nonempty fibre of , with pairs. Every pair in it looks identical in both worlds:

  • the ideal world puts on each pair — it spreads the that gives to uniformly over the pairs;
  • the real world puts on each pair.

So the absolute difference is one constant across all pairs of the fibre, and summed over the fibre it is

a single term of the regularity distance. The cancels inside the first fraction. The fibre size enters only as in that term, which is identical for every nonempty fibre — the ideal-world law is uniform within the fibre whatever its size, so all pairs share one probability. Summing over the nonempty fibres gives the part of the regularity distance with .

Why both directions come out at the same

The empty fibres require our attention in one direction only.

When the real law overshoots the ideal one, the fallback only raises the ideal law's probabilities, which shrinks . So this direction is bounded by the nonempty part of the regularity distance alone.

When the ideal law overshoots the real one, the fallback contributes a fallback mass , spread over all pairs, where is the number of group elements the two-term sum misses — the mass sends to those missed elements. That mass is exactly the empty-fibre part of the same regularity distance: an empty fibre has , so its term is , and there are of them, totalling . So the nonempty part and the fallback mass together are the whole regularity distance . The fallback fills in the terms the nonempty part left out, and the bound stays at .

From one query to many

A single-query bound does not immediately bound a distinguisher that makes many adaptive queries — later queries may depend on earlier answers. The adaptive hybrid runFreshPMF_eventBiasLE (in Zcash/Common/Oracle/) bridges the gap: it charges the one-squeeze bias once per query node, so a -query tree turns a single-query bias into an overall bias of at most , even when the query tree is fully adaptive. Repeated queries to the same point are first collapsed by dedup, so a point asked twice keeps one answer rather than drawing a fresh one.

What is proved, and what is modelled

It's important to be precise about the status of each part.

  • Formalized and machine-checked. The regularity distance (TwoTermUniformity, conditional on the Weil bound), the single-term fibre bound (card_mapToCurve_fibre_le), the single-query bias in both directions (Sampler.lean), its composition into the full distinguisher-advantage bound at the deployed mappings (Indiff.lean), the collapse of the two-oracle game onto that one-oracle form (TwoOracle.lean), and the rejection-sampling simulator — its round-count laws, its output law's distance to the fibre sampler, and the composition with the simulator as the exhibited ideal-world witness (Simulator.lean and the capped section of Indiff.lean).
  • An unformalized mathematical input. The regularity distance rests on Weil's theorem at the two branch covers — the CharSumBounded inputs discussed in Calculating the Weil constant. The bound calculation between those inputs and the endpoints is machine-checked; the inputs themselves are cited — stating them needs function-field vocabulary that Mathlib does not yet have (CompElliptic#30).
  • A modelling choice, not a theorem. That behaves like a random oracle is a heuristic (see the note above). The indifferentiability argument is what makes that heuristic transfer from to the group hash ; it does not remove it.

Conclusion

The question this page set out to answer is: "can we formally justify modelling the deployed group hash as a random oracle into the curve group, given that is so modelled?" The formalization now carries the whole argument, machine-checked at the deployed Pallas and Vesta instances.

A distinguisher that makes queries, and sees both the field-element hash and the group hash built from it, can tell the real construction from a random oracle with advantage at most (pallas_indiffFromRO, vesta_indiffFromRO, via the two-oracle collapse twoOracleIndiffFromRO). The only unformalized mathematical input is Weil's theorem at the two branch covers, discussed above. The budget absorbs the regularity distance, about (the arithmetic is at the end of the regularity section), and the zero-repair transport , roughly .

The ideal world in that statement is played by a simulator, and the simulator is a real algorithm, not just a distribution: it hashes once, then rejection-samples a preimage pair, giving up after rounds. Its cost is pinned down exactly — the chance that it is still running after rounds decays geometrically. The answers it returns differ from the idealized ones by at most per query, where is a round's chance of accepting, so the cap makes that difference as small as desired. The indifferentiability statement holds with this algorithmic simulator in place of the idealized one, at the cost of that same per-query term (pallas_indiffFromROCapped, vesta_indiffFromROCapped), conditional on the Weil bound hypothesis.

Two things remain, both tracked in issues:

  • The Weil bound rests on a cited input: Weil's theorem at the two branch covers. The calculation from that input to the deployed constant is formalized, and so are the paper proof's supporting facts (CompElliptic's Hashing/WeilSupport.lean) — the delivered scope of CompElliptic#28. The input's own statement needs function-field vocabulary (genus, places, covers) that Mathlib does not yet have, tracked at CompElliptic#30.
  • The security games that want to use this result need the group hash added to their adversary's interface first (#188). The composition requirement for multi-stage games (the † note above) applies at each consumption site.