← churper
machine learning·5 min·Wed, Nov 11

A connectome model only wins if you tune its physiology until it does: how to tell real wiring from a tuned baseline

When a fly-wiring model beats shuffled controls, the wiring may not deserve the credit. The physiology parameters nobody measured were often tuned on the real wiring, and the paper shows exactly when that tuning flips the verdict.

1

the line

What you could say out loud at dinner.

Some claims that "the fly's real wiring matters" fall apart when you stop letting the modellers tune the knobs until the real wiring wins, and the paper shows how to tell which claims survive.

2

the mechanism

How it actually works. Select a passage to dive deeper.

A connectome tells you who connects to whom and how many synapses each contact has. It does not tell you how excitable each cell type is, how strong a synapse is per contact, how fast neurons integrate, or how fast plasticity runs. Those are unmeasured physiology, and the modeller has to pick them. In practice, the modeller picks them once, and often picks them by adjusting until the real-wiring model performs the task. Then the shuffled wiring is given the same physiology, which was chosen for the real wiring. The comparison is now conditioned on success of the real wiring, so it is no longer a fair test of which wiring is better.

The paper separates three estimands. P1 is current practice: tune the physiology until the real wiring works, then transfer those parameters to the nulls. P2 tunes each wiring on its own, so each wiring gets its best case. P3 restricts each wiring's best case to physiology that independent data (like calcium imaging) actually allow. The paper's central empirical finding is that these can give different answers, and sometimes opposite answers. In the mushroom body of the fly, the real Kenyon-cell output wiring beats a shuffled wiring by 0.09 under P1 and P2, but loses to it by 0.09 under P3, where imaging says the share of one subtype of active cells should be high. In the head-direction ring attractor, by contrast, the real wiring wins under every protocol.

The paper traces the mushroom-body reversal to one unmeasured quantity: the share of active Kenyon cells that come from one subtype. At low shares, the real wiring wins. At high shares, the real wiring loses. Because tuning until the model learns selects the low-share regime, the real wiring looks better. But at the share implied by imaging, the real wiring model fails to learn at all. That is a model inadequacy, not a wiring deficit, and no real-versus-null comparison can reveal it on its own.

3

the math

Worked from the paper's own notation.

Let $\theta$ collect the unmeasured physiological parameters, drawn from a distribution $\pi$ over plausible ranges. For each $\theta$ and seed $s$, compute $S_R(\theta,s)$ and $S_N(\theta,s)$ for the real wiring $R$ and null $N$, with paired difference $\Delta_s(\theta)=S_R(\theta,s)-S_N(\theta,s)$, and $\bar\Delta(\theta)$ the seed-averaged difference.

The three protocols are:

P1 (shared, tuned until real works): $\mathbb{E}_{\pi}[\bar\Delta(\theta)\mid L_R(\theta)\geq c]$, where $L_R(\theta)\geq c$ is the behavioural criterion that the real wiring must meet to be considered working.

P2 (each wiring tuned on its own): $\mathrm{top}_K\,\bar S_R - \mathrm{top}_K\,\bar S_N$, where $\mathrm{top}_K$ is the mean of the $K=5$ best parameter sets.

P3 (P2 within data-consistent physiology): $\mathrm{top}_K\,\bar S_R|_{\Theta_{\mathrm{data}}} - \mathrm{top}_K\,\bar S_N|_{\Theta_{\mathrm{data}}}$, where $\Theta_{\mathrm{data}}\subset\Theta$ is the region allowed by independent physiological data.

The theory of P1's bias is a selection effect. Proposition 1 says: if $S_R$ and $S_N$ are exchangeable under $\theta\sim\pi$ (so wiring has no effect in distribution), then for any threshold $c$ with $\Pr(S_R\geq c)>0$,

$$\mathbb{E}[S_R-S_N\mid S_R\geq c]=\frac{\tfrac12\,\mathbb{E}\big[(S_R-S_N)(\mathbf{1}\{S_R\geq c\}-\mathbf{1}\{S_N\geq c\})\big]}{\Pr(S_R\geq c)}\;\geq\;0,$$

with equality only if $S_R$ and $S_N$ fall on opposite sides of $c$ with probability zero. Both factors in the numerator always share a sign because the indicator is monotone. Selection on the real wiring's success therefore guarantees a positive bias even when there is no real effect.

Proposition 2 gives a closed form. Let $X=S_R-S_N$ and $L_R$ be jointly Gaussian, with $\mathbb{E}X=\mu$ the true effect, $p=\Pr(L_R\geq c)$, $\alpha=\Phi^{-1}(1-p)$, and $\lambda(\alpha)=\phi(\alpha)/p$ the inverse Mills ratio. Then

$$\mathbb{E}[X\mid L_R\geq c]=\mu+\frac{\operatorname{Cov}(X,L_R)}{\sigma_L}\,\lambda(\alpha).$$

If selection is on the score itself ($L_R=S_R$) and $(S_R,S_N)$ are exchangeable with variance $\sigma^2$ and correlation $\rho$, the bias is $\sigma(1-\rho)\lambda(\alpha)$. The sign of the bias is the sign of $\operatorname{Cov}(X,L_R)$: selection on a criterion favours the real wiring only when the criterion tracks the real model's score more than the null's. The bias grows as the criterion becomes more selective (as $\lambda$ increases when $p$ falls), as scores spread more across $\theta$ and seeds ($\sigma$), and as the two wirings respond less alike ($1-\rho$). P2 has no such bias: under exchangeability the best-of-$K$ statistics of the two wirings are identically distributed, so $\mathbb{E}[\mathrm{P2}]=0$.

Proposition 3 covers the case where a null is retrained from a real-trained solution for a fixed budget. Near their optima, let each wiring's loss be quadratic, $\mathcal{L}_G(\varphi)=\mathcal{L}_G^*+\tfrac12(\varphi-\varphi_G^*)^\top H_G(\varphi-\varphi_G^*)$ with $H_G\succeq 0$. Run gradient descent with step $0<\eta\leq 1/\lambda_{\max}(H_G)$ for $t$ steps from $\varphi_0=\varphi_R^*$. Then the measured gap is

$$M_N(t)=\mathcal{L}_N(\varphi_t)-\mathcal{L}_R(\varphi_R^*)=(\mathcal{L}_N^*-\mathcal{L}_R^*)+E_N(t),\qquad E_N(t)=\tfrac12\sum_i\lambda_i(1-\eta\lambda_i)^{2t}\,(q_i^\top d_N)^2,$$

where $(\lambda_i,q_i)$ are eigenpairs of $H_N$ and $d_N=\varphi_R^*-\varphi_N^*$. $E_N(t)\geq 0$ is non-increasing in $t$. P1 is the case $t=0$. Longer budgets shrink the bias, but slowly along soft directions (small eigenvalues), which are common in circuit models. A corollary: if $N_1$ is equivalent to the real wiring and $N_2$ is genuinely worse by $g>0$, then $N_1$ is ranked below $N_2$ whenever $E_{N_1}(t)>g+E_{N_2}(t)$, which happens when $N_1$'s optimum lies far from $\varphi_R^*$ along soft directions.

The paper's specific mushroom-body result, in numbers: the KC $\to$ MBON claim against the identity shuffle is P1 $+0.091$ [0.068, 0.115], P2 $+0.093$ [0.045, 0.131], P3 $-0.092$ [-0.138, -0.052]. The deciding parameter is the $\alpha'\beta'$ share of active Kenyon cells, Spearman $\rho=-0.88$ with $\Delta(\theta)$, Sobol first-order index 0.88. The zero crossing near 0.177 is fixed by construction: at that share, the active cells are, by subtype, an unbiased sample of all KCs, so the subtype organisation of outputs stops mattering to first order. Imaging data place the share at about 0.44, where the real wiring model fails to learn (1 of 120 $\theta$ meet the learning criterion; retention near chance at 0.51).
4

what it overturns

What to stop believing, and what would make this wrong.

Before this paper, a well-informed reader seeing "real wiring beats shuffled controls" would take it as evidence that the measured wiring matters. That is the same logic the reader has accepted in many other places: ablations against shuffled baselines in deep learning, circuit perturbations in neuroscience, organizational-design studies. This paper says: the comparison is not about the wiring until you control what the unmeasured physiology is doing, and the physiology is often tuned on the real wiring until the real wiring wins.

What survives in this paper: the dopamine segregation claim (segregated reward and punishment dopamine improves learning) holds under every protocol tested. The head-direction topography claim (real topology is required for a persistent bump) holds under every protocol tested. The KC $\to$ MBON claim does not survive: it reverses under P3, and at the imaging-implied share of active cells, the real wiring model fails to learn entirely, which is a model inadequacy, not a wiring deficit.

The paper's own limits: the paper is not about discovering the bias, but measuring it in biological conclusions. The bias is well known in machine learning (under-tuned baselines, weight inheritance, winner's curse). The closed-form bias in Proposition 2 assumes a Gaussian approximation; the paper does not prove the sign of the bias when selection is on a separate criterion (it is an empirical question, answered by placebo nulls). The mirror analysis in flyvis is consistent with the quadratic model of Proposition 3, but the paper notes the flyvis Hessian is not measured, so slow convergence along soft directions is a plausible explanation, not a demonstrated one. The paper does not tell you what the fly's wiring actually does; it tells you which claims are well supported by the comparison protocols used to date, and which are fragile to the tuning protocol.