Perfect In-Distribution Accuracy Does Not Imply Learned Physics: Measuring Cross-Family Generalization of Metamaterial Homogenization Surrogates
Machine-learning surrogates for the effective elastic properties of mechanical metamaterials are almost always trained and evaluated within a single parametric family of unit-cell geometries, with the test set drawn from the same generator as the training set. Under this protocol reported accuracies are excellent and are frequently described as evidence of generalization. This work measures what that protocol does not: transfer to unit-cell topologies the surrogate has never seen. Using a homogenization engine written from scratch and validated against exact analytical results to 1e-15, six parametric design families are constructed - circular, square, elliptical and cross-shaped perforations, re-entrant honeycombs, and random strut lattices - and all are rejection-sampled into a common solid-fraction band [0.45, 0.75] so that cross-family comparison is not confounded by density. Effective stiffness is dominated by volume fraction, so without this control any failure would reflect density extrapolation, not topology. RESULTS (target C11, 48x48 grid, 750 cells per family, 4,500 total) Volume fraction only : R2 +0.559 in-distribution -> -1.350 across families; 2.4x MAE increase; 18 of 30 pairs below R2 = 0 Random Forest (image): R2 +0.933 -> -0.877; 9.2x MAE increase; 18 of 30 below 0 CNN (image) : R2 +0.959 -> -1.376; 10.2x MAE increase; 23 of 30 below 0 R2 below 0 means worse than predicting the dataset mean. The CNN reached R2 of approximately 1.00 on five of six families in distribution and still fell below 0 on 23 of 30 cross-family pairs. SCALING MAKES TRANSFER WORSE. Tripling the sample count and increasing resolution improved in-distribution R2 from 0.923 to 0.942 while degrading cross-family R2 from -0.597 to -1.000 and increasing MAE inflation from 6.1x to 9.2x. Additional within-family data buys a better fit to the training family, not generalization beyond it. THE FAILURE HAS A MECHANISM. It is far worse for property components encoding directional structure: predicting C22 gives cross-family R2 = -9.31 against -1.00 for C11, at comparable in-distribution accuracy. The mean ratio C11/C22 per family is exactly 1.000 for the circular, square and cross families - these are isotropic in-plane by symmetry - 1.113 for rotated ellipses, and 5.85 for re-entrant honeycombs. Four of six families therefore never present a case where the two directions differ, so a surrogate trained on them has seen no anisotropy at all and, tested on honeycombs, predicts C22 approximately equal to C11 and is wrong by roughly a factor of six. This suggests a diagnostic: auditing which symmetries a training set leaves unbroken predicts which predictions fail out of family. Results are stable across seeds: per-seed cross-family R2 for the Random Forest over five seeds was [-0.583, -0.542, -0.578, -0.636, -0.647], i.e. -0.597 +/- 0.039, against an in-distribution +0.923 +/- 0.008. ENGINE VALIDATION, reproducible from the included code: a homogeneous cell returns the input stiffness to 1.5e-16; a rank-1 laminate matches exact laminate theory to 5e-15; Voigt/Reuss bounds bracket the result; 90-degree-symmetric cells give C11 = C22 to 1.2e-15; mesh convergence 5.9% -> 0.59%; and a Gibson-Ashby re-entrant honeycomb gives nu21 = -0.865 against an analytical -1.0, the sign change emerging from geometry alone. PyTorch was unavailable in the environment used, so the convolutional network is implemented directly in NumPy and its backpropagation is verified against finite differences to a maximum relative gradient error of 2.3e-9. CONTENTS. Paper (PDF and LaTeX), homogenization engine and validation suite, six family generators, the matched sampler, the full dataset (4,500 cells at 48x48 plus a 1,500-cell pilot at 32x32), all surrogate experiments including the from-scratch CNN, and figure scripts. Code MIT; data, figures and paper CC-BY-4.0. Needs Python 3 with NumPy, SciPy, scikit-learn, Matplotlib - no FE library, GPU or network. LIMITATIONS. Two-dimensional and linear-elastic; 48x48 discretization; 750 cells per family. The CNN is small, trained with one seed per family, and was neither architecture-searched nor ensembled. The MLP baseline is data-starved and its extreme cross-family values partly reflect unbounded extrapolation; the Random Forest and CNN results are the cleaner ones. The random strut lattice family is intrinsically difficult (in-distribution CNN R2 = 0.77). Some cross-shaped and random-lattice cells are geometrically disconnected and correctly return near-zero stiffness; they are retained and may be filtered. No claim of priority is made: repeated literature searches did not find an equivalent systematic cross-family measurement for homogenization surrogates, but absence of evidence is not proof of absence. Also included: the 1,500-cell pilot dataset (250 per family, 32x32) used for the scale comparison, first released as MetaXFam: https://doi.org/10.6084/m9.figshare.32993597