Hyperparameter selection

Running an experiment requires the user to select simulation, preprocessing, and refinement settings. These choices are recorded in experiment.yaml. diffBloch keeps this file short by supplying defaults for common settings and automatically deriving quantities already defined by the input data. Importantly, values specified in the experiment.yaml will override defaults.

This page lists every config, its default, and what it controls. For guidance in selecting appropriate g_max, sg_max, and rocking_curve_sampling, see Convergence testing.

Values read from .cif and .cif_pets

Some values are deliberately not config fields in diffBloch, they are read from the structure .cif or .cif_pets file at load time. These include:

Value	Default	Where it is read	Notes
Electron energy / wavelength	None; required	.cif_pets _diffrn_radiation_wavelength	Specified during data reduction.
Unit cell (a, b, c, α, β, γ)	None; required	.cif_pets _cell_*, checked against the structure .cif	The .cif_pets cell is authoritative for all simulation geometry. See Inputs and outputs.
UB orientation matrix	None; required	.cif_pets _diffrn_orient_matrix_UB_*	The UB matrix and each rotation’s goniometer angles supply its starting orientation.
Data-collection geometry	None; required	.cif_pets _diffrn_measurement_details, line data collection geometry:	PETS values continuous rotation and precession, respectively.
Integration semiangle	None; required	Per-rotation .cif_pets _diffrn_zone_axis_precession_angle	Interpreted as the continuous-rotation tilt half-width or the fixed precession-cone angle. I
Apparent mosaicity (degrees)	None	.cif_pets _diffrn_measurement_details, line mosaicity:	Only used when blochwave.mosaicity: true; a missing value then raises rather than defaulting silently.

sample

Fixed sample properties (diffBloch.config.schema.SampleConfig).

Field	Default	What it does
thicknesses	(820.0,)	Seed thickness candidates in Å, shared by every rotation unless mean_thickness_by_dataset is set.
mean_thickness_by_dataset	{}	Optional starting mean thickness in Å for each pooled dataset, keyed by its exact inputs.exp_data path. When set, every dataset must have one entry.

inputs:
  multi_dataset: true
  exp_data: [dataset_1.cif_pets, dataset_2.cif_pets]

sample:
  mean_thickness_by_dataset:
    dataset_1.cif_pets: 400.0
    dataset_2.cif_pets: 800.0

blochwave

Bloch wave simulation hyperparameters (BlochwaveConfig).

Field	Default	What it does
solver	"matrix_exp"	Solver used for preprocessing, inference, and refinement. Use matrix_exp when absorption is enabled – the alternative, bloch_eigen, isn’t safe for the non-Hermitian absorptive structure matrix.
absorption	false	Include absorption as an imaginary structure-factor contribution.
rsg	0.66	Relative excitation-error cutoff. See rsg and dsg.
dsg	0.0015	Absolute excitation-error margin. See rsg and dsg.
rocking_curve_sampling	50	Tilt samples integrated per rocking curve. See Convergence testing.
mosaicity	false	true reads the apparent mosaicity in degrees from .cif_pets and converts it into an internal moving-average sample span. No additional orientations are simulated.
coupling_mode	"union"	See Union coupling.
union_adaptive	true	Choose union sections adaptively. See Union coupling.
fixed_n_segments	12	Number of union sections when adaptive splitting is disabled. See Union coupling.
union_max_new_beams_pct	0.01	Threshold for adaptive splitting. See Union coupling.
g_max	2.25	Largest reflection \(g\) vector simulated (Å⁻¹). Off-diagonal structure factors extend to \(2g_\mathrm{max}\).
sg_max	0.01	Maximum excitation-error magnitude (Å⁻¹) for a beam to enter the simulation at a sampled tilt.
ignore_orientations	()	Zero-based .cif_pets rotation indices to exclude from the experiment.

rsg and dsg

Continuous-rotation data is recorded in the .cif_pets file as overlapping virtual frames (Klar et al., 2023). Because the frames overlap, a given reflection can appear in several neighbouring frames, but should only be fully integrated in one of them. rsg and dsg identify the frames in which a reflection passes sufficiently far through the Bragg condition to be treated as fully integrated.

For each reflection, diffBloch calculates its excitation error \(|S_g|\) at the centre of the frame and the excitation-error half-range \(\Delta S_g\) swept from the centre to the edge of that frame. The reflection is retained when both conditions are satisfied:

\[\frac{|S_g|}{\Delta S_g} < rsg, \qquad \Delta S_g - |S_g| > dsg.\]

rsg is dimensionless. It limits the reflection’s central excitation error relative to its swept range. Increasing rsg retains reflections farther from the centre of that range. dsg is an absolute margin in Å⁻¹. Increasing dsg rejects reflections that only just enter the integration range. These parameters select the reflections compared with experiment; they do not select the beams included in the Bloch wave calculation.

Union coupling

Each virtual frame covers a small angular range of the crystal’s rotation. To simulate its integrated intensity, diffBloch samples a rocking curve across that range. rocking_curve_sampling sets the number of samples. Each sample is a tilt: one crystal orientation within the virtual frame. The intensities calculated at all tilts are summed to give the simulated integrated intensity for that frame.

Changing the tilt changes every reflection’s excitation error \(S_g\). At each tilt, beams are selected when \(|S_g| < sg_\mathrm{max}\). In per_tilt mode, diffBloch applies this test and builds a separate structure matrix at every tilt. This keeps each matrix as small as possible, but repeatedly calculates \(S_g\) for all candidate reflections and prevents the tilts from being solved together efficiently.

union mode groups neighbouring tilts. Every tilt in a group uses one combined beam set containing the beams selected at the group’s boundaries. The off-diagonal elements describe coupling between these beams through the structure factors and are reused across the group. The diagonal contains the excitation error of each beam and is recalculated for every tilt. The matrices are then solved as a batch, making better use of GPU parallelization. This is usually faster when the individual structure matrices are small or moderate in size.

The combined beam set is larger than the set needed by any one tilt. Union mode therefore solves larger matrices and stores several of them on the GPU at once. When the matrices are already large, or GPU memory is nearly full, this cost can outweigh the benefit of batching. In that case, coupling_mode: "per_tilt" is likely to be faster and use less memory. See Devices and scaling for measured comparisons.

With union_adaptive: true, diffBloch makes shorter groups where the beam set changes rapidly and longer groups where it remains stable: starting from one chunk spanning the whole rocking curve, each chunk’s midpoint tilt is checked against the beam union already covered by its two endpoints, and the chunk is split in two and recursed into if the midpoint would add more than union_max_new_beams_pct of new beams beyond that union. Adaptive chunking costs more beam-set unions up front (to evaluate the split predicate) but generally solves fewer total beams than a fixed split sized conservatively enough to avoid under-coupling.

With union_adaptive: false, fixed_n_segments sets the number of equal groups regardless of how much the beam set actually changes across them – more predictable when a fixed, known chunk count is wanted instead.

preprocess

Which preprocessing steps run and their search bounds (PreprocessConfig).

Field	Default	What it does
optimize_orientation	true	Run the per-rotation Nelder-Mead orientation search.
optimize_thickness	true	Run the per-rotation thickness grid search.
stage_order	"thickness_first"	Which optimization stage runs first when both are enabled. "thickness_first": optimize thickness against the seed orientation, then orientation against the optimized thickness. "orientation_first": reversed.

preprocess.orientation.nelder_mead

Bounds for the local orientation search (NelderMeadOptimizationConfig).

Field	Default	What it does
step_size	0.05	Initial simplex step size, degrees.
max_iterations	60	Maximum Nelder-Mead iterations (maxiter).
x_tolerance	0.001	Simplex convergence tolerance on the orientation parameters (xatol).
f_tolerance	0.001	Simplex convergence tolerance on the objective (fatol).
penalize_fewer_reflections	true	Penalize trial orientations that match fewer reflections than the seed, discouraging the search from drifting to a trivially-easier beam set.

preprocess.thickness

Bounds for the per-rotation thickness grid search (ThicknessOptimizationConfig).

Field	Default	What it does
min_thickness	5.0	Lower bound, Å.
max_thickness	2000.0	Upper bound, Å.
n_steps	100	Number of evenly-spaced grid candidates.
plot	false	Write one wR2-vs-thickness PNG per rotation (<inputs.structure's directory>/thickness_optim). Reporting-only — never affects the fitted Plan and is excluded from the reproducibility digest.

loss_metrics

The one residual driving both preprocessing search and gradient refinement (LossMetricsConfig). Top-level, not nested under refinement, because it governs preprocessing too.

Field	Default	What it does
residual	"wr2"	"wr2" or "robs". Both re-fit an optimal multiplicative scale between calculated and observed intensities before scoring (core.losses.optimal_scale) because calculated and .cif_pets intensities use different scales.

refinement

Field	Default	What it does
steps	40	Number of gradient-refinement epochs.
optimizer.name	"adam"	"adam", "adamw", or "lbfgs".
optimizer.lr	1e-3	Learning rate (Adam/AdamW; L-BFGS uses its own internal line search).

refinement.trainable

Field	Default	What it does
positions	"all"	"all" or "none": whether atomic positions are refined.
adp	"all"	"all" or "none": whether atomic displacement parameters are refined.
occupancy	"none"	"all" or "none": whether site occupancies are refined.

refinement.split

Train/validation split (DataSplitConfig).

Field	Default	What it does
train_test	false	false: train on every rotation, no held-out set. true: hold out an evenly-spaced val_frac of rotations from the refinement objective (preprocessing still fits their orientation/thickness) and report their wR2/R_obs separately.
val_frac	0.2	Fraction of rotations held out when train_test: true. Must be strictly between 0 and 1.

refinement.thickness_nn

Apparent-thickness neural network used by the default refinement path (ThicknessNNConfig).

Field	Default	What it does
enabled	true	Whether a learned thickness model replaces the fixed per-rotation seed during refinement.
num_samples	40	Number of thickness samples the network’s input encoding uses.
sample_thickness	false	Whether thickness itself is sampled as part of the network’s forward pass.
form	"min_thickness"	Functional form of the network’s output (currently only one implemented).
min_thickness	100.0	Lower bound, Å, for the network’s thickness output.
max_thickness	2000.0	Upper bound, Å, for the network’s thickness output.
init_seed	0	Random seed for the network’s initial weights.

With inputs.multi_dataset: true, one network is trained per dataset, each scoped to its own rotations and normalizing its own tilt range; these fields configure every network identically (including init_seed — per-dataset seeds would make results depend on exp_data ordering).

inputs

Input file references, relative to the experiment directory only (Inputs).

Field	Default	What it does
structure	required	Relative path to the structure .cif.
exp_data	required	Relative path to a .cif_pets, or (with multi_dataset: true) a list of 2+ paths.
multi_dataset	false	See Combining multiple datasets below.
load_hydrogens	false	Include hydrogen atom sites from the structure CIF (molecular crystals).
isotropic_displacements_only	false	Force every atom to refine with isotropic ADPs. CIF Uani sites are seeded from their crystallographic Ueq; see ADP parameterization.

Combining multiple datasets

inputs.multi_dataset: true pools rotations from several .cif_pets files (inputs.exp_data as a list of 2+ paths) into one experiment. Each dataset keeps its own orientation, integration geometry, and thickness seed. Two situations call for it:

• Beam damage series — repeat measurements of the same crystal taken at increasing dose. Each dataset is its own .cif_pets file, but they refine one shared structure.

• Low-symmetry structures — a single tilt series from one crystal orientation range may not cover enough of reciprocal space to constrain the structure well when the space group has few symmetry operations to fill in the gaps. Multiple datasets from different crystal orientations/mounts fill in coverage that one series alone would leave thin.

Each dataset is preprocessed and checkpointed on its own (plan.<stem>.npz per file, with its own integration geometry – precession angles may differ between files), and the settled per-dataset plans are pooled in memory just before refinement. The learned thickness model follows the same per-dataset shape: one refinement.thickness_nn network per file, reported per dataset. See Inputs and outputs for the mechanics (per-dataset checkpoints, the first-file authoritative cell, one-energy rule, rotation-index offsets) and Reproducibility for what each per-dataset lock verifies.

Set a different initial mean thickness for each dataset like this:

inputs:
  structure: structure.cif
  multi_dataset: true
  exp_data:
    - dataset_1.cif_pets
    - dataset_2.cif_pets

sample:
  mean_thickness_by_dataset:
    dataset_1.cif_pets: 400.0
    dataset_2.cif_pets: 800.0

Every rotation in each dataset starts from that dataset’s assigned value.