Decision log¶
Every non-obvious choice in this project, the evidence behind it, and what was rejected. Measurements were taken on the two machines this project actually uses; where a number appears, it was measured, not estimated.
Hosts referenced below:
- M3: MacBook Pro, Apple M3, 16 GB unified memory, macOS 27, PyTorch 2.12 (MPS).
- Ada: peacock06, NVIDIA RTX 6000 Ada (sm_89, 142 SMs, 48 GB), PyTorch 2.13+cu126, Slurm. The training card -- see D19.
- A100: peacock04 (A100-PCIE-40GB) and peacock06 (A100 80GB PCIe), sm_80, 108 SMs, same software stack. Used as the control in D19.
D1. Rebuild rather than extend the previous repository¶
Evidence. The prior revision contained no trained model. Its only learning
code (lightweight.py) trained a closed-set classifier over 60 hardcoded SMILES
and reached 5.6% exact match on its own synthetic data; six of the eleven
registered "models" were ExternalPlaceholderAdapter, which can only emit
skipped. _TinyViT stored its layers on a plain attribute rather than as
submodules, so parameters() returned an unregistered set and .to(device)
moved a shadow copy.
Decision. Keep the ideas that were sound -- canonical-SMILES scoring, the "unavailable systems are skipped, not failed" contract, the frozen manifest format -- and rewrite the rest around an actual trainable model.
Rejected. Incremental repair. A closed-set classifier cannot become an open-vocabulary recognizer by tuning; the output space is wrong.
D2. Sequence decoding, not graph decoding¶
Evidence. Training images come from RDKit rendering, which yields an exact SMILES per image and no atom coordinates. Graph decoders (MolGrapher, MolScribe's graph head) need atom-level coordinate supervision.
Decision. Autoregressive SMILES decoder.
Consequence, stated up front. Graph decoding is the stronger formulation at the top of the field. This is a supervision-availability decision, not a claim that sequences win.
D3. Hybrid conv + attention encoder at 384 px¶
Evidence. Attention cost scales with token count; convolution cost scales with pixels. Early OCSR features (strokes, junctions, glyphs) are local. At 384 px with a /16 stem the encoder attends over 576 tokens.
Decision. Conv stem to a 24x24 grid, then 4 transformer blocks with axial 2D RoPE.
On resolution. 224 px was rejected because atom labels, charges and subscripts stop being legible inside dense fused-ring systems. Resolution must follow the declared molecular scope: the scope here admits up to 48 heavy atoms.
D4. Atom-level SMILES tokenization¶
Evidence. Measured on this corpus: 88 vocabulary entries cover 2.6M ChEMBL molecules with 1 of 5000 validation molecules containing an unknown token. Atom-level sequences are ~40% shorter than character-level.
Decision. Regex atom-level tokenizer; Br, [C@@H], %10 are single tokens.
Rejected. Character-level (longer sequences, and the model must learn bracket well-formedness from scratch). SELFIES (guarantees validity, but its non-locality makes the image-to-token alignment harder to learn, and validity is already ~100% here after a short training run).
D5. On-the-fly rendering instead of a frozen training set¶
Evidence. Measured RDKit MolDraw2DCairo throughput: 542 img/s per core at
224 px, 139 img/s per core at 320 px with the full augmentation pipeline. One
M3 training step at the tiny preset processes 63 img/s. Rendering is therefore
never the bottleneck.
Decision. Generate every training image fresh. steps x batch_size samples,
essentially none repeated.
Consequence. The augmentation distribution is the specification. Anything not in it is out of distribution at test time regardless of compute spent.
D6. Per-worker RNG seeding¶
Evidence. macOS spawns DataLoader workers rather than forking, so every worker re-imports the module and starts from an identically seeded RNG.
Decision. Derive each worker's seed from (seed, worker_id, epoch).
tests/test_data.py::test_workers_with_different_ids_draw_different_samples
is the regression guard.
Why it matters. Without this the effective data diversity silently drops by
a factor of num_workers and presents as a training plateau, not as an error.
D7. fp32 on MPS, bf16 on CUDA¶
Evidence. Measured matmul throughput on the M3: fp32 2083 GFLOPS, bf16 2436
GFLOPS, fp16 2530 GFLOPS -- a 17% ceiling for mixed precision. On CUDA the gain
is not an estimate either: the D19 probe measured the whole training step at
base, batch 64, and bf16 was 1.9x on the Ada (228 -> 441 img/s) and
2.2x on the A100 (199 -> 439 img/s).
Decision. --amp none locally, --amp bf16 on CUDA.
Rationale. 17% does not justify autocast numerics debugging on the platform where iteration speed matters most.
D8. WSD learning-rate schedule¶
Evidence. Cluster runs share resources and may be cut short or extended. Cosine bakes the total step count into every step's LR.
Decision. Warmup-stable-decay: flat peak, decay only in the final 20%.
Consequence. Any mid-run checkpoint is usable, and extending the budget moves the decay rather than invalidating what came before.
D9. ChEMBL 37 as the molecule corpus¶
Evidence. Procedural fragment assembly gives unlimited molecules but a
distribution that does not match real chemistry, which is what the model is
asked to generalize to. ChEMBL 37 chemreps is 279 MB and yields 2.9M real
drug-like molecules.
Decision. ChEMBL 37, curated to the declared scope: 2,628,559 molecules kept of 2,897,819 read (269,119 rejected as unparseable or out of scope, 141 duplicates by canonical form).
Note. This required a network download; the user was asked and approved it before it was performed.
D10. Leakage guard against the hand-drawn evaluation set¶
Evidence. DECIMER's hand-drawn dataset was drawn from published structures, so overlap with ChEMBL was expected. Measured: 210 molecules appear in both.
Decision. Subtract every DECIMER HDM molecule from the training corpus by
canonical SMILES before splitting, and record the count in corpus_meta.json.
Train/val/test disjointness is asserted, not assumed.
Why it matters. Without this, the headline "generalizes to real hand-drawn structures" number would have been partly a memorization measurement.
D11. Curation parses each molecule once, across processes¶
Evidence. The first implementation canonicalized and then scope-checked, parsing each SMILES twice: 200k molecules in ~4 minutes, projecting to ~60 minutes for the full corpus.
Decision. Parse once, pass the parsed molecule to the scope filter, and fan parsing out across processes while the parent owns the dedup set. The full 2.9M corpus then curated in well under two minutes.
Guard. test_curate_is_identical_with_and_without_workers asserts the
parallel and serial paths return the same list.
D12. Train on the cluster, run inference on the laptop¶
Evidence. Measured M3 throughput at the tiny preset, 256 px: 63 img/s. The
base preset at 384 px is roughly an order of magnitude more compute per image.
Reaching a useful sample budget locally would take days.
Decision. Train on one cluster GPU; ship the checkpoint back and benchmark inference on the M3, which is where the model has to run. Which GPU is settled by measurement in D19 -- it is the RTX 6000 Ada, not the A100.
Cluster occupancy at submission time (scontrol show node -d): peacock04
1/4 A100 in use, peacock06 1/4 A100 and 1/3 RTX 6000 Ada in use, peacock05 7/8
A100 in use, V100 nodes full, Slurm queue empty. A single-GPU job was
non-disruptive.
D13. MolScribe baseline moved off the laptop¶
Evidence. The local .venv310 MolScribe environment no longer imports:
macOS 27's dyld rejects the Python 3.10 scipy binaries
(__DATA/__thread_bss zero-fill section with a nonzero offset), and scipy >= 1.16
requires Python 3.11+ while MolScribe pins torch<2. Upgrading scipy inside the
3.10 environment did not resolve it.
Consequence. The MolScribe numbers recorded in the previous report came from an environment that no longer runs on this machine, so they cannot be reproduced locally and are not carried forward.
Decision. Run MolScribe on Linux for the comparison, behind a subprocess
boundary (scripts/molscribe_predict.py) so its old dependency stack never
enters the training environment. Report the local unavailability as a finding in
its own right, since "runs on this laptop" is part of the goal.
D14. Font families differ by host, so font generalization is measured, not hidden¶
Evidence. available_fonts() returns 12 fonts on each host, but the two
sets are disjoint: peacock offers DejaVu variants (/usr/share/fonts/dejavu-*),
macOS offers Arial and Georgia variants (/System/Library/Fonts/Supplemental).
Training runs on peacock. The first frozen evaluation set was rendered on macOS.
The hazard. Left alone, every varied and degraded score would be
depressed by an uncontrolled train/eval font mismatch, and the drop would be
misattributed to augmentation difficulty.
Decision. Build the evaluation set twice, on both hosts, from the same molecules and seed, and score both:
| eval set | rendered on | fonts relative to training |
|---|---|---|
chembl_test_linux |
peacock | same family pool as training |
chembl_test_macfonts |
M3 | disjoint family pool |
The first isolates recognition accuracy; the difference between them is the model's font generalization, reported as its own number instead of contaminating the headline.
Rejected. Installing matching fonts on one host to make the mismatch disappear. It would have hidden a real generalization axis, and metric-compatible clones (Liberation for Arial) are not the same glyphs anyway.
D15. Latency is not comparable across the two measurement paths¶
Evidence. MolMiniPredictor loads its checkpoint in __init__, so the
recorded latency_ms is inference only. SubprocessRecognizer divides total
wall time by the batch, which folds MolScribe's 1.1 GB checkpoint load into the
per-image figure.
Decision. Report MolMini and MolScribe latency from separate, explicitly labelled measurements, and never in an unannotated shared column. MolMini is additionally measured on the M3, because that is the machine the model has to run on; MolScribe cannot run there at all (D13).
D16. In-training eval is the hardest tier; the report's headline is not¶
Evidence. Trainer.eval_render_config and eval_augment_config are both
fully randomized, so the [eval] exact values in train_log.jsonl are
degraded-tier numbers, and best.pt is selected on degraded performance.
Decision. Keep it -- selecting on the hardest tier is the conservative choice -- and state the difference in the report, so the training log and the final table are not read as contradicting each other.
Consequence. The report's clean-tier exact match will be substantially
higher than any number printed during training.
D17. The memorization floor needs a gallery it did not see¶
Evidence. Scored with the evaluation manifest as its own gallery,
image_hash_nn reaches 0.997 exact match -- it is retrieving the query image
itself, which measures nothing.
Decision. Build the gallery from 3,000 training molecules rendered
clean. It then scores 0.000 on the test molecules, which is the informative
result: nearest-neighbour retrieval cannot recognize a molecule it has not
already seen, so any score the model earns above this line is recognition
rather than lookup.
D18. The bug that cost a training run: encoder gradient explosion at step 0¶
This is the most important entry in this log. Both first training runs were cancelled at ~10k steps because of it.
Symptom. Loss fell normally (1.03) and teacher-forced token accuracy reached
0.77, but greedy decoding returned the identical SMILES string for every
image, degenerating into a long CCCCC... run. Held-out exact match was
0.000 and validity had dropped to 0.000.
Diagnosis. Measured on a mid-training checkpoint: encoder memory had cosine similarity 1.00004 between two different images, and per-token spread 0.0004. The encoder was emitting one constant vector regardless of input; the decoder had learned an unconditional SMILES prior. SMILES is predictable enough that an unconditional language model alone reaches ~0.6-0.77 token accuracy, so nothing in the loss curve exposed this.
Ruling out an architecture bug. Overfitting 8 fixed (image, SMILES) pairs memorized 8/8 with the cross-attention pathway carrying the signal. The architecture was fine; the failure was in optimization.
Root cause. Per-module gradient norms at step 0, on real depictions:
| gradient norm at step 0 | |
|---|---|
| encoder | 108,675 |
| decoder | 12.2 |
init_weights applied a fixed trunc_normal_(std=0.02) to every layer. That
constant is calibrated for GPT-2's width; applied to a convolution with fan-in
576 it attenuates activations ~2x per layer, and measured stem output std was
0.017. Every normalization layer's backward gain is proportional to
1/RMS(input), so tiny activations became enormous gradients. Global
clip_grad_norm_(1.0) then divided the entire gradient by ~10^5: the decoder
received effectively no update, and the encoder received one huge, badly scaled
update that destroyed it on step one. From step 30 onward the encoder gradient
was ~0.03 -- dead -- and the differential component of the memory decayed
monotonically while the shared component stayed at 13.7.
Fixes.
- Fan-in scaled initialization: Kaiming for convolutions,
1/sqrt(fan_in)for linear layers, instead of a fixed 0.02. - Per-image input standardization in the conv stem. Depictions are 1-3% ink, so raw input variance is ~0.01 and moves with augmentation.
RMSNormafter the stem's projection, so visual tokens enter the transformer at unit scale.- Cross-attention output projections excluded from the depth-based residual rescale. It is the only path carrying image information into the decoder; damping it at initialization makes the prior shortcut cheaper than looking at the picture.
ConvNeXtBlockLayerScale initialized at 0.1 rather than 1e-5. The small value is a very-deep-network device, pointless at three blocks per stage, and under bf16 a 1e-5 residual contribution falls below the mantissa of the activation it is added to -- the block would contribute nothing at all.
Measured effect (identical 800-step run on the real corpus, before/after):
| encoder memory cosine | token spread | |
|---|---|---|
| before | 1.00000 (collapsed) | 0.0056, decaying |
| after | 0.61, still falling | 0.70, growing |
Step-0 gradients after the fix: encoder 8.7, decoder 11.3.
Guards added so this cannot recur silently.
tests/test_init_health.pyasserts the two towers' step-0 gradients are within 50x, that the encoder distinguishes real rendered depictions, that zeroing the memory changes the decoder logits, and that stem activations are in a trainable range. Verified to fail on the pre-fix code (gradient imbalance 5754x,stem output std 0.01742) and pass after.Trainer.encoder_healthlogs memory cosine and token spread at every logging interval, and the run aborts if cosine exceeds 0.995 after step 500. A dead run now stops instead of consuming its budget.
Lesson worth stating plainly. For an encoder-decoder where one side can solve part of the task alone, a falling loss is not evidence that both sides are learning. The diagnostic has to be a direct measurement of whether the conditioning signal is used.
D19. The RTX 6000 Ada, not the A100¶
The assumption this replaces. The A100 was treated as the training target
throughout (D12, and the base preset comment in models/molmini.py), on the
usual reasoning that a data-center card beats a workstation card by 1.3-1.8x.
For this model that reasoning is simply wrong, and it cost nothing to check:
scripts/slurm/probe_roofline.sbatch measures both ceilings in isolation.
Evidence -- GPU ceiling (synthetic batches, no dataloader, so no H2D and no collate; 23.8M params at 384 px, img/s):
| batch | Ada (sm_89, 142 SM) | A100 80GB PCIe (sm_80, 108 SM) |
|---|---|---|
| 16 bf16 | 327.1 | 296.5 |
| 32 bf16 | 453.5 | 379.2 |
| 64 bf16 | 440.6 | 438.7 |
| 64 fp32 | 228.2 | 199.0 |
The Ada saturates at batch 32 -- going to 64 lowers throughput, 453 -> 441 -- while the A100 is still climbing at 64. That asymmetry is the whole finding: a 24M-parameter model at 384 px issues matmuls too small to use the A100's scale, so what decides the race is clock and SM count, where the Ada leads 142 to 108. The A100's HBM bandwidth is never the binding resource here.
Evidence -- end-to-end, the number that actually decides. Same recipe
(base, batch 64, 14 render workers, bf16, 240k steps), same code, one card
swapped:
| preset | A100 | Ada | |
|---|---|---|---|
base @384 |
353.7 img/s (peacock04, alone on the node) | 397 img/s (peacock06, sharing CPUs with a second run) | +12% |
small @320 |
672 img/s | 761 img/s | +13% |
The Ada reaches 90% of its own ceiling; the A100 reached 82% of its. The Ada result is the conservative one -- it was measured while a second training job shared the node's cores, and it still won.
Decision. Train on gpu:RTX_6000_Ada:1. Hand the A100s back.
Consequence. base drops from ~12.1 h to ~10.7 h of wall clock for its
15.36M-sample budget, and the scarcer card is freed for other users -- peacock06
has three Ada GPUs that sat idle while its A100s were contended.
Evidence -- rendering is not the bottleneck, at the right worker count. Per-sample CPU work (render, augment, to-tensor) on the cluster Xeon scales near-linearly: 1 core 77.4 img/s, 4 cores 284, 6 cores 422, 8 cores 534. Extrapolated to the 14 workers a training job actually gets, ~935 img/s -- roughly twice the GPU ceiling, so the GPU is the constraint. But the margin is not large: the same probe's end-to-end section, restricted to 6 workers, landed at 400 img/s against that configuration's 422 img/s render ceiling, i.e. fully data-bound. Do not run below ~10 workers.
What is left on the table. At 14 workers the run still sits ~10-18% below the GPU ceiling. That gap is collate and host-to-device transfer -- 590 KB of float32 per sample -- not RDKit, so more cores will not close it. The fix, if it is ever worth the complexity, is to hand uint8 out of the worker and normalize on the GPU.
Caveat, stated so the table is not over-read. The A100 probe ran while a MolScribe benchmark shared the node; its CPU section shows the tell (8 cores slower than 6, 306 vs 326 img/s), so its end-to-end figure is depressed. The GPU-ceiling rows are far less sensitive to that, and the end-to-end comparison above is drawn from full training runs rather than from the contended probe.
D20. Greedy decoding is the default; beam search does not earn its cost¶
Measured, on 300 frozen validation images with the same checkpoint:
| decoding | exact | validity | wall clock |
|---|---|---|---|
| greedy | 0.743 | 0.990 | 34 s |
| beam-5 (length penalty 0.6) | 0.730 | 0.993 | 223 s |
Reading this honestly. At n=300 the standard error on a ~0.74 rate is about 2.5 points, so the 1.3-point gap is inside the noise. The correct statement is not "beam is worse" but "beam-5 shows no measurable benefit at 6.6x the cost". Validity improves slightly, which is consistent with beam search preferring well-formed sequences.
Ruling out an implementation bug. Beam sizes 2 and 5 reproduce greedy's output on 8/8 spot-checked images and score identically on them, so the beam path is functioning; the result is a property of the task, not a defect.
Decision. Greedy is the default, and it is what the reported numbers use.
Beam search stays available (--beam-size) because it may help a checkpoint
trained differently, but it is not recommended, and it cannot batch — beam
search decodes one image at a time, which is where most of the 6.6x comes from.
Why this matters for the stated goal. Running comfortably on a laptop is part of the requirement, and the decoding strategy that is 6.6x cheaper is also the more accurate one here. There is no accuracy/latency trade-off to manage.
D21. MolMini beats MolScribe on this benchmark, and that must not be reported as superiority¶
Measured, identical frozen manifest (chembl_test_linux, 3000 images),
MolMini small at step 60k of 240k, mid-training, before the LR decay phase:
| tier | MolMini 12.1 M | MolScribe ~90 M |
|---|---|---|
| clean | 0.835 | 0.823 |
| varied | 0.822 | 0.767 |
| degraded | 0.679 | 0.350 |
| pooled | 0.779 | 0.647 |
| validity | 0.985 | 0.934 |
Why this is not a superiority claim. The evaluation images are produced by the same renderer and the same augmentation pipeline MolMini trained on. The molecules are held out — splits are disjoint by canonical SMILES and asserted — but the rendering distribution is identical. This is MolMini's home turf and MolScribe's away turf.
The degraded tier makes it unmistakable: 0.679 against 0.350. MolMini trained on exactly those corruptions (this project's blur, JPEG, perspective, speckle, stroke morphology); MolScribe never saw them. A 2x gap on the tier that most directly reflects the training augmentation is a measurement of distribution match, not of recognition ability.
The reverse experiment confirms it. On DECIMER hand-drawn images — unfamiliar to MolMini, closer to what MolScribe was built for — the ordering flips: MolMini 0.036, MolScribe 0.099.
What the numbers do support:
- MolMini reads this depiction distribution very well, at 12.1 M parameters, on a laptop, mid-training.
- The training pipeline works: the augmentation is learnable and the model generalizes across held-out molecules within its distribution.
- A model trained on a distribution beats a stronger general model on that distribution. That is the expected result, and it is why benchmark provenance has to be stated before any comparison is read.
What they do not support: any claim that MolMini is a better OCSR system than MolScribe, or that it would transfer to patent figures, journal depictions, or scans. Nothing here tests that.
How the report must present it. Distribution provenance stated before the table, both directions reported together (in-distribution and hand-drawn), and no headline of the form "small model beats MolScribe".
D22. The held-out split is far less novel than SMILES-disjointness implies¶
This materially qualifies every accuracy number in this project.
Splits are disjoint by isomeric canonical SMILES and asserted at build time. That is the standard guarantee. It is also much weaker than it sounds.
Measured: maximum Morgan(r=2) Tanimoto from each of 400 held-out molecules to all 2,618,349 training molecules.
| statistic | value |
|---|---|
| median | 0.806 |
| 25th / 75th percentile | 0.729 / 0.868 |
| 95th percentile | 1.000 |
| share with a near-twin (>= 0.9) | 0.158 |
| share with a close analogue (>= 0.7) | 0.833 |
| share genuinely novel (< 0.5) | 0.007 |
83% of held-out molecules have a close analogue in training and 16% have a near-twin. A 95th percentile of 1.000 means at least 5% are fingerprint-identical to a training molecule -- Morgan fingerprints ignore stereochemistry by default, so a stereoisomer of a training molecule is a different isomeric SMILES, passes the disjointness assertion, and is fingerprint-identical.
This is what ChEMBL is: a database organized around congeneric series, where hundreds of analogues of one scaffold are deposited together. A random split of it cannot produce novel chemotypes.
Why the first measurement was wrong. Against a 60,000-molecule sample (2.3% of the corpus) the same script reported median 0.509 and a 0.5% near-twin share. A maximum taken over a sample underestimates the true nearest neighbour, and here it did so drastically -- the sampled figure would have supported a claim that is simply false. The sampled run was labelled optimistic when it was produced; the size of the gap was still a surprise.
What this means for the reported accuracy. Exact-match scores on the rendered ChEMBL test set substantially reflect interpolation within congeneric series, not recognition of unfamiliar chemotypes. The model is being asked to read a drawing of a molecule very similar to ones it has seen thousands of times.
What it does not undermine. The images are still unseen, and the
degraded tier still measures robustness to corruption. The DECIMER hand-drawn
result -- where MolMini scores 0.036 -- is unaffected and is the honest measure of
transfer to unfamiliar input.
What should be done differently. A scaffold split (Bemis-Murcko) or a similarity-capped split would give a defensible measure of generalization to new chemotypes. That requires retraining and is out of scope for this run; the report states the limitation rather than quietly reporting the random-split number as if it measured novelty. (Superseded in part by D23: the scaffold axis was subsequently tested directly and shows no effect on accuracy, which narrows what a retrained split would add to the fingerprint-similarity and stereoisomer axes.)
Addendum to D22 — the stereoisomer channel, counted. Of the 5,000 test molecules, 409 (8.2%) have a training molecule with the same stereo-flattened skeleton. They are different isomeric SMILES, so they pass the disjointness assertion, but the model has already seen the 2D structure and only the wedge/hash reading is new. Not pure leakage — stereochemistry is genuinely part of the OCSR task — but for one test molecule in twelve, the skeleton-recognition half is memorized. (Within the training set itself, 2,618,349 molecules reduce to 2,497,511 distinct skeletons.)
D23. Accuracy does not degrade on unseen scaffolds — which partly answers D22¶
D22 established that the random split has little chemotype novelty: 80% of test molecules are built on a Bemis-Murcko scaffold that appears in training. The obvious worry is that the headline accuracy is inflated by scaffold familiarity. This tests it directly, without the retrain a proper scaffold split would need.
The 965 test molecules (19.3% of 5,000) whose scaffold appears nowhere in the 2.6M-molecule training corpus were rendered into a frozen three-tier eval set and scored with the same checkpoint used on the full test set.
| set | molecules | exact | clean | varied | degraded |
|---|---|---|---|---|---|
| full test set | 1000 | 0.779 | 0.835 | 0.822 | 0.679 |
| novel scaffolds | 965 | 0.811 | 0.860 | 0.849 | 0.725 |
Stated carefully. The difference is +0.033 with a 95% interval of ±0.036 — 1.8 sigma, not significant at p < 0.05. The defensible claim is no detectable degradation on unseen scaffolds, not that novel scaffolds are easier.
An external difficulty calibration, which is the strongest evidence here. MolScribe was scored on both subsets too. It never trained on this ChEMBL split at all, so its score is an independent probe of how hard each subset is:
| model | full test set | novel scaffolds | difference |
|---|---|---|---|
| MolScribe (external) | 0.647 | 0.644 | -0.002 |
| MolMini | 0.779 | 0.811 | +0.033 |
MolScribe finds the two subsets equally hard. The novel-scaffold subset is therefore not intrinsically easier, and MolMini's result cannot be explained by subset difficulty. This is better evidence than the property-matching below, because it does not depend on having thought of the right property to match.
That both an externally-trained model and a model trained on this corpus show no degradation on unseen scaffolds is what one would expect if OCSR is genuinely a visual task: reading a drawing does not require having seen that skeleton before.
Confounds checked and ruled out. The novel-scaffold subset could have been easier for reasons unrelated to scaffolds. It is not: compared with the full test set it has slightly larger molecules (mean 29.2 vs 28.4 heavy atoms), larger scaffolds (median 25 vs 22 atoms), the same ring count (median 4), the same stereocentre density (mean 1.18 vs 1.17), a similar share carrying any stereochemistry (0.507 vs 0.527), and the same median SMILES length (50).
What this means. The concern raised in D22 — that a low-novelty split inflates the number — is not borne out for scaffold familiarity specifically. The model reads molecules built on skeletons it has never seen just as well as familiar ones, which is what one would hope from a system doing visual recognition rather than retrieval from a memorized chemical vocabulary.
What it does not resolve. Scaffold novelty is one axis. 83% of held-out molecules still have a close analogue by fingerprint similarity, and the stereoisomer channel (8.2%, D22 addendum) is untouched by this test. A scaffold split at training time remains the right experiment.
D24. Capacity buys in-distribution accuracy and costs out-of-distribution transfer¶
Both presets trained to completion on identical data, schedule and sample budget (240,000 steps, 15.36M samples), so the comparison is clean.
| evaluation | small 12.1M |
base 23.8M |
change |
|---|---|---|---|
| rendered, training fonts | 0.857 | 0.897 | +0.040 |
| rendered, unseen fonts | 0.859 | 0.888 | +0.029 |
| rendered, novel scaffolds | 0.866 | 0.894 | +0.028 |
| hand-drawn (out of distribution) | 0.057 | 0.037 | -0.020 |
The larger model is better everywhere inside its training distribution and worse outside it. Doubling capacity bought a 4-point gain on rendered images and cost 2 points on hand-drawn ones.
Reading this carefully. On 1000 hand-drawn images the standard error at a ~0.05 rate is about 0.7 points, so a 2-point gap is roughly 2.9 sigma -- real, but small in absolute terms. Both models are poor at hand-drawn input; this is a difference between two weak results, not a reversal of who is usable there. MolScribe, at 0.099, beats both.
The mechanism is the obvious one. Extra capacity is spent fitting the training distribution more exactly, and the training distribution is this project's renderer plus augmentation pipeline. Nothing in it produces human line breaks or inconsistent bond lengths, so the sharper fit does not transfer.
Practical consequence. For reading rendered or programmatically generated
depictions, base is the better model. For anything closer to hand-drawn input,
neither is adequate and small is marginally less bad -- but the honest answer
there is MolScribe, or training on data that actually contains hand-drawn
depictions.
What this says about the augmentation. It is the specification (D5), and this is the clearest evidence of that: the model that fits it best generalizes outside it worst.