Pic2SMILES / MolMini — technical report¶
1. What was asked, and what was delivered¶
Train a small optical chemical structure recognition (OCSR) model — picture of a 2D chemical structure in, canonical SMILES out — that runs comfortably on a 16 GB M3 MacBook Pro, using current small-model practice, and report it reproducibly.
Delivered: MolMini, an image-to-SMILES encoder-decoder in three sizes (3.6 M / 12.1 M / 23.8 M parameters), trained from scratch on 2.6 M ChEMBL molecules rendered on the fly, plus a benchmark harness that scores it against MolScribe and two reference points on identical frozen images.
chembl_test_linux
| model | params | checkpoint | n | valid | exact | exact/no-stereo | tanimoto | measured on |
|---|---|---|---|---|---|---|---|---|
| oracle | 0 | - | 3000 | 1.000 | 1.000 | 1.000 | 1.000 | Darwin-arm64 |
| molmini | 23.8M | step 240,000 | 3000 | 0.997 | 0.897 | 0.937 | 0.965 | Darwin-arm64 |
| molmini | 12.1M | step 240,000 | 3000 | 0.998 | 0.857 | 0.927 | 0.958 | Darwin-arm64 |
| molscribe | pretrained external | - | 900 | 0.934 | 0.647 | 0.760 | 0.892 | Linux-x86_64 |
| image_hash_nn | 0 | - | 3000 | 1.000 | 0.000 | 0.000 | 0.118 | Darwin-arm64 |
2. Starting point¶
The repository contained a benchmark harness and no model. Its only learning
code trained a closed-set classifier over 60 hardcoded SMILES, reaching 5.6%
exact match on its own synthetic data; six of eleven registered "models" were
placeholders that could only emit skipped. A latent bug in _TinyViT stored
layers on a plain attribute, so parameters() returned an unregistered set and
.to(device) moved a shadow copy.
A closed-set classifier cannot become an open-vocabulary recognizer by tuning — the output space is wrong — so the modelling half was rebuilt. What was kept: canonical-SMILES scoring, the "unavailable systems are skipped, not failed" contract, and the frozen-manifest format.
3. Method¶
3.1 Output representation¶
Autoregressive SMILES decoding, not graph decoding. This follows from the supervision actually available: training images come from RDKit rendering, which yields an exact SMILES per image and no atom coordinates, while graph decoders (MolGrapher, MolScribe's graph head) need atom-level coordinate supervision. Graph decoding is the stronger formulation at the top of the field; this is a supervision-availability decision, not a claim otherwise.
3.2 Architecture¶
384x384x1
-> conv 7x7 stride 4 -> 96x96x64
-> conv stride 2 + 3 ConvNeXt blocks -> 48x48x128
-> conv stride 2 + 3 ConvNeXt blocks -> 24x24x256
-> linear + RMSNorm -> 576 tokens x 384
-> 4 transformer blocks, axial 2D RoPE
-> 6 causal decoder blocks, 1D RoPE, cross-attention, tied embeddings
Convolutions carry pixels down to a 24x24 token grid because early OCSR features (strokes, junctions, glyphs) are local; global attention then runs only on the 576 surviving tokens, where relating distant parts of a drawing — the two ends of a ring closure — actually requires it. RMSNorm, SwiGLU, QK-norm, fan-in scaled init, WSD schedule, weight EMA.
Resolution follows the declared scope: at 224 px, atom labels and charges inside dense fused-ring systems stop being legible.
3.3 Data¶
Training images are generated fresh every step. Rendering costs ~2 ms of CPU
against a far more expensive GPU step, so a frozen image set would buy nothing
and cost generalization. Every render randomizes layout engine, rotation, bond
width, font, atom palette, condensed abbreviations (Ph, OMe) and comic mode;
the image is then corrupted with stroke morphology, blur, downscaling, JPEG,
perspective warp, paper texture and speckle.
The consequence, stated plainly: the augmentation distribution is the specification. Anything absent from it is out of distribution at test time regardless of compute spent.
3.4 Scope and leakage control¶
| constraint | value |
|---|---|
| heavy atoms | 4 to 48 |
| elements | B, Br, C, Cl, F, I, N, O, P, S, Se, Si |
| formal charges | allowed |
| multi-fragment | excluded |
| curation | count |
|---|---|
| read | 2,897,819 |
| rejected (unparseable or out of scope) | 269,119 |
| duplicate by canonical SMILES | 141 |
| removed by the DECIMER leakage guard | 210 |
| kept | 2,628,559 |
| train / val / test | 2,618,349 / 5,000 / 5,000 |
| vocabulary | 88 tokens |
210 molecules appear in both ChEMBL and the DECIMER hand-drawn evaluation set. They were subtracted from training by canonical SMILES before splitting. Without that, any claim about generalizing to real hand-drawn structures would have been partly a memorization measurement.
4. The failure that cost a training run¶
The first two runs were cancelled at ~10k steps. Loss fell normally and teacher-forced token accuracy reached 0.77, but greedy decoding returned the same SMILES string for every image.
Measured: encoder memory cosine similarity 1.00004 between two different images. The encoder emitted one constant vector regardless of input and the decoder had learned an unconditional SMILES prior — which reaches 0.6–0.77 token accuracy on its own, so the loss curve showed nothing.
An 8-pair overfit test memorized 8/8, ruling out an architecture bug. Per-module gradient norms at step 0 located it:
| gradient norm, step 0 | |
|---|---|
| encoder | 108,675 |
| decoder | 12.2 |
init_weights applied a fixed trunc_normal_(std=0.02) everywhere. That constant
is calibrated for GPT-2's width; on a convolution with fan-in 576 it attenuated
activations to std 0.017, and normalization backward gain scales as 1/RMS(input),
so tiny activations became enormous gradients. clip_grad_norm_(1.0) then divided
the whole gradient by ~1e5 — the decoder received no update and the encoder was
destroyed on step one.
Fixes: fan-in scaled init, per-image input standardization in the stem, RMSNorm after the stem projection, cross-attention excluded from the depth-based residual rescale, ConvNeXt LayerScale 1e-5 → 0.1 (1e-5 falls below the bf16 mantissa of the activation it is added to).
| identical 800-step run | memory cosine | token spread |
|---|---|---|
| before | 1.00000 (collapsed) | 0.0056, decaying |
| after | 0.61, still falling | 0.70, growing |
Generalizable lesson. For an encoder-decoder where one side can solve part of the task alone, a falling loss is not evidence that both sides are learning. The diagnostic has to directly measure whether the conditioning signal is used.
Guards: tests/test_init_health.py (verified to fail on the pre-fix code) and
Trainer.encoder_health, which logs memory cosine every interval and aborts the
run above 0.995.
5. Evaluation design¶
Three properties make the numbers mean something:
- Frozen images. Evaluation images are written once with a fixed seed. With on-the-fly evaluation, a regression and an unlucky render are indistinguishable.
- A validated scoring path.
oraclereads answers from the manifest and must score exactly 1.000. It does, on all 3000 images of both sets. - A memorization floor.
image_hash_nnretrieves the nearest training image. It scores 0.000, so anything the model earns above that line is recognition rather than lookup.
Three difficulty tiers — clean, varied, degraded — plus the DECIMER
hand-drawn set as the out-of-distribution test.
Font control. The two hosts offer disjoint font families (DejaVu on Linux, Arial/Georgia on macOS). Rather than hide that, the eval set was built on both hosts from the same molecules and seed. MolScribe scores 0.647 and 0.646 across the pair — a 0.001 gap — which establishes that the font families do not move a competent OCSR model, so any gap MolMini shows there is its own font sensitivity.
How novel are the held-out molecules?¶
Splits are disjoint by isomeric canonical SMILES and asserted at build time. Measured against all 2,618,349 training molecules, that guarantee turns out to be much weaker than it sounds:
| statistic | value |
|---|---|
| median maximum Morgan(r=2) Tanimoto to training | 0.806 |
| 95th percentile | 1.000 |
| share with a near-twin (>= 0.9) | 0.158 |
| share with a close analogue (>= 0.7) | 0.833 |
| share genuinely novel (< 0.5) | 0.007 |
83% of held-out molecules have a close analogue in training. A 95th percentile of 1.000 means at least 5% are fingerprint-identical: Morgan ignores stereochemistry, so a stereoisomer of a training molecule is a different isomeric SMILES, passes the disjointness assertion, and is fingerprint-identical.
Quantified precisely: 409 of the 5,000 test molecules (8.2%) share their stereo-flattened skeleton with a training molecule. For those, the model has already seen the 2D structure and only the wedge/hash reading is genuinely new. That is not pure leakage — stereochemistry is part of what an OCSR model must read off the image — but the skeleton-recognition half of the task is memorized for one test molecule in twelve.
This is what ChEMBL is — congeneric series deposited together — and a random split of it cannot produce novel chemotypes. The natural conclusion is that exact-match on the rendered test set substantially measures interpolation within congeneric series rather than recognition of unfamiliar scaffolds.
That conclusion is testable without retraining, and it does not survive the test. The 965 test molecules (19.3%) whose Bemis-Murcko scaffold appears nowhere in the training corpus were scored separately:
| set | molecules | exact |
|---|---|---|
| full test set | 1000 | 0.779 |
| novel scaffolds only | 965 | 0.811 |
The difference is +0.033 against a 95% interval of ±0.036 — 1.8 sigma, not significant. So the honest reading is no detectable degradation on unseen scaffolds, not that they are easier. Confounds were checked: the novel-scaffold subset has slightly larger molecules (29.2 vs 28.4 heavy atoms), larger scaffolds (25 vs 22 atoms), and matched ring count, stereocentre density and SMILES length.
MolScribe was scored on both subsets as an external calibration. It never trained on this ChEMBL split, so its score probes how hard each subset is independently of any model trained here:
| model | full test set | novel scaffolds | difference |
|---|---|---|---|
| MolScribe (external) | 0.647 | 0.644 | -0.002 |
| MolMini | 0.779 | 0.811 | +0.033 |
MolScribe finds the two equally hard, so the novel-scaffold subset is not intrinsically easier and MolMini's result is not a subset-difficulty artifact. Two models — one trained on this corpus, one not — both show no degradation on unseen scaffolds, which is what one would expect if OCSR is genuinely a visual task: reading a drawing does not require having seen that skeleton before.
That is reassuring — the model appears to be doing visual recognition rather than retrieval from a memorized chemical vocabulary — but it addresses only the scaffold axis. The fingerprint-similarity result above and the stereoisomer channel are untouched by it.
Also unaffected: the images are still unseen, the degraded tier still measures
robustness to corruption, and the DECIMER hand-drawn result remains the honest
measure of transfer to unfamiliar input.
Where this leaves the split. Low chemotype novelty is a real property of the data and is stated rather than hidden, but the one axis that could be tested directly — scaffold familiarity — shows no effect on accuracy. A scaffold or similarity-capped split at training time remains worth running, since it would also cover the fingerprint-similarity and stereoisomer axes this test does not.
What this benchmark does and does not measure¶
This must be read before the results table.
The rendered evaluation sets are produced by the same renderer and the same augmentation pipeline the model trained on. The molecules are held out — splits are disjoint by canonical SMILES and asserted at build time — but the rendering distribution is identical. So these sets measure:
given depictions drawn from the distribution MolMini was trained on, and molecules it has never seen, how well does it read them?
That is a real and useful question. It is not the question "is MolMini a
better OCSR system than MolScribe". Any external system is being scored away
from its own training distribution here, and this project's model is being
scored at home. A large margin on the degraded tier in particular reflects
that the corruption pipeline is MolMini's and not MolScribe's.
The DECIMER hand-drawn set is the control that keeps this honest: it is unfamiliar to MolMini and closer to what MolScribe was built for. The ordering there is mixed rather than a clean reversal -- the 12.1M model scores above MolScribe (0.125 against 0.099) while the 23.8M model scores below it (0.088) -- and in absolute terms every system is weak on this set. Both directions are reported together, always.
6. Results¶
chembl_test_linux
| model | params | checkpoint | n | valid | exact | exact/no-stereo | tanimoto | measured on |
|---|---|---|---|---|---|---|---|---|
| oracle | 0 | - | 3000 | 1.000 | 1.000 | 1.000 | 1.000 | Darwin-arm64 |
| molmini | 23.8M | step 240,000 | 3000 | 0.997 | 0.897 | 0.937 | 0.965 | Darwin-arm64 |
| molmini | 12.1M | step 240,000 | 3000 | 0.998 | 0.857 | 0.927 | 0.958 | Darwin-arm64 |
| molscribe | pretrained external | - | 900 | 0.934 | 0.647 | 0.760 | 0.892 | Linux-x86_64 |
| image_hash_nn | 0 | - | 3000 | 1.000 | 0.000 | 0.000 | 0.118 | Darwin-arm64 |
| style | model | params | checkpoint | n | valid | exact | tanimoto |
|---|---|---|---|---|---|---|---|
| clean | image_hash_nn | 0 | - | 1000 | 1.000 | 0.000 | 0.129 |
| clean | molmini | 12.1M | step 240,000 | 1000 | 0.999 | 0.917 | 0.996 |
| clean | molmini | 23.8M | step 240,000 | 1000 | 0.998 | 0.952 | 0.997 |
| clean | molscribe | pretrained external | - | 300 | 0.977 | 0.823 | 0.995 |
| clean | oracle | 0 | - | 1000 | 1.000 | 1.000 | 1.000 |
| degraded | image_hash_nn | 0 | - | 1000 | 1.000 | 0.000 | 0.112 |
| degraded | molmini | 12.1M | step 240,000 | 1000 | 0.995 | 0.748 | 0.882 |
| degraded | molmini | 23.8M | step 240,000 | 1000 | 0.997 | 0.806 | 0.901 |
| degraded | molscribe | pretrained external | - | 300 | 0.863 | 0.350 | 0.678 |
| degraded | oracle | 0 | - | 1000 | 1.000 | 1.000 | 1.000 |
| varied | image_hash_nn | 0 | - | 1000 | 1.000 | 0.000 | 0.114 |
| varied | molmini | 12.1M | step 240,000 | 1000 | 0.999 | 0.905 | 0.996 |
| varied | molmini | 23.8M | step 240,000 | 1000 | 0.996 | 0.934 | 0.997 |
| varied | molscribe | pretrained external | - | 300 | 0.963 | 0.767 | 0.978 |
| varied | oracle | 0 | - | 1000 | 1.000 | 1.000 | 1.000 |
chembl_test_novel_scaffold
| model | params | checkpoint | n | valid | exact | exact/no-stereo | tanimoto | measured on |
|---|---|---|---|---|---|---|---|---|
| molmini | 23.8M | step 240,000 | 2895 | 0.996 | 0.894 | 0.936 | 0.965 | Darwin-arm64 |
| molmini | 12.1M | step 240,000 | 2895 | 0.993 | 0.866 | 0.927 | 0.960 | Darwin-arm64 |
| molscribe | pretrained external | - | 900 | 0.910 | 0.644 | 0.757 | 0.898 | Linux-x86_64 |
| style | model | params | checkpoint | n | valid | exact | tanimoto |
|---|---|---|---|---|---|---|---|
| clean | molmini | 12.1M | step 240,000 | 965 | 0.998 | 0.917 | 0.993 |
| clean | molmini | 23.8M | step 240,000 | 965 | 0.998 | 0.938 | 0.994 |
| clean | molscribe | pretrained external | - | 300 | 0.987 | 0.827 | 0.988 |
| degraded | molmini | 12.1M | step 240,000 | 965 | 0.989 | 0.777 | 0.895 |
| degraded | molmini | 23.8M | step 240,000 | 965 | 0.995 | 0.808 | 0.908 |
| degraded | molscribe | pretrained external | - | 300 | 0.797 | 0.347 | 0.692 |
| varied | molmini | 12.1M | step 240,000 | 965 | 0.994 | 0.904 | 0.993 |
| varied | molmini | 23.8M | step 240,000 | 965 | 0.996 | 0.936 | 0.993 |
| varied | molscribe | pretrained external | - | 300 | 0.947 | 0.760 | 0.978 |
chembl_test_macfonts
| model | params | checkpoint | n | valid | exact | exact/no-stereo | tanimoto | measured on |
|---|---|---|---|---|---|---|---|---|
| oracle | 0 | - | 3000 | 1.000 | 1.000 | 1.000 | 1.000 | Darwin-arm64 |
| molmini | 23.8M | step 240,000 | 3000 | 0.998 | 0.888 | 0.926 | 0.962 | Darwin-arm64 |
| molmini | 12.1M | step 240,000 | 3000 | 0.997 | 0.859 | 0.925 | 0.959 | Darwin-arm64 |
| molscribe | pretrained external | - | 900 | 0.913 | 0.646 | 0.757 | 0.910 | Linux-x86_64 |
| image_hash_nn | 0 | - | 3000 | 1.000 | 0.000 | 0.000 | 0.119 | Darwin-arm64 |
| style | model | params | checkpoint | n | valid | exact | tanimoto |
|---|---|---|---|---|---|---|---|
| clean | image_hash_nn | 0 | - | 1000 | 1.000 | 0.000 | 0.130 |
| clean | molmini | 12.1M | step 240,000 | 1000 | 0.996 | 0.917 | 0.996 |
| clean | molmini | 23.8M | step 240,000 | 1000 | 0.997 | 0.949 | 0.997 |
| clean | molscribe | pretrained external | - | 300 | 0.977 | 0.823 | 0.996 |
| clean | oracle | 0 | - | 1000 | 1.000 | 1.000 | 1.000 |
| degraded | image_hash_nn | 0 | - | 1000 | 1.000 | 0.000 | 0.114 |
| degraded | molmini | 12.1M | step 240,000 | 1000 | 0.994 | 0.771 | 0.890 |
| degraded | molmini | 23.8M | step 240,000 | 1000 | 0.997 | 0.794 | 0.901 |
| degraded | molscribe | pretrained external | - | 300 | 0.803 | 0.357 | 0.721 |
| degraded | oracle | 0 | - | 1000 | 1.000 | 1.000 | 1.000 |
| varied | image_hash_nn | 0 | - | 1000 | 1.000 | 0.000 | 0.113 |
| varied | molmini | 12.1M | step 240,000 | 1000 | 1.000 | 0.890 | 0.990 |
| varied | molmini | 23.8M | step 240,000 | 1000 | 0.999 | 0.922 | 0.987 |
| varied | molscribe | pretrained external | - | 300 | 0.960 | 0.757 | 0.981 |
| varied | oracle | 0 | - | 1000 | 1.000 | 1.000 | 1.000 |
decimer_hdm
Corrected after issue #4. These hand-drawn numbers were re-measured once
image_to_tensor stopped resizing non-square input without preserving aspect
ratio; 179 of 200 sampled DECIMER images are non-square, so every one of them
had been reaching the model stretched. On the same 1000 images with the same
checkpoints, exact match moved from 0.037 to 0.088 (23.8M) and from 0.057 to
0.125 (12.1M). The rendered evaluation sets are square 384x384 renders, where
the padding is a no-op, so their numbers are unchanged. MolScribe runs through
its own preprocessing and is unaffected.
| model | params | checkpoint | n | valid | exact | exact/no-stereo | tanimoto | measured on |
|---|---|---|---|---|---|---|---|---|
| molscribe | pretrained external | - | 1000 | 0.852 | 0.099 | 0.126 | 0.392 | Linux-x86_64 |
| molmini | 12.1M | step 240,000 | 1000 | 0.934 | 0.125 | 0.170 | 0.403 | Darwin-arm64 |
| molmini | 23.8M | step 240,000 | 1000 | 0.950 | 0.088 | 0.112 | 0.336 | Darwin-arm64 |
| style | model | params | checkpoint | n | valid | exact | tanimoto |
|---|---|---|---|---|---|---|---|
| hand_drawn | molmini | 12.1M | step 240,000 | 1000 | 0.934 | 0.125 | 0.403 |
| hand_drawn | molmini | 23.8M | step 240,000 | 1000 | 0.950 | 0.088 | 0.336 |
| hand_drawn | molscribe | pretrained external | - | 1000 | 0.852 | 0.099 | 0.392 |
A caveat on the degraded tier. The corruption pipeline samples its
parameters independently, and at the extreme end it produces images that are not
legible to a human either — heavy paper texture plus blur can bury a thin-bonded
structure entirely. Some of the degraded failures are therefore not the model
failing to read a readable drawing. That tier should be read as "robustness
under corruption up to and including illegibility", not as a clean lower bound
on real-world scans.
7. Limitations¶
- The random split does not test novel chemotypes. 83% of held-out molecules have a close analogue in training (see section 5). Accuracy on the subset with genuinely unseen scaffolds shows no degradation, so this does not appear to inflate the number; the fingerprint-similarity and stereoisomer axes remain untested and would need a scaffold or similarity-capped split at training time.
- The rendered benchmark is this model's home turf. Any external system is scored away from its training distribution on it. The hand-drawn set is the counterweight, and there the gap closes to a few points in either direction depending on model size -- no system reads these drawings well.
- Hand-drawn depictions are out of distribution. RDKit
comicModegives wobbly strokes but does not reproduce human line breaks, inconsistent bond lengths or misplaced labels. - Markush structures, R groups, reaction schemes, multi-molecule pages: not in the corpus, not rendered, not supported.
- Scope-bounded. Numbers do not transfer outside the declared element set and heavy-atom range.
- Training is not bit-deterministic. GPU kernel scheduling is not; a repeat run lands near, not exactly on, these numbers.
[eval]lines in a training log are not comparable to benchmark tables — different molecules, different pixels, resampled every call.
8. Reproduction¶
See docs/REPRODUCE.md for every command. Design rationale with the measurement
behind each choice is in docs/DECISIONS.md.