Skip to content

Pic2SMILES / MolMini benchmark

Generated: 2026-08-20T10:43:53
Host: macOS-27.0-arm64-arm-64bit | Python 3.11.16

Declared scope

The model was trained and is evaluated only inside this molecular scope. Numbers below do not transfer to molecules outside it.

{
  "source": "/home/dzhou/pic2smiles/data/corpus/chembl_37_smiles.txt",
  "holdout": "/home/dzhou/pic2smiles/data/corpus/decimer_holdout.smi",
  "holdout_removed": 210,
  "scope": {
    "min_heavy_atoms": 4,
    "max_heavy_atoms": 48,
    "allowed_elements": [
      "B",
      "Br",
      "C",
      "Cl",
      "F",
      "I",
      "N",
      "O",
      "P",
      "S",
      "Se",
      "Si"
    ],
    "allow_charges": true,
    "allow_multi_fragment": false,
    "max_smiles_tokens": 160
  },
  "curate_stats": {
    "read": 2897819,
    "rejected": 269119,
    "duplicate": 141,
    "kept": 2628559
  },
  "splits": {
    "train": 2618349,
    "val": 5000,
    "test": 5000
  },
  "vocab_size": 88,
  "val_molecules_with_unknown_tokens": 1,
  "seed": 20260819,
  "generated_at": "2026-08-19T21:54:23"
}

Headline results (all styles pooled)

model params dataset n skipped valid exact exact/no-stereo formula tanimoto edit dist ms/img checkpoint measured on
oracle 0 chembl_test_linux 3000 0 1.000 1.000 1.000 1.000 1.000 0.000 0.0 - Darwin-arm64
molmini 23.8M chembl_test_linux 3000 0 0.997 0.897 0.937 0.941 0.965 0.030 78.7 step 240,000 Darwin-arm64
molmini 12.1M chembl_test_linux 3000 0 0.998 0.857 0.927 0.929 0.958 0.039 56.4 step 240,000 Darwin-arm64
molscribe pretrained external chembl_test_linux 900 0 0.934 0.647 0.760 0.761 0.892 0.107 741.1 - Linux-x86_64
image_hash_nn 0 chembl_test_linux 3000 0 1.000 0.000 0.000 0.000 0.118 0.779 0.6 - Darwin-arm64
oracle 0 chembl_test_macfonts 3000 0 1.000 1.000 1.000 1.000 1.000 0.000 0.0 - Darwin-arm64
molmini 23.8M chembl_test_macfonts 3000 0 0.998 0.888 0.926 0.929 0.962 0.035 79.1 step 240,000 Darwin-arm64
molmini 12.1M chembl_test_macfonts 3000 0 0.997 0.859 0.925 0.928 0.959 0.039 56.4 step 240,000 Darwin-arm64
molscribe pretrained external chembl_test_macfonts 900 0 0.913 0.646 0.757 0.760 0.910 0.087 762.3 - Linux-x86_64
image_hash_nn 0 chembl_test_macfonts 3000 0 1.000 0.000 0.000 0.000 0.119 0.780 0.6 - Darwin-arm64
molmini 23.8M chembl_test_novel_scaffold 2895 0 0.996 0.894 0.936 0.940 0.965 0.032 79.2 step 240,000 Darwin-arm64
molmini 12.1M chembl_test_novel_scaffold 2895 0 0.993 0.866 0.927 0.930 0.960 0.038 56.7 step 240,000 Darwin-arm64
molscribe pretrained external chembl_test_novel_scaffold 900 0 0.910 0.644 0.757 0.760 0.898 0.097 783.3 - Linux-x86_64
molscribe pretrained external decimer_hdm 1000 0 0.852 0.099 0.126 0.127 0.392 0.444 598.2 - Linux-x86_64
molmini 12.1M decimer_hdm 1000 0 0.911 0.057 0.088 0.092 0.307 0.586 47.3 step 240,000 Darwin-arm64
molmini 23.8M decimer_hdm 1000 0 0.925 0.037 0.049 0.051 0.237 0.690 70.0 step 240,000 Darwin-arm64

Per-style breakdown

model dataset style n valid exact tanimoto ms/img
image_hash_nn chembl_test_linux clean 1000 1.000 0.000 0.129 0.5
image_hash_nn chembl_test_linux degraded 1000 1.000 0.000 0.112 0.9
image_hash_nn chembl_test_linux varied 1000 1.000 0.000 0.114 0.5
molmini chembl_test_linux clean 1000 0.998 0.952 0.997 79.5
molmini chembl_test_linux clean 1000 0.999 0.917 0.996 56.0
molmini chembl_test_linux degraded 1000 0.997 0.806 0.901 79.2
molmini chembl_test_linux degraded 1000 0.995 0.748 0.882 56.7
molmini chembl_test_linux varied 1000 0.996 0.934 0.997 77.7
molmini chembl_test_linux varied 1000 0.999 0.905 0.996 55.4
molscribe chembl_test_linux clean 300 0.977 0.823 0.995 734.2
molscribe chembl_test_linux degraded 300 0.863 0.350 0.678 798.2
molscribe chembl_test_linux varied 300 0.963 0.767 0.978 741.1
oracle chembl_test_linux clean 1000 1.000 1.000 1.000 0.0
oracle chembl_test_linux degraded 1000 1.000 1.000 1.000 0.0
oracle chembl_test_linux varied 1000 1.000 1.000 1.000 0.0
image_hash_nn chembl_test_macfonts clean 1000 1.000 0.000 0.130 0.5
image_hash_nn chembl_test_macfonts degraded 1000 1.000 0.000 0.114 1.0
image_hash_nn chembl_test_macfonts varied 1000 1.000 0.000 0.113 0.6
molmini chembl_test_macfonts clean 1000 0.997 0.949 0.997 79.1
molmini chembl_test_macfonts clean 1000 0.996 0.917 0.996 56.1
molmini chembl_test_macfonts degraded 1000 0.997 0.794 0.901 79.8
molmini chembl_test_macfonts degraded 1000 0.994 0.771 0.890 57.1
molmini chembl_test_macfonts varied 1000 0.999 0.922 0.987 77.5
molmini chembl_test_macfonts varied 1000 1.000 0.890 0.990 55.3
molscribe chembl_test_macfonts clean 300 0.977 0.823 0.996 762.3
molscribe chembl_test_macfonts degraded 300 0.803 0.357 0.721 819.8
molscribe chembl_test_macfonts varied 300 0.960 0.757 0.981 760.5
oracle chembl_test_macfonts clean 1000 1.000 1.000 1.000 0.0
oracle chembl_test_macfonts degraded 1000 1.000 1.000 1.000 0.0
oracle chembl_test_macfonts varied 1000 1.000 1.000 1.000 0.0
molmini chembl_test_novel_scaffold clean 965 0.998 0.938 0.994 79.7
molmini chembl_test_novel_scaffold clean 965 0.998 0.917 0.993 56.6
molmini chembl_test_novel_scaffold degraded 965 0.995 0.808 0.908 79.2
molmini chembl_test_novel_scaffold degraded 965 0.989 0.777 0.895 57.5
molmini chembl_test_novel_scaffold varied 965 0.996 0.936 0.993 79.2
molmini chembl_test_novel_scaffold varied 965 0.994 0.904 0.993 56.6
molscribe chembl_test_novel_scaffold clean 300 0.987 0.827 0.988 783.3
molscribe chembl_test_novel_scaffold degraded 300 0.797 0.347 0.692 868.8
molscribe chembl_test_novel_scaffold varied 300 0.947 0.760 0.978 763.0
molmini decimer_hdm hand_drawn 1000 0.925 0.037 0.237 70.0
molmini decimer_hdm hand_drawn 1000 0.911 0.057 0.307 47.3
molscribe decimer_hdm hand_drawn 1000 0.852 0.099 0.392 598.2

Metric definitions

  • valid: prediction parses as a molecule under RDKit.
  • exact: canonical SMILES of the prediction equals the reference. The headline OCSR metric.
  • exact/no-stereo: same comparison after stripping stereochemistry; the gap to exact isolates stereo errors.
  • formula: molecular formula matches, even when connectivity does not.
  • tanimoto: Morgan(r=2, 2048-bit) similarity, over parseable predictions only.
  • edit dist: token-level Levenshtein between canonical SMILES, normalized by reference length.
  • skipped: system could not run at all; excluded from every rate above.
  • ms/img and measured on: read together. Rows measured on different machines are not comparable on latency. MolMini is timed on the laptop it is meant to run on; MolScribe cannot run on macOS 26+ at all and is timed on Linux.

Notes

  • Rates exclude systems recorded as skipped.
  • MolMini latency is inference only; MolScribe latency includes amortized checkpoint load.
  • MolMini is measured on the laptop; MolScribe cannot run on macOS 26+ and is measured on Linux. See the 'measured on' column.
  • Checkpoint selection (best.pt vs last.pt) was decided on a frozen validation-split set, not on these test sets.