gemmi

Library:

• A breaking change in Op:

  Until now parse_triplet("h,k,l") was equivalent to parse_triplet("x,y,z"). This is not how it's handled in cctbx and in, for example Pointless. In the Pointless documentation of the REINDEX keyword there is such a note:

  Note that the real and reciprocal space operators correspond to mutually transposed matrices, eg "x-y,-y,-z" corresponds to "h,-h-k,-l".

  Gemmi Op was changed to store the notation kind that was used at creation, so now parse_triplet("h,k,l") .triplet() gives "h,k,l", not "x,y,z". New Op methods have been added: is_hkl(), as_hkl() and as_xyz() Apart from this, in case of hkl-Ops parse_triplet and triplet silently transpose the rotation matrix. See #359 for details. This change was extensively tested with Mtz::reindex() – against Pointless. It could happen that it causes problems in other scenarios.

• DDL2: added alternate (deposition) checks

• Mtz: added C++ Mtz::write_to_buffer() and Python Mtz.write_to_bytes() (used for streaming MTZ files from web servers) and C++ size_to_write() These new C++ functions are currently undocumented, not sure if they are useful. Changed Mtz::write_to_string() to clear the string instead of appending to it. Probably nobody expected the latter. Reorganized MTZ reading to avoid fseek() – so we can read a stream or gzipped file, without storing/unpacking it into memory buffer as we did before (i.e. reading huge gzipped MTZ files uses less memory)

• added UnitCell::find_nearest_pbc_images()

• CIF reading: added optional argument int check_level=1 (see docs for details)

• Form factors: support for custom form factors (in the form of sums of 5 Gaussians) – for a Cambridge project on chemistry-dependent form factors.

• to_mmcif: write _atom_site_anisotrop.pdbx_PDB_model_num if there are 2+ models

• to_pdb: split option use_linkr into use_linkr and use link_id

• the built-in residue list is now partly editable (example in docs)

• mmcif: workaround for a problem with reading 7pvv.cif (#369) occupancy and B_iso_or_equiv are now optional (because of #375)

• pdb: added option ignore_ter to PdbReadOptions

• started working on secondary structure determination – that's unfinished and unusable yet

• renamed interpolate_grid_of_aligned_model2() to interpolate_grid_around_model()

• added functions interpolate_points() and interpolate_grid_flexible() (#363), needed in Pandda2

Python:

• added python bindings to Ccp4Base::ccp4_header and to some functions related to gemmi::ChemComp

Program

• gemmi-validate: new check for monomer files, and new option --depo to check against PDB's deposition criteria (from DDL2)

Library

• reading mmcif: added reading of TLS information
• writing mmcif: added a few new items
• using Logger also in classes Ddl and CifToMtz (Logger is now used in all library functions that output warnings or messages)
• improved and documented mmCIF validation (with DDL2)
• added read_ccp4_header() for reading only map header when a map is a huge file
• added a few functions related to TLS (will be documented later)
• documented: working with XDS files, normalization of amplitudes (F->E)
• calculating merging statistics (R-merge, R-meas, R-pim, CC1/2) in various ways: Gemmi can calculate R-merge (and other R-*) in 3 different ways that are present in the literature and other programs; for CC1/2, the sigma-tau method is used
• internal refactoring of file reading
• misc bug fixes

Python

• functions Mtz.filtered() and XdsAscii.filtered()
• a number of other additions and a few small changes/fixes
• all cif-reading function consistently read gzipped files (previously, cif.read_file() and gemmi.read_small_structure() didn't)

Program

• gemmi fprime supports ranges of energies
• gemmi merge – added new options, most importantly --stats to print quality metrics
• gemmi convert – option --add-tls to convert "residual" B-factors (from Refmac) to full B-factors
• gemmi mask – solvent masking that takes into account alternative conformers and atom occupancy (experimental)

C++14 (or later) is required to build the library, C++17 (or later) to build Python bindings. Expect breaking changes, especially in Python bindings. The lists below are not complete, but should cover most of the changes.

Library

• Added unified logging of warnings/errors from various gemmi functions (class Logger)
• replaced string Model::name with int Model::num
• mmcif: better handling of null auth_comp_id
• fixes for mmJSON
• Removed deprecated functions:
  • UnitCell.fractionalization_matrix and orthogonalization_matrix – use frac.mat and orth.mat
  • count_hydrogen_sites() – use has_hydrogen() or count_atom_sites(gemmi.Selection('[H,D]')
  • Grid::resample_to() – use interpolate_grid()
• unified API of Grid interpolation functions. They now have parameter order that can be 0 (nearest value), 1 (linear interpolation), or 3 (cubic). In C++ there are also functions such as trilinear_interpolation() to ensure no overhead.
• to_pdb: write HET records
• Extended selection syntax with: [metals] and [nonmetals].
• Added function set_is_metal() intended for debatable metalloids
• improved interoperability with MMDB (a CCP4 library)
• MonLib: removed read_cif args
• mtz: fixed writing BATCH records
• hydrogen placement: fixes needed for new files with metals in CCP4 Monomer Library
• pdb: fixed reading TLS S tensor
• Structure metadata: expanded RefinementInfo

Python

• Python bindings migrated from pybind11 to nanobind.
  • Much lower runtime overhead, faster build times, better error diagnostics.
  • Built-in typing stubs.
  • Only Python 3.8+.
  • Sadly, no support for Buffer Protocol. It was replaced with NumPy __array__ methods. For NumPy, you can also use .array properties that were available also in the previous releases.
  • No implicit conversions from list to ndarray, and from bytes to string (let me know where it causes problems)
  • gemmi.ValueSigmaAsuData.value_array has now shape (N,2)
• Added pickling support for Structure, Model, Chain, Residue, Atom, cif.Document, cif.Block.
• Added function interpolate_position_array (#323).
• Python extension module is now installed into site-packages/gemmi/ (this change should be invisible to the user)

Program

• gemmi convert --sifts-num is now more customizable
• gemmi sf2map: added option --check (see docs)
• gemmi cif2mtz: add a rule to spec to convert pdbx_F_calc_with_solvent to F-model (+phase)
• gemmi xds2mtz: handles merged files from XSCALE
• gemmi mtz2cif and merge: recognize extension .ahkl as XDS file

This is primarily a bug-fix release. New Python bindings are not included yet.

Enhancements:

• New subcommand gemmi set for changing coordinates, B-factors and occupancies in coordinate files (mmCIF and PDB). Unlike other tools, it replaces numbers while leaving the rest of the file intact. An alternative to CCP4 PDBSET keywords: BFACTOR, OCCUPANCY, SHIFT, NOISE. Note that gemmi convert offers overlapping capabilities. For instance, gemmi convert --apply-symop=x+0.123,y,z shifts the coordinates similarly to gemmi set --shift='9.3 0 0' (the latter takes the shift in Angstroms).

• Improved anisotropic scaling of structure factors. More work is planned in this area.

Fixes:

• fixed reading of mmCIF files without _atom_site.auth_seq_id
• in Topology preparation: fixed a couple of bugs, peptide links are now assumed to be CIS for ω=0±60° (previously, ω=0±30°)
• fixed re-assignment of ATOM/HETATM record types (gemmi convert --assign-records)
• fixed gemmi convert --sifts-num for UniProt sequence numbers >5000

And various minor changes that are hard to describe concisely.

Library:

• SmallStructure: changed how the space group is read and accessed. Relying on H-M space group names alone was not always sufficient. The new mechanism uses the list of operations and Hall symbol in preference to the H-M symbol – the order is configurable.
• symmetry triplets: parse decimal fractions (small molecule files may use notation such as x+0.25 instead of x+1/4)
• tabulated space groups: a few more settings: B 1 2 1, B 1 21 1, F 1 m 1, F 1 d 1, F 1 2 1
• X-ray scattering coefficients: changed the default value of IT92::ignore_charge to true (i.e. charges are now ignored by default; before version 0.6.3 they were always ignored)
• cif::Table: added method ensure_loop() that converts tag-value pairs into a loop; might be needed before calling append_row()
• place_hydrogens(): fix for NH3-like configurations
• improved gemmi->mmdb conversion
• Grid: tweaked good_grid_size() to ensure that when creating a grid up to a certain d_min, all reflections up to d_min are in the grid (it matters when no oversampling is applied)
• DensityCalculator: deprecated function set_grid_cell_and_spacegroup(), use grid.setup_from()
• fixed TNT-compatible reciprocal space ASU calculation for non-standard settings
• infer_polymer_end(): complicate the heuristic even more, to detect files that have HETATM incorrectly used for standard residues in a polymer (such files were reported, they are either a result of mutating from non-standard residues, or a buggy program)
• added function assign_het_flags() to re-set ATOM/HETATM flags
• Model: added funtions calculate_b_iso_range() and calculate_b_aniso_range(); the first one can be used to detect if pLDDT is in the range 0-100 (like from AlphaFold) or 0-1 (like from ESMFold)
• writing mmCIF: write _entity_poly_seq.hetero
• added flag Entity::reflects_microhetero that shows if sequences were read from SEQRES (and don't account for point mutations) or from _entity_poly_seq; new function add_microhetero_to_sequences() changes the former to the latter

Program:

• gemmi sfcalc: added a few more options
• gemmi convert: added options --assign-records[=A|H], improved --sifts-num, adding microheterogeneities to _entity_poly_seq when converting from PDB
• gemmi cifdiff: added option -t for basic comparison of values for a single tag

Other:

• minimal WebAssembly port (C++ code compiled with emscripten) of Structure, as a proof-of-concept and for reading mmCIF files in UglyMol
• examples/to_rdkit.py: example of conversion of gemmi ChemComp to RDKit Mol

and a number of less important changes

Library:

• gemmi can now be built with zlib-ng, a faster fork of zlib (good for working with large, compressed files)
• experimental: binary serialization of Structure (contained objects, such as Model, Chain or UnitCell, can also be serialized separately)
• finalized handling of 5-character monomer names; uses the tilde-hetnam extension (ABCDE ↔ ~DE) for PDB files
• when atom names in the coordinate file match previous names (_chem_comp_atom.alt_atom_id) from the monomer library (the names in the CCD and therefore also in the ML change occasionally), print better diagnostic; added function MonLib::update_old_atom_names() to update the names in a Structure
• topology: fixed handling of two bonds between the same two residues
• options for handling mmCIF files with incorrect entities (modified add_entity_ids() when called with overwrite=true)
• added function Intensities::prepare_merged_mtz()
• a few bug fixes (for instance, in handling of negative residue numbers in the selection syntax)

Python bindings:

• generating type stubs - see #293
• python: cif.Loop.val() has been replaced with __getitem__/__setitem__
• fixed Mtz.Batch.ints and Mtz.Batch.floats

Program

• subcommand diff has been renamed to cifdiff
• subcommand prep has been renamed to crd
• validate: more options for checking monomer files
• gemmi-grep: added option --extended-regexp
• mtz2cif: added column names Iplus/Iminus (used by ccp4i2) to the default conversion spec

Note: this list is meant to show important changes only.

Library

• completely changed build system for Python module, from setuptools to scikit-build-core
• optimized electron density calculation: single-precision version is now about 2x faster and slightly less exact; some other grid-based calculations also got optimized in the process
• as part of the above optimizations, some of the grid computations require that the model is in the standard orientation (conventional axis directions); in other cases (which are very rare after the remediation of non-standard coordinate frames in the PDB) call standardize_crystal_frame()
• CIF output: more flexible formatting
• mmCIF writing: category _entity_poly is included by default, with pdbx_strand_id and pdbx_seq_one_letter_code
• minor changes in reading mmCIF coordinate files
• cif: added functions Loop::add_columns(), Loop::remove_column(), Column::erase()
• MRC map format: ORIGIN record is ignored (previously, if ORIGIN was non-zero, Ccp4::full_cell() returned false and some map properties were not set)
• new function Grid::symmetrize_avg()
• fixed bug in ReciprocalGrid::prepare_asu_data()
• added function read_pir_or_fasta() for reading sequences (previously it was undocumented and more limited)
• added function pdbx_one_letter_code() which returns a string like AA(MSE)H…, for _entity_poly.pdbx_seq_one_letter_code
• new functions expand_one_letter() and expand_one_letter_sequence() that take ResidueKind.AA/RNA/DNA as argument replaced expand_protein_one_letter*()
• adjusted weights in align_sequence_to_polymer()
• added function assign_best_sequences()
• PDB reading: added Structure::ter_status flag to indicate if TER records were: absent, present, clearly in wrong places
• experimental (not documented yet) new functions: Model::get_cra(), Model::get_parent_of()
• Topo::Bond stores a flag for bonds between different symmetry images
• ChemComp::Atom: store _chem_comp_atom.alt_atom_id as old_id, use it in new function update_old_atom_names()
• riding hydrogens: added H had wrong occupancy in special, rare cases
• added Vec3f – Vec3 with single-precision numbers
• minor API changes: Binner::setup() doesn't return anything, changed argument types of Scaling::scale_data(), align_sequences()

Program

• new tool gemmi-diff that compares categories and tags in two (mm)CIF files
• gemmi-align prints vertical list with option --verbose
• gemmi-residues has new options: -e, -sss, --chains
• gemmi-rmsz: added option --missing to print missing atoms
• gemmi-validate: more options for validating monomer files
• gemmi-h: more options
• gemmi-mtz: prints info about SYMM records

• new: normalization of amplitudes using so-called "Karle" approach, similar as in the CCP4 program ECALC
• added X-ray scattering coefficients for ions (previously, the charge of atom was ignored)
• pdb: reading CONECT records, and an option to also write them
• when reading pdb, if any chain has 2+ TER records, all TER records are ignored
• more configuration options for writing pdb files
• added functions Mtz::expand_to_p1() and Mtz::read_file_gz()
• cif::Block::find_value(tag) now returns also value from the corresponding loop if that loop has only one row
• changes in gemmi-validate related to validation with DDL2
• gemmi-sfcalc: added option --sigma-cutoff
• gemmi sf2map --mapmask: if the unit cells in coordinate file is different than in SF file, use only the latter
• improved transform_to_assembly(), expand_ncs() and rename_chain()
• cif2mtz: Mtz column for pdbx_DELPHWT has now label PHDELWT (#272)
• fixed ensure_asu(): phase-shift (for phases and H-L coefficients) was wrong
• fixed UnitCell::find_nearest_image() for non-crystals with NCS
• fixed DensityCalculator::requested_grid_spacing()
• changes and enhancements in add_chemcomp_to_block(), in solvent masking, in mtz2cif, and in several other places
• added python bindings to MtzToCif, cif::Ddl, PdbWriteOptions, changed how options for PDB writing are passed, more bindings for Mtz::Batch

• a number of fixes, mostly in topology preparation
• support for extended (longer) CCD and PDB codes that are about to be introduced by the PDB
• gemmi-convert: added option to rename a monomer
• a few changes and additions in cif2mtz, including:
  • anomalous data written as separate rows for F+ and F- is now converted as expected
  • _refln.F_squared_meas is now a synonym for F_squared_meas
• gemmi-grep: new option --only-tags
• gemmi-validate: a couple of new checks and options
• pdb and mmCIF: convert MODRES <-> _pdbx_struct_mod_residue
• cif.Block: blocks with no name (just data_) used to have the name set to "#", now it's " "

• changed how CISPEP is stored: previously, it was assumed that a link between two residues is either TRANS or CIS; if the residues have atoms with alternative conformations, both link types can be present at the same time
• riding hydrogens: previously, hydrogens had the same altlocs as the parent atom; now if the parent atom has a single conformation, but it has neighbors in multiple conformations, the hydrogens will be also added in multiple conformations
• major changes in NearestNeighbor: now it's possible to search atoms not only in the first nearest cells, but in any number of nearest cells; find_nearest_atom() was changed to find, by default, a nearest atom within any distance
• changes in Mtz.reindex(), primarily to fix determination of the new space group
• gemmi-convert: added option --all-auth to write _atom_site.auth_atom_id and auth_comp_id, which are skipped by default (because they are always the same as label_…_id)
• added more options to gemmi-rmsz, gemmi-xds2mtz, gemmi-cif2mtz
• fix a recent regression in check_polymer_type(): RNA was returned instead of DNA
• improved heuristic of detecting where the polymer ends (if TER record is missing)
• selection syntax: fix parsing a single sequence id such as "A/208" (it was parsed as A/208-/)
• removed SMat33::calculate_eigenvector() – use eigen_decomposition() instead; SoftwareItem::pdbx_ordinal; NeighborSearch::Mark::x,y,z (use ::pos)
• more code was moved from headers to src/*.cpp

• C++ library is no longer header-only, several function were moved from headers to src/ to make compilation faster
• major changes in cmake build, requiring now cmake 3.15+
• improvements in calculating riding hydrogen positions
• changed again the scheme of automatically assigned subchain names (A-p -> Axp, because PDB software can't handle non-alphanumeric characters there)
• a function for calculating polarization correction for XDS INTEGRATE.HKL
• improvements in xds2mtz, converting more data and filling more records in MTZ batch headers
• added SpaceGroup::change_of_hand_op()
• various bug fixes and small improvements

• gemmi program has new subcommand xds2mtz that converts from XDS_ASCII to multirecord MTZ
• subcommand gemmi-residues has new option -s (--short) for shorter overview of model chains (can be used twice)
• cif2mtz: more flexible spec for converting symbols to numbers
• preparation of Refmac intermediate files – it can be used to substitute a part of Refmac
• MonLib and Topo: a number of changes related to reading a monomer library and in prepare_topology()
• changed the scheme of automatically assigned subchain names (Apoly -> A-p)
• read_structure(): added optional arg save_doc that stores cif.Document if the read file is mmCIF or mmJSON
• reading PDB files: more metadata is read by default
• writing mmCIF files: _atom_site.group_PDB is written by default
• support for mmCIF extension _atom_site.ccp4_deuterium_fraction
• added function copy_from_mmdb(mmdb::Manager* manager) -> Structure
• improved check_polymer_type()
• Grid::set_size_from_spacing() with different rounding modes
• rename src/ to prog/; in the next version the library won't be fully header-only, cpp files will go into src/

• new functions for working with Structure: assign_serial_numbers(), assign_cis_flags(), has_hydrogen()
• enhanced transform_to_assembly()
• functions count_atom_sites() and count_occupancies() now take Selection as an optional arg
• deprecated count_hydrogen_sites(): can be replaced with has_hydrogen() or count_atom_sites(Selection("[H,D]"))
• selection syntax extended with ";polymer" and ";solvent" (it can also be "!polymer,solvent")
• improved preparation of the intermediate file (crd) for Refmac
• gemmi program: several options were added to subprograms
• Ccp4.setup() now returns void (previously – number)
• python: cif.Block.find_pair() and Item.pair were changed to return tuple (previously – list)

• calculating ASU brick for given space group settings
• add reciprocal ASU definitions used by TNT
• add option style to Op.triplet()
• add cif.Block.set_pairs() – it adds name-value pairs to a given category
• various bug fixes, as usual
• Python 2.7 is no longer listed as supported

• determining lattice symmetry and (psuedo-)merohedral twinning laws
• added GroupOps::derive_symmorphic() and GroupOps::add_inversion()
• getting change-of-basis in the Niggli reduction
• added transform_to_assembly() – it does a bit more than make_assembly()
• removed Op::negated()
• python: add bindings to UnitCell.orth and UnitCell.frac
• additional options in gemmi-mtz2cif and gemmi-cif2mtz
• bug fixes (in Ccp4::set_extent(), merge_atoms_in_expanded_model(), status column in mtz2cif)

• new sub-program under development: gemmi-prep
• parse pdb AUTHOR record and corresponding mmCIF category
• new function flood_fill_above()
• mtz2cif: rhombohedral groups in hexagonal settings start with H now
• renamed MonLib::find_link() to get_link(), modified MonLib::match_link()
• CIF files can be written with left-aligned columns
• bug fixes and minor changes (in particular in class Topo)

• API change: changed arguments for Ccp4.setup()
• added Binner class for analyzing reflections in resolution bins
• added support for neutron scattering in gemmi-sfcalc and the library
• new functions: seitz_to_op, normalize_grid, Grid.get_subarray, Grid.set_subarray
• minor improvements in programs mtz2cif, cif2mtz, map
• fix: CIF tags are now case insensitive
• changes in preparing topology accounting for changes in Refmac and CCP4 monomer library

• various bug fixes and small improvements, many of them for the MTZ -> mmCIF conversion
• new and documented functions: cif.Table.move_row(), ResidueInfo.fasta_code(), GroupOps.add_missing_elements(), NeighborSearch.add_chain()
• added customizable form factors for unknown element (X) #167
• improved reading of unusual PDB files #169
• calculate_superposition(): added option to use backbone only
• calculating RMSD without superposition is a separate function now: calculate_current_rmsd()
• simplified MaskedGrid, Grid.asu() renamed to masked_asu()
• renamed Intensities.Type to DataType
• a few experimental functions (not documented and not sufficiently tested yet):
  • read_first_block_gz() – read only first block of CIF without reading the whole file
  • Mtz.ensure_asu() – changes Miller indices together with phase shifts and anomalous data swapping
  • interpolate_grid_of_aligned_model() – for grid interpolation based on model alignment (currently global alignment only, local alignment such as in PanDDA is not implemented yet)
  • check_data_type_under_symmetry() – for checking if mmCIF refln category contains mean, anomalous or unmerged data

• various bug fixes
• support for huge MTZ files (>2GB) using the format extension added in CCP4 8.0 – implemented by Claus
• better handling of NaNs in maps
• option to detect the format of a coordinate file (format=CoorFormat.Detect)
• major changes in class Topo (which is undocumented yet)
• store _atom_site.label_entity_id from mmCIF as Residue::entity_id
• two more tabulated space group settings: C 4 2 2 and C 4 2 21
• new functions Mtz.copy_column() and Mtz.replace_column(), SmallStructure.make_cif_block()
• and a number of minor additions including: Grid.clone(), Grid.change_values(), Model.remove_*(), transform_pos_and_adp(), MonLib.mon_lib_list, SmallStructure.wavelength, UnitCell.approx(), SMat33.multiply(), Structure.input_format
• new options in the command-line program: gemmi mtz --histogram, gemmi cif2mtz --add

• bug fixes
• Selling-Delaunay reduction
• gemmi program: enhancements in mtz2cif, cif2mtz and merge
• SymImage renamed to NearestImage, its properties has also changed
• calculate_superposition(): optional outlier rejection (inspired by PyMOL)

• centred to primitive lattce transformation
• Niggli reduction and Buerger reduction for lattice basis
• added UnitCell::is_compatible_with_spacegroup()
• Topology-related improvements (Keitaro)
• functions SMat33::elements() and change_basis() were replaced – each was replaced with two new functions
• more options in gemmi-merge
• more Python bindings
• various fixes, in particular in validation in gemmi mtz2cif --depo.

• reindexing MTZ files,
• tricubic map interpolation,
• extended selection syntax,
• merge_atoms_in_expanded_model() – sort out atoms on special positions after expanding NCS or making bioassembly,
• IT92::normalize() – makes exactly the same scattering coefficients that Refmac uses,
• Mtz::remove_column(),
• understanding triplets with x/N ("h/2+k/2, -h/2+k/2, l"),
• better inference of elements in a PDB file with missing element field,
• Topo::Force was renamed to Topo::Rule,
• various other additions and, most importantly, bug fixes.

Python porting note: CIF parsing may raise ValueError; opening a file may rise IOError. Previously it was RuntimeError.