This pysam release wraps htslib/samtools/bcftools version 1.23.1 (PR #1343, PR #1372, and PR #1396) and now requires Cython 3.

It has been tested with Python versions 3.8 through 3.14, and wheels are available via PyPI for all of those Python versions. Wheels are built for macOS and Linux (manylinux_2_28 and musllinux_1_2) on both ARM and x86-64.

This release is the first to wrap htslib 1.22 or later, which contains significant CRAM-related operational changes inherited by pysam:

• The default output CRAM version is now 3.1 rather than 3.0. The version can be explicitly specified by using AlignmentFile(…, "wc", format_options=["version=3.0"]).

• CRAM reference data is no longer fetched from EBI's server by default. Use the REF_CACHE and REF_PATH environment variables if necessary to set up an alternative source of CRAM reference data.

The final pysam release that supported Python 3.6 and 3.7 was v0.23.0. This release is expected to be the last release supporting Python 3.8.

Bugs fixed:

• Removed AlignedSegment.modified_bases's limitation of only five modification types in MM/ML fields. (#1348, reported by @lpryszcz)

• Fixed AlignedSegment.modified_bases crash when given a degenerate MM field that modifies no bases at all. (#1355, reported by @JMencius)

• Fixed type hints for AlignmentFile and VariantFile constructor filename and mode parameters. (#1405, reported by @msto)

• Fixed coverage functionality for Python 3.13 and later. Pysam's Cython code is no longer built with Python profiling enabled by default. Profiling can be re-enabled by using the PYSAM_PROFILE environment variable when building pysam from source. (#1350, reported by @fedarko)

• Fixed capturing pysam.samtools.calmd() output, which fixed tests on 32-bit i386. Also suppressed TestBinaryCompatibility test cases on platforms where we have not calculated the expected sizes. (#1367, reported by @emollier for Debian)

• Adjusted HTSFile.add_hts_options() for Python 3: it now expects a list of str as documented, rather than needing bytes. This method's format_options argument is no longer optional. Similarly AlignmentFile(format_options=…) now accepts a list of str as documented.

New functionality:

• Updated pysam/libchtslib.pxd declarations to the HTSlib 1.23.1 API. This is the first comprehensive update that these Cython HTSlib API declarations have received. (PR #1362)

• Optimized pysam.array_to_qualitystring() using Cython memoryviews, resulting in an approximately 70×–100× speed improvement in this function. (PR #1363, thanks to @jchorl)

Documentation improvements:

• The numerous internal classes representing VariantHeader and VariantRecord fields are now documented. (#407 et al; PR #1358, PR #1359, PR #1361, PR #1394, thanks to @TimD1)

• Clarified AlignedSegment.query_alignment_length et al behaviour for unmapped reads. (#1319, reported by @schorlton-bugseq)

This is a bugfix release, still wrapping htslib/samtools/bcftools 1.21.

It has been tested with Python versions 3.8 through 3.13, and wheels are available via PyPI for all of those Python versions. Wheels are built for macOS and Linux (manylinux_2_28 and musllinux_1_2) on both ARM and x86-64.

The final pysam release that supported Python 3.6 and 3.7 was v0.23.0.

Bugs fixed:

• Worked around Cython 3.1.2 bug whereby a new Cython-internal function incorrectly triggered the pysam build system's duplicate symbol check.

This is a bugfix release, still wrapping htslib/samtools/bcftools 1.21.

It has been tested with Python versions 3.8 through 3.13, and wheels are available via PyPI for all of those Python versions. Wheels are built for macOS and Linux (manylinux_2_28 and musllinux_1_2) on both ARM and x86-64.

The final pysam release that supported Python 3.6 and 3.7 was v0.23.0.

Bugs fixed:

• Pysam 0.23.1 inadvertently broke binary compatibility for Cython projects by changing the size of AlignedSegment. That class has been restored to its previous size, so v0.23.2 restores binary compatibility with Cython projects previously compiled against v0.23.0 and earlier. (This bug does not affect pure Python projects using pysam.)

• Improved I/O exception messages in AlignmentFile and VariantFile, and ensured that error codes returned by all HTSlib function invocations result in an exception being raised.

This is a bugfix release, still wrapping htslib/samtools/bcftools 1.21.

It has been tested with Python versions 3.8 through 3.13, and wheels are available via PyPI for all of those Python versions. Wheels are built for macOS and Linux (manylinux_2_28 and musllinux_1_2) on both ARM and x86-64.

This release inadvertently broke binary compatibility for Cython projects. This should not affect Python projects using pysam. However Cython projects should update to release 0.23.2.

The final pysam release that supported Python 3.6 and 3.7 was v0.23.0.

Bugs fixed:

• Worked around Cython 3.1.0 behaviour change so that pysam.CMATCH etc remain as synonyms for pysam.CIGAR_OPS.CMATCH etc. Note that a future release will remove these synonyms to comply with Cython's new convention. (#1339, PR #1340, reported by @Adamtaranto, @andyb3, et al)

• The Python interpreter no longer exits abruptly on x86-64 Linux when pysam.bcftools.*() commands terminate with error messages. (#1333, #1335, reported by @3ng7n33r and @Ben-Habermeyer)

• Fixed AlignedSegment.get_aligned_pairs() type hint, which now has overloaded hints reflecting the different tuples returned. (PR #1342, thanks to @gshiba)

• Corrected “invalid type for record” exception message. (PR #1334, thanks to @gtamazian)

• Fixed TabixIterator error handling. (#1328, reported by @Fan-iX)

• Fixed bugs in the caching of AlignedSegment.query_sequence and AlignedSegment.query_qualities values.

• Pysam once again builds on CentOS 7, as it now works around limitations in that obsolete platform's obsolete GCC 4.8.x system compiler. (#1327, reported by @mfasold)

New functionality:

• New AlignedSegment.query_qualities_str property enables accessing the QUAL field as an ASCII-encoded base quality string. The existing AlignedSegment.query_qualities property can now take an array or such a string when setting the value. (PR #1324, PR #1341, requested by @nh13)

• Improved AlignedSegment.cigarstring performance. (PR #1295, thanks to @limwz01)

• Wheels are now built for musllinux in addition to manylinux and macOS. (Requested by @nh13)

This pysam release wraps htslib/samtools/bcftools 1.21 (PR #1310).

It has been tested with Python versions 3.6 through 3.13, and wheels are available via PyPI for all of those Python versions. Python versions 3.6 through 3.8 are end-of-life; particularly if you use pysam with one of these versions, please vote in the version survey at issue #1230.

This is expected to be the last release supporting Python 3.6 and 3.7.

Bugs fixed:

• Fix pysam.samtools.command(save_stdout=filename) redirection of output to a file. (#677, reported by @haowenz and analysed by @yhoogstrate)

• Reinstate HTTPS/S3/GCS support in pre-built Linux wheels when used on Debian and Ubuntu: these wheels now work around the discrepancy between Red Hat and Debian CA Certificate file locations. (#1257, #1268, reported by @daisieh and @litaifang)

• Calling VariantHeader.new_record() repeatedly with the same samples object now sets GT correctly every time. (#1308, reported by @awgymer)

• Correct the exception produced when AlignedSegment.set_tags() is used with an invalid value type. (#1233, PR #1235, reported by @weishwu and @marcus1487)

• Numerous type hinting corrections. (#1298, #1316, PR #1296, PR #1306, PR #1313, PR #1315, thanks to @vepain, @awgymer, @mshunjan, and @msto)

• The undocumented pysam.samtools.import_() alias for invoking samtools import has been removed; use pysam.samtools.fqimport() instead.

• Corrections to several test data files to account for HTSlib 1.20 and later's improved validity checking. (#1291, reported by @SoapGentoo; etc)

New functionality:

• AlignedSegment.get_aligned_pairs() now optionally returns the associated CIGAR operator in each position tuple. (#1292, PR #1294, thanks to @LaraFuhrmann and @DrYak)

• New AlignmentFile.flush() and VariantFile.flush() methods for flushing buffered output to streams. (#1299, requested by @blex-max)

• Improved str() for AlignedSegment now displays reference sequence names when they are available and uses #N only when necessary. (#1318, requested by @creaturemoon)

• Implement repr() for AlignedSegment so that it displays the most useful fields. (PR #1267, thanks to @marcelm)

• Pysam's tests can now be run in parallel. (#1284, reported by @yurivict)

Documentation improvements:

• More complete documentation of invoking SAMtools and BCFtools subcommands via Pysam. (#1096, #1241, PR #1261, PR #1275, #1323, et al, reported by @sbslee, @robertwhbaldwin, @mbhall88, @indraniel, and @shokrofont)

• Use the Pysam names for CIGAR operators rather than internal C names. (#1255, reported by @notestaff)

• Mention AlignmentFile.fetch("*") in the FAQ entry on fetching unmapped reads. (#424, reported by @bw2)

This bug fix release wraps htslib/samtools/bcftools 1.18.

Bugs fixed:

• Preserve all header field tags defined in the SAM specification (notably TP) in AlignmentHeader.from_dict() and AlignmentHeader.to_dict() (#1237, PR #1238, thanks to @tfenne and @nh13)

• Adjust HTSlib's Makefile so that make distclean no longer tries to rebuild the htscodecs configury (PR #1247, reported by @nsoranzo)

• Reinstate S3 support in pre-built Linux wheels: support for this protocol was inadvertently omitted from the pre-built 0.22.0 wheels on Linux (#1249, #1277, etc varying circumstances; likely it is this that was reported by @Mathew-B-SDGS, @BenjaminSargsyan, et al)

• Add missing AlignedSegment.is_mapped etc properties to type stubs (PR #1273, thanks to @msto)

• Fix off-by-one NamedTupleProxy, asBed, etc array bounds check (#1279, reported by @dbolser)

• Make pysam's klib headers compatible with C++ (reported by @martin-g)

This pysam release wraps htslib/samtools/bcftools 1.18 (PR #1208).

It has been tested with Python versions 3.6 through 3.12, and wheels are available via PyPI for all of those Python versions. Python versions 3.6 and 3.7 are end-of-life; particularly if you use pysam with either of these versions, please vote in the version survey at #1230.

The final pysam release that supported Python 2.7 was v0.20.0.

Bugs fixed:

• Remove Cython from runtime dependencies (PR #1186, thanks to @nsoranzo, also reported by @aryarm in PR #1194)

• Miscellaneous dependency improvements (PR #1216, #1217, PR #1218, PR #1219, thanks to @althonos and @ahvigil)

• Suppress spurious “Could not retrieve index file” message when opening an AlignmentFile (#939, #1214, reported by @cytham and @sroener)

• Propagate parsing errors encountered in AlignedSegment.fromstring() (#1196, reported by @dvklopfenstein)

• Accept invalid MD:A tagged fields produced by HTSeq instead of crashing in get_aligned_pairs(with_seq=True) (#1226, reported by @isaacvock)

• Fix multiarch macOS CI builds by removing brewed liblzma (#1205, reported by @tedil)

• Fix VariantRecordSample.alleles type hint (#1179, reported by @SoapGentoo)

New functionality:

• Add optional HTSFile.seek(..., whence) parameter and clarify which functions use libc.SEEK_SET vs io.SEEK_SET (#1185, requested by @ylluX)

• File handling improvements in samtools & bcftools commands (should improve #1193 and #1195, reported by @biermanr and @schorlton)

• Improve FastxFile performance (PR #1227, thanks to @kloetzl and @valentynbez)

• Improve the accuracy of type hints for AlignmentFile iteration (#1184, PR #1189, reported by @PikalaxALT)

Documentation improvements:

• Clarify that AlignedSegment.get_aligned_pairs() results are 0-based (#1180, reported by @semenko)

• Clarify AlignedSegment.get_reference_positions() documentation (#836, #838, reported by @weir12 and @NickSto)

• Clarify that installation via pip usually uses a wheel, and that configuring the build via $HTSLIB_CONFIGURE_OPTIONS etc only applies when installing from an sdist (#1086, reported by @aiqc)

A message from pysam's founder, @AndreasHeger:

As many of you will have noticed, John Marshall has been effectively maintaining pysam and supporting users over the last few years. I, Andreas, am very grateful for the countless hours he has contributed. Unfortunately, I will not be able to contribute much in the near and intermediate future. To keep pysam going, John has kindly agreed to continue maintaining and supporting pysam as the principal developer of pysam. I am very happy to know that pysam is in good hands and want to thank again John and the wider pysam community for their suggestions, bug reports, code contributions and general support.

Thank you Andreas for all your work over the years and the solid foundations that pysam enjoys and the useful functionality it provides.

Release 0.21.0

This release wraps htslib/samtools/bcftools version 1.17.

Pysam is now compatible with Python 3.11. We have removed python 2.x
support. Pysam is tested with python versions 3.6 to 3.11.

• [#1175] VariantHeader.new_record: set start/stop before alleles
• [#1173] Add multiple build improvements in htscodecs on multi-arch macOS
• [#1148] Ignore CIGAR-less reads in find_introns.
• [#1172] Add new samtools cram-size and samtools reset commands
• [#1169] Fix CRAM index-related crash when using the musl C standard library.
• [#1168] Add a minimal pyproject.toml for PEP517.
• [#1158] Fix type hints and add FastqProxy type hints.
• [#1147] Py3.11 compatibility, get shared object suffix from EXT_SUFFIX.
• [#1143] Add mypy symbols for samtools and bcftools.
• [#1155] Fix pysam.index() when using recent samtools index options.
• [#1151] Test suite py3.11 compatibility, work around failing test case.
• [#1149] MacOS universal build compatibility.
• [#1146] Fix build when CFLAGS/etc environment variables are set.

Big thanks to @jmarshall who again has done most of the heavy lifting for this release.

29 October 2022

This release wraps htslib/bcftools version 1.16 and samtools version 1.16.1.

• [#1113] Full compatibility with setuptools v62.1.0's build directory name changes

• [#1121] Build-time symbol check portability improved

• [#1122] Fix setting sample genotype using .alleles property

• [#1128] Fix test suite failure when using a libdeflate-enabled samtools

Many additional type hints have been provided by the community, thanks!

27 May 2022

This release wraps htslib/samtools/bcftools version 1.15.1.

• [#1104] add an add_samples() method to quickly add multiple samples to VCF.

30 March 2022

This release wraps htslib/samtools/bcftools version 1.15.

• [#1085] Improve getopt()/getopt_long() resetting when running samtools/bcftools commands

• [#1078] Support BAM_CPAD in get_aligned_pairs

• [#1063] Run flake8 and fix some linting issues

• [#1088] Add AlignedSegment is_mapped/mate_is_mapped/is_forward/mate_is_forward properties

• Write an absent AlignedSegment.qual as all-bytes-0xff

• Fix BGZFile.read() behaviour near or at EOF

• First API for the htslib modified bases interface

17 November 2021

This release wraps htslib/samtools/bcftools version 1.14.

• [#1048] and [#1060], clarify documentation of index statistics with CRAM files

• Prevent "retval may be used uninitialised" warning.

• Add new "samples" subcommand to pysam/samtools.py

• Introduce TupleProxyIterator iterator object class

30 September 2021

This release wraps htslib/samtools/bcftools version 1.13. Corresponding to new samtools commands, pysam.samtools now has additional functions ampliconclip, ampliconstats, fqimport, and version.

Bugs fixed:

• [#447] The maximum QNAME length is fully restored to 254

• [#506, #958, #1000] Don't crash the Python interpreter on pysam.bcftools.*() errors

• [#603] count_coverage: ignore reads that have no SEQ field

• [#928] Fix pysam.bcftools.mpileup() segmentation fault

• [#983] Add win32/*.[ch] to MANIFEST.in

• [#994] Raise exception in get_tid() if header could not be parsed

• [#995] Choose TBI/CSI in tabix_index() via both min_shift and csi

• [#996] AlignmentFile.fetch() now works with large chromosomes longer than 229 bases

• [#1019] Fix Sphinx documentation generation by avoiding Python 2 ur'string' syntax

• [#1035] Improved handling of file iteration errors

• [#1038] tabix_index() no longer leaks file descriptors

• [#1040] print(aligned_segment) now prints the correct TLEN value (it also now prints RNAME/RNEXT more clearly and prints POS/PNEXT 1-based)

• setup.py longer uses setup(use_2to3) for compatibility with setuptools >= v58.0.0

New facilities:

• [PR #963] Additional VCF classes are exposed to pysam programmers

• [#998, PR #1001] Add get/set_encoding_error_handler() to control UTF-8 conversion

• [PR #1012] Running python setup.py sdist now automatically runs cythonize

• Running tests with pytest now automatically runs make to generate test data

Documentation improvements:

• [#726] Clarify get_forward_sequence/get_forward_qualities documentation

• [#865] Improved example

• [#968] get_index_statstics parameters

• [#986] Clarify VariantFile.fetch start/stop region parameters are 0-based and half-open.

• [#990] Corrected PileupColumn.get_query_sequences documentation

• [#999] Fix documentation for AlignmentFile.get_reference_length()

• [#1002] Document the default min_base_quality for pileup()

8 June 2020

This release wraps htslib/bcftools version 1.10.2 and samtools version 1.10. The following bugs reported against pysam are fixed due to this:

• [#447] Writing out QNAME longer than 251 characters corrupts BAM

• [#640, #734, #843] Setting VariantRecord pos or stop raises error

• [#738, #919] FastxFile truncates concatenated plain gzip compressed files

Additional bugfixes:

• [#840] Pileup doesn't work on python3 when index_filename is used

• [#886] FastqProxy raises ValueError when instantiated from python

• [#904] VariantFile.fetch() throws ValueError on files with no records

• [#909] Fix incorrect quoting in VariantFile contig records

• [#915, #916] Implement pileup() for unindexed files and/or SAM files

Backwards incompatible changes:

• The samtools import command was removed in samtools 1.10, so pysam no longer exports a samimport function. Use pysam.view() instead.

Bugfix release. Principal reason for release is to update cython version in order to fix pip install pysam with python 3.8.

• [#879] Fix add_meta function in libcbcf.pyx, so meta-information lines in header added with this function have double-quoting rules in accordance to rules specified in VCF4.2 and VCF4.3 specifications

• [#863] Force arg to bytes to support non-ASCII encoding

• [#875] Bump minimum Cython version

• [#868] Prevent segfault on Python 2.7 AlignedSegment.compare(other=None)

• [#867] Fix wheel building on TravisCI

• [#863] Force arg to bytes to support non-ASCII encoding

• [#799] disambiguate interpretation of bcf_read return code

• [#841] Fix silent truncation of FASTQ with bad q strings

• [#846] Prevent segmentation fault on ID, when handling malformed records

• [#829] Run configure with the correct CC/CFLAGS/LDFLAGS env vars

Bugfix release.

• [#824] allow reading of UTF-8 encoded text in VCF/BCF files.

• [#780] close all filehandles before opening new ones in pysam_dispatch

• [#773] do not cache VariantRecord.id to avoid memory leak

• [#781] default of multiple_iterators=True is changed to False for CRAM files.

• [#825] fix collections.abc import

• [#825] use bcf_hdr_format instead of bcf_hdr_fmt_text, fix memcpy bug when setting FORMAT fields.

• [#804] Use HTSlib's kstring_t, which reallocates and enlarges its memory as needed, rather than a fixed-size char buffer.

• [#814] Build wheels and upload them to PyPI

• [#755] Allow passing flags and arguments to index methods

• [#763] Strip 0 in header check

• [#761] Test Tabix index contents, not the compression

Bugfix release.

• [#746] catch pileup itorator out-of-scope segfaults

• [#747] fix faixd fetch with region

• [#748] increase max_pos to (1<<31)-1

• [#645] Add missing macOS stub files in MANIFEST.in, @SoapZA

• [#737] Fix bug in get_aligned_pairs, @bkohrn

Bugfix release.

• [#716] raise ValueError if tid is out of range when writing

• [#697] release version using cython 0.28.5 for python 3.7 compatibility

This release wraps htslib/samtools/bcftools version 1.9.

• [#673] permit dash in chromosome name of region string

• [#656] Support text when opening a SAM file for writing

• [#658] return None in get_forward_sequence if sequence not in record

• [#683] allow lower case bases in MD tags

• Ensure that = and X CIGAR ops are treated the same as M

This is mostly a bugfix release, though bcftools has now also been upgraded to 1.7.0.

• [#621] Add a warning to count_coverage when an alignment has an empty QUAL field

• [#635] Speed-up of AlignedSegment.find_intro()

• treat border case of all bases in pileup column below quality score

• [#634] Fix access to pileup reference_sequence

Bugfix release to solve compilation issue due to missinge bcftools/config.h file.

Bugfix release to address two issues:

• Changes in 0.11.2.1 broke the GTF/GFF3 parser. Corrected and more tests have been added.

• [#479] Correct VariantRecord edge cases described in issue

Release to fix release tar-ball containing 0.11.1 pre-compiled C-files.

This release wraps htslib/samtools/bcfools versions 1.4.1 in response to a security fix in these libraries. Additionally the following issues have been fixed:

• [#452] add GFF3 support for tabix parsers

• [#461] Multiple fixes related to VariantRecordInfo and handling of INFO/END

• [#447] limit query name to 251 characters (only partially addresses issue)

VariantFile and related object fixes

• Restore VariantFile.__dealloc__

• Correct handling of bcf_str_missing in bcf_array_to_object and bcf_object_to_array

• Added update() and pop() methods to some dict-like proxy objects

• scalar INFO entries could not be set again after being deleted

• VariantRecordInfo.__delitem__ now allows unset flags to be deleted without raising a KeyError

• Multiple other fixes for VariantRecordInfo methods

• INFO/END is now accessible only via VariantRecord.stop and VariantRecord.rlen. Even if present behind the scenes, it is no longer accessible via VariantRecordInfo.

• Add argument to issue a warning instead of an exception if input appears to be truncated

Other features and fixes:

• Make AlignmentFile __dealloc__ and close more stringent

• Add argument AlignmentFile to issue a warning instead of an exception if input appears to be truncated

Bugfix release

• [#440] add deprecated 'always' option to infer_query_length for backwards compatibility.

This release implements further functionality in the VariantFile API and includes several bugfixes:

• treat special case -c option in samtools view outputs to stdout even if -o given, fixes #315

• permit reading BAM files with CSI index, closes #370

• raise Error if query name exceeds maximum length, fixes #373

• new method to compute hash value for AlignedSegment

• AlignmentFile, VariantFile and TabixFile all inherit from HTSFile

• Avoid segfault by detecting out of range reference_id and next_reference in AlignedSegment.tostring

• Issue #355: Implement streams using file descriptors for VariantFile

• upgrade to htslib 1.3.2

• fix compilation with musl libc

• Issue #316, #360: Rename all Cython modules to have lib as a prefix

• Issue #332, hardclipped bases in cigar included by pysam.AlignedSegment.infer_query_length()

• Added support for Python 3.6 filename encoding protocol

• Issue #371, fix incorrect parsing of scalar INFO and FORMAT fields in VariantRecord

• Issue #331, fix failure in VariantFile.reset() method

• Issue #314, add VariantHeader.new_record(), VariantFile.new_record() and VariantRecord.copy() methods to create new VariantRecord objects

• Added VariantRecordFilter.add() method to allow setting new VariantRecord filters

• Preliminary (potentially unsafe) support for removing and altering header metadata

• Many minor fixes and improvements to VariantFile and related objects

Please note that all internal cython extensions now have a lib prefix to facilitate linking against pysam extension modules. Any user cython extensions using cimport to import pysam definitions will need changes, for example:

cimport pysam.csamtools

will become:

cimport pysam.libcsamtools

This is a bugfix release addressing some installation problems in pysam 0.9.0, in particular:

• patch included htslib to work with older libcurl versions, fixes #262.

• do not require cython for python 3 install, fixes #260

• FastaFile does not accept filepath_index any more, see #270

• add AlignedSegment.get_cigar_stats method.

• py3 bugfix in VariantFile.subset_samples, fixes #272

• add missing sysconfig import, fixes #278

• do not redirect stdout, but instead write to a separately created file. This should resolve issues when pysam is used in notebooks or other environments that redirect stdout.

• wrap htslib-1.3.1, samtools-1.3.1 and bcftools-1.3.1

• use bgzf throughout instead of gzip

• allow specifying a fasta reference for CRAM file when opening for both read and write, fixes #280

Overview

The 0.9.0 release upgrades htslib to htslib 1.3 and numerous other enhancements and bugfixes. See below for a detailed list.

The Htslib 1.3 release comes with additional capabilities for remote file access which depend on the presence of optional system libraries. As a consequence, the installation script setup.py has become more complex. For an overview, see installation. We have tested installation on linux and OS X, but could not capture all variations. It is possible that a 0.9.1 release might follow soon addressing installation issues.

The ~.pysam.VariantFile class provides access to vcf and bcf formatted files. The class is certainly usable and interface is reaching completion, but the API and the functionality is subject to change.

Detailed release notes

• upgrade to htslib 1.3

• python 3 compatibility tested throughout.

• added a first set of bcftools commands in the pysam.bcftools submodule.

• samtools commands are now in the pysam.samtools module. For backwards compatibility they are still imported into the pysam namespace.

• samtools/bcftools return stdout as a single (byte) string. As output can be binary (VCF.gz, BAM) this is necessary to ensure py2/py3 compatibility. To replicate the previous behaviour in py2.7, use:

  pysam.samtools.view(self.filename).splitlines(True)

• get_tags() returns the tag type as a character, not an integer (#214)

• TabixFile now raises ValueError on indices created by tabix <1.0 (#206)

• improve OSX installation and develop mode

• FastxIterator now handles empty sequences (#204)

• TabixFile.isremote is not TabixFile.is_remote in line with AlignmentFile

• AlignmentFile.count() has extra optional argument read_callback

• setup.py has been changed to:

  • install a single builtin htslib library. Previously, each pysam module contained its own version. This reduces compilation time and code bloat.

  • run configure for the builtin htslib library in order to detect optional libraries such as libcurl. Configure behaviour can be controlled by setting the environment variable HTSLIB_CONFIGURE_OPTIONS.

• get_reference_sequence() now returns the reference sequence and not something looking like it. This bug had effects on get_aligned_pairs(with_seq=True), see #225. If you have relied on on get_aligned_pairs(with_seq=True) in pysam-0.8.4, please check your results.

• improved autodetection of file formats in AlignmentFile and VariantFile.

This release contains numerous bugfixes and a first implementation of a pythonic interface to VCF/BCF files. Note that this code is still incomplete and preliminary, but does offer a nearly complete immutable Pythonic interface to VCF/BCF metadata and data with reading and writing capability.

Potential issues when upgrading from v0.8.3:

• binary tags are now returned as python arrays

• renamed several methods for pep8 compatibility, old names still retained for backwards compatibility, but should be considered deprecated.

  • gettid() is now get_tid()

  • getrname() is now get_reference_name()

  • parseRegion() is now parse_region()

• some methods have changed for pep8 compatibility without the old names being present:

  • fromQualityString() is now qualitystring_to_array()

  • toQualityString() is now qualities_to_qualitystring()

• faidx now returns strings and not binary strings in py3.

• The cython components have been broken up into smaller files with more specific content. This will affect users using the cython interfaces.

Edited list of commit log changes:

• fixes AlignmentFile.check_index to return True

• add RG/PM header tag - closes #179

• add with_seq option to get_aligned_pairs

• use char * inside reconsituteReferenceSequence

• add soft clipping for get_reference_sequence

• add get_reference_sequence

• queryEnd now computes length from cigar string if no sequence present, closes #176

• tolerate missing space at end of gtf files, closes #162

• do not raise Error when receiving output on stderr

• add docu about fetching without index, closes #170

• FastaFile and FastxFile now return strings in python3, closes #173

• py3 compat: relative -> absolute imports.

• add reference_name and next_reference_name attributes to AlignedSegment

• add function signatures to cvcf cython. Added note about other VCF code.

• add context manager functions to FastaFile

• add reference_name and next_reference_name attributes to AlignedSegment

• PileupColumn also gets a reference_name attribute.

• add context manager functions to FastaFile

• TabixFile.header for remote files raises AttributeError, fixes #157

• add context manager interface to TabixFile, closes #165

• change ctypedef enum to typedef enum for cython 0.23

• add function signatures to cvcf cython, also added note about other VCF code

• remove exception for custom upper-case header record tags.

• rename VALID_HEADER_FIELDS to KNOWN_HEADER_FIELDS

• fix header record tag parsing for custom tags.

• use cython.str in count_coverage, fixes #141

• avoid maketrans (issues with python3)

• refactoring: AlignedSegment now in separate module

• do not execute remote tests if URL not available

• fix the unmapped count, incl reads with no SQ group

• add raw output to tags

• added write access for binary tags

• bugfix in call to resize

• implemented writing of binary tags from arrays

• implemented convert_binary_tag to use arrays

• add special cases for reads that are unmapped or whose mates are unmapped.

• rename TabProxies to ctabixproxies

• remove underscores from utility functions

• move utility methods into cutils

• remove callback argument to fetch - closes #128

• avoid calling close in dealloc

• add unit tests for File object opening

• change AlignmentFile.open to filepath_or_object

• implement copy.copy, close #65

• add chaching of array attributes in AlignedSegment, closes #121

• add export of Fastafile

• remove superfluous pysam_dispatch

• use persist option in FastqFile

• get_tag: expose tag type if requested with with_value_type

• fix to allow reading vcf record info via tabix-based vcf reader

• add pFastqProxy and pFastqFile objects to make it possible to work with multiple fastq records per file handle, unlike FastqProxy/FastqFile.

• release GIL around htslib IO operations

• More work on read/write support, API improvements

• add phased property on VariantRecordSample

• add mutable properties to VariantRecord

• BCF fixes and start of read/write support

• VariantHeaderRecord objects now act like mappings for attributes.

• add VariantHeader.alts dict from alt ID->Record.

• Bug fix to strong representation of structured header records.

• VariantHeader is now mutable

• samtools command now accept the "catch_stdout" option.

• get_aligned_pairs now works for soft-clipped reads.

• query_position is now None when a PileupRead is not aligned to a particular position.

• AlignedSegments are now comparable and hashable.

• Installation bugfix release.

• Pysam now wraps htslib 1.2.1 and samtools version 1.2.

• Added CRAM file support to pysam.

• New alignment info interface.

  • opt() and setTag are deprecated, use get_tag() and set_tag() instead.

  • added has_tag()

  • tags is deprecated, use get_tags() and set_tags() instead.

• FastqFile is now FastxFile to reflect that the latter permits iteration over both fastq- and fasta-formatted files.

• A Cython wrapper for htslib VCF/BCF reader/writer. The wrapper provides a nearly complete Pythonic interface to VCF/BCF metadata with reading and writing capability. However, the interface is still incomplete and preliminary and lacks capability to mutate the resulting data.

• Pysam now wraps htslib and samtools versions 1.1.

• Bugfixes, most notable:

  • issue #43: uncompressed BAM output

  • issue #42: skip tests requiring network if none available

  • issue #19: multiple iterators can now be made to work on the same tabix file

  • issue #24: All strings returned from/passed to the pysam API are now unicode in python 3

  • issue #5: type guessing for lists of integers fixed

• API changes for consistency. The old API is still present, but deprecated. In particular:

  • Tabixfile -> TabixFile

  • Fastafile -> FastaFile

  • Fastqfile -> FastqFile

  • Samfile -> AlignmentFile

  • AlignedRead -> AlignedSegment

    • qname -> query_name

    • tid -> reference_id

    • pos -> reference_start

    • mapq -> mapping_quality

    • rnext -> next_reference_id

    • pnext -> next_reference_start

    • cigar -> cigartuples

    • cigarstring -> cigarstring

    • tlen -> template_length

    • seq -> query_sequence

    • qual -> query_qualities, now returns array

    • qqual -> query_alignment_qualities, now returns array

    • tags -> tags

    • alen -> reference_length, reference is always "alignment", so removed

    • aend -> reference_end

    • rlen -> query_length

    • query -> query_alignment_sequence

    • qstart -> query_alignment_start

    • qend -> query_alignment_end

    • qlen -> query_alignment_length

    • mrnm -> next_reference_id

    • mpos -> next_reference_start

    • rname -> reference_id

    • isize -> template_length

    • blocks -> get_blocks()

    • aligned_pairs -> get_aligned_pairs()

    • inferred_length -> infer_query_length()

    • positions -> get_reference_positions()

    • overlap() -> get_overlap()

  • All strings are now passed to or received from the pysam API as strings, no more bytes.

Other changes:

• AlignmentFile.fetch(reopen) option is now multiple_iterators. The default changed to not reopen a file unless requested by the user.

• FastaFile.getReferenceLength is now FastaFile.get_reference_length

Backwards incompatible changes

• Empty cigarstring now returns None (instead of '')

• Empty cigar now returns None (instead of [])

• When using the extension classes in cython modules, AlignedRead needs to be substituted with AlignedSegment.

• fancy_str() has been removed

• qual, qqual now return arrays

• Disabled features

  • IteratorColumn.setMask() disabled as htslib does not implement this functionality?

• Not implemented yet:

  • reading SAM files without header

Tabix files between version 0.7.8 and 0.8.0 are not compatible and need to be re-indexed.

While version 0.7.8 and 0.8.0 should be mostly compatible, there are some notable exceptions:

• tabix iterators will fail if there are comments in the middle or the end of a file.

• tabix raises always ValueError for invalid intervals. Previously, different types of errors were raised (KeyError, IndexError, ValueError) depending on the type of invalid intervals (missing chromosome, out-of-range, malformatted interval).

• added AlignedRead.setTag method

• added AlignedRead.blocks

• unsetting CIGAR strings is now possible

• empty CIGAR string returns empty list

• added reopen flag to Samfile.fetch()

• various bugfixes

• added Fastafile.references, .nreferences and .lengths

• tabix_iterator now uses kseq.h for python 2.7

• added inferred_length property

• issue 122: MACOSX getline missing, now it works?

• seq and qual can be set None

• added Fastqfile

• switch to samtools 0.1.19

• issue 122: MACOSX getline missing

• issue 130: clean up tempfiles

• various other bugfixes

• further bugfixes to setup.py and package layout

• further bugfixes to setup.py

• upgraded distribute_setup.py to 0.6.34

• bugfix in installer - failed when cython not present

• changed installation locations of shared libraries

• bugfix: missing PP tag PG records in header

• added pre-built .c files to distribution

• switch to tabix 0.2.6

• added cigarstring field

• python3 compatibility

• added B tag handling

• added check_sq and check_header options to Samfile.__init__

• added lazy GTF parsing to tabix

• reworked support for VCF format parsing

• bugfixes

• switch to samtools 0.1.18

• various bugfixes

• removed references to deprecated 'samtools pileup' functionality

• AlignedRead.tags now returns an empty list if there are no tags.

• added pnext, rnext and tlen

• switch to samtools 0.1.12a and tabix 0.2.3

• added snp and indel calling.

• switch from pyrex to cython

• changed handling of samtools stderr

• various bugfixes

• added Samfile.count and Samfile.mate

• deprecated AlignedRead.rname, added AlignedRead.tid

• switch to samtools 0.1.8

• added support for tabix files

• numerous bugfixes including

• permit simultaneous iterators on the same file

• working access to remote files