fugashi

This release addresses one of the longest standing issues, #55. Many thanks to @nikitalita figuring out how to cross-compile MeCab for wheels.

There are no other changes.

This release adds wheels for Python 3.11, with no other changes.

This release of fugashi adds one new feature: Tagger.nbestToNodeList returns the top N possible tokenizations of a string as node lists. Many thanks to @teowenshen for the implementation (#61).

This release also drops support for Python 3.6 and earlier versions. While the current source should still work with 3.5 and 3.6, wheels are not provided, and it is recommended you upgrade your Python version to one that has not reached end-of-life status. If you must use an older version, you can continue using v1.1.2.

This release adds long overdue wheels for Python 3.10. There are no changes in functionality or API.

On the backend, in addition to fixing issues with the 3.10 version number and quoting, the build process was cleaned up considerably. Many thanks to @lambdadog for the bugfixes and cleanup!

This release does not include wheels for M1 Macs - those may be working, but I've been unable to confirm it. See #55 for details or to help out.

This release has a number of stability and API improvements.

• fugashi-build-dict didn't work in its initial release, that has been fixed.
• Calls to parseToNode no longer invalidate old node surfaces (#38)
• Initialization errors now throw an Exception rather than printing output directly (https://github.com/explosion/spaCy/releases/tag/v3.0.7)

Note that the fix to #38 has a number of side effects that may need more extensive evaluation. In particular:

• memory use will grow very slowly over the life of a Tagger object
• execution speed will be a bit slower, up to around 10%

It's expected that these will both be addressed before long; despite the issues, the current fix has been deemed suitable for a release because in the vast majority of use cases it will behave more correctly than the previous release.

One feature fugashi hasn't had until now is the ability to build user dictionaries. This feature can be important for improving tokenization quality in many applications. This release adds fugashi-build-dict, a wrapper for MeCab's mecab-dict-index command. You can use it like this:

fugashi-build-dict -d [system-dic-dir] -u mydic.dic input.csv

If you're familiar with MeCab's user dictionary creation process nothing has changed, so any feedback on use or any errors you encounter would be appreciated. If you're not familiar with the dictionary process, just wait a bit - a guide should be released soon.