API

Cihai core functionality.

class cihai.core.Cihai(config=None, unihan=True)

Bases: object

Central application object.

By default, this automatically adds the UNIHAN dataset.

config

Type:

dict

Notes

Inspired by the early pypa/warehouse applicaton object [1]_.

Configuration templates

The config dict parameter supports a basic template system for replacing XDG Base Directory directory variables, tildes and environmentas variables. This is done by passing the option dict through cihai.config.expand_config() during initialization.

Examples

To use cihai programatically, invoke and install the UNIHAN [2]_ dataset:

#!/usr/bin/env python
from cihai.core import Cihai


def run(unihan_options=None):
    c = Cihai()

    if not c.unihan.is_bootstrapped:  # download and install Unihan to db
        c.unihan.bootstrap(unihan_options)

    query = c.unihan.lookup_char("好")
    glyph = query.first()
    print("lookup for 好: %s" % glyph.kDefinition)

    query = c.unihan.reverse_char("good")
    print('matches for "good": %s ' % ", ".join([glph.char for glph in query]))


if __name__ == "__main__":
    run()

Above: is_bootstrapped can check if the system has the database installed.

References

[1]

PyPA Warehouse on GitHub. https://github.com/pypa/warehouse. Accessed sometime in 2013.

[2]

UNICODE HAN DATABASE (UNIHAN) documentation. https://www.unicode.org/reports/tr38/. Accessed March 31st, 2018.

Parameters:

• config (dict, optional) –

• unihan (boolean, optional) – Bootstrap the core UNIHAN dataset (recommended)

default_config = {'database': {'url': 'sqlite:///{user_data_dir}/cihai.db'}, 'debug': False, 'dirs': {'cache': '{user_cache_dir}', 'data': '{user_data_dir}', 'log': '{user_log_dir}'}}

dict of default config, can be monkey-patched during tests

config

Configuration dictionary

sql

Database instance

Type:

cihai.db.Database

bootstrap()

add_dataset(_cls, namespace)

classmethod from_file(config_path=None, *args, **kwargs)

Create a Cihai instance from a JSON or YAML config.

Parameters:

config_path (str, optional) – path to custom config file

Returns:

application object

Return type:

Cihai

Configuration

cihai.config.expand_config(d, dirs)

Expand configuration XDG variables, environmental variables, and tildes.

Parameters:

• d (dict) – config information

• dirs (appdirs.AppDirs) – XDG application mapping

Notes

Environmentable variables are expanded via os.path.expandvars(). So ${PWD} would be replaced by the current PWD in the shell, ${USER} would be the user running the app.

XDG variables are expanded via str.format(). These do not have a dollar sign. They are:

• {user_cache_dir}

• {user_config_dir}

• {user_data_dir}

• {user_log_dir}

• {site_config_dir}

• {site_data_dir}

See also

os.path.expanduser, os.path.expandvars

class cihai.config.Configurator(namespace='')

Bases: object

Manage config. Provides facilities for loading / writing configs.

Used on Cihai and available for its extensions.

Parameters:

namespace (str, optional) –

Creates a configuration object that reads/writes on a namespace.

Leaving this empty / reads and writes to the root level cihai config.

Namespace is designed for plugins to manage settings.

dirs

XDG App directory locations for cihai

Type:

appdirs.AppDirs

Class Attributes

----------------

data

property / getter for options

Type:

dict

_data

where the raw dictionary resides

Type:

dict

get_names()

Return a list of possible places config can reside, and order of search.

This is based on XDG. So it will look for

property file

Find a config file where it exists, as the first place.

read()

Read to dictionary.

get_delta(**updates)

Returns the difference of whatever user customizations differ from cihai.constants.DEFAULT_CONFIG.

write(**updates)

If no delta is created from DEFAULT, it not write.

If file doesn’t exist, it will create.

Database

Cihai core functionality.

class cihai.db.Database(config)

Bases: object

Cihai SQLAlchemy instance

reflect_db()

No-op to reflect db info.

This is available as a method so the database can be reflected outside initialization (such bootstrapping unihan during CLI usage).

engine = None

sqlalchemy.engine.Engine instance.

metadata = None

sqlalchemy.schema.MetaData instance.

session = None

sqlalchemy.orm.session.Session instance.

base = None

sqlalchemy.ext.automap.AutomapBase instance.

Extending

Cihai Plugin System

Status: Experimental, API can change

As a pilot, the UNIHAN library, and an plugin for it, in #131 [1]_

You can bring any data layout / backend you like to cihai.

For convenience, you can use cihai’s configuration namespace and SQLAlchemy settings.

You can also create plugins which extend another. So if Unihan doesn’t have a lookup for variant glyphs, this can be added.

class cihai.extend.ConfigMixin

Bases: object

This piggybacks cihai’s global config state, as well as your datasets.

Cihai will automatically manage the user’s config, as well as your datasets, neatly in XDG.

Raises:

• Functions inside, and what you write relating to dataset config should return –

• CihaiDatasetConfigException (CihaiDatasetException) –

• config.cihai = links directly back to Cihai's configuration dictionary –

• (todo note – make this non-mutable property):

:raises config : dict: your local user’s config :raises check() : function, optional: this is ran on start. it can raise DatasetConfigException :raises default_config : your dataset’s default configuration: :raises get_default_config : override function in case you’d like custom configs (for: instnace if you want a platform to use a different db driver, or do version checks, etc.) internal functions use get_default_config()

class cihai.extend.SQLAlchemyMixin

Bases: object

Your dataset can use any backend you’d like, we provide a backend for you, that automatically piggybacks on cihai’s zero-config, XDG / SQLAchemy configuration. So it’s preconfigured for the user.

In addition, this mixin gives you access to any other of the user’s sqlalchemy sql that use this mixin. So if you want a dataset that utilitizes UNIHAN, you can access that easily.

This will provide the following instance-level properties in methods:

When you have access, it’s expected to keep your tables / databases namespaced so they don’t clobber.

engine = None

sqlalchemy.engine.Engine instance.

metadata = None

sqlalchemy.schema.MetaData instance.

session = None

sqlalchemy.orm.session.Session instance.

base = None

sqlalchemy.ext.automap.AutomapBase instance.

class cihai.extend.Dataset

Bases: object

Cihai dataset, e.g. UNIHAN.

See also

cihai.data.unihan.dataset.Unihan

reference implementation

bootstrap()

add_plugin(_cls, namespace, bootstrap=True)

class cihai.extend.DatasetPlugin

Bases: object

Extend the functionality of datasets with custom methods, actions, etc.

See also

cihai.data.unihan.dataset.UnihanVariants

reference implementation

Constants

cihai.constants.DEFAULT_CONFIG = {'database': {'url': 'sqlite:///{user_data_dir}/cihai.db'}, 'debug': False, 'dirs': {'cache': '{user_cache_dir}', 'data': '{user_data_dir}', 'log': '{user_log_dir}'}}

Default configuration

cihai.constants.UNIHAN_CONFIG = {'datasets': {'unihan': 'cihai.data.unihan.dataset.Unihan'}}

User will be prompted to automatically configure their installation for UNIHAN

UNIHAN Dataset

Bootstrapping

cihai.data.unihan.bootstrap.bootstrap_unihan(metadata, options=None)

cihai.data.unihan.bootstrap.flatten_datasets(d)

cihai.data.unihan.bootstrap.is_bootstrapped(metadata)

Return True if cihai is correctly bootstrapped.

cihai.data.unihan.bootstrap.create_unihan_table(columns, metadata)

Create table and return sqlalchemy.Table.

Parameters:

• columns (list) – columns for table, e.g. ['kDefinition', 'kCantonese']

• metadata (sqlalchemy.schema.MetaData) – Instance of sqlalchemy metadata

Returns:

Newly created table with columns and index.

Return type:

sqlalchemy.schema.Table

class cihai.data.unihan.dataset.Unihan

Bases: Dataset, SQLAlchemyMixin

bootstrap(options=None)

lookup_char(char)

Return character information from datasets.

Parameters:

char (str) – character / string to lookup

Returns:

list of matches

Return type:

sqlalchemy.orm.query.Query

reverse_char(hints)

Return QuerySet of objects from SQLAlchemy of results.

Parameters:

hints (list of str) – strings to lookup

Returns:

reverse matches

Return type:

sqlalchemy.orm.query.Query

with_fields(*fields)

Returns list of characters with information for certain fields.

Parameters:

*fields (list of str) – fields for which information should be available

Returns:

list of matches

Return type:

sqlalchemy.orm.query.Query

property is_bootstrapped

Return True if UNIHAN and database is set up.

Returns:

True if Unihan application fixture data installed.

Return type:

bool

add_plugin(_cls, namespace, bootstrap=True)

base = None

sqlalchemy.ext.automap.AutomapBase instance.

engine = None

sqlalchemy.engine.Engine instance.

metadata = None

sqlalchemy.schema.MetaData instance.

session = None

sqlalchemy.orm.session.Session instance.

cihai.data.unihan.constants.UNIHAN_FILES = ['Unihan_DictionaryLikeData.txt', 'Unihan_IRGSources.txt', 'Unihan_NumericValues.txt', 'Unihan_RadicalStrokeCounts.txt', 'Unihan_Readings.txt', 'Unihan_Variants.txt']

Mapping of files from unihan-etl (UNIHAN database)

cihai.data.unihan.constants.UNIHAN_FIELDS = ['kAccountingNumeric', 'kCangjie', 'kCantonese', 'kCheungBauer', 'kCihaiT', 'kCompatibilityVariant', 'kDefinition', 'kFenn', 'kFourCornerCode', 'kFrequency', 'kGradeLevel', 'kHDZRadBreak', 'kHKGlyph', 'kHangul', 'kHanyuPinlu', 'kHanyuPinyin', 'kJapaneseKun', 'kJapaneseOn', 'kKorean', 'kMandarin', 'kOtherNumeric', 'kPhonetic', 'kPrimaryNumeric', 'kRSAdobe_Japan1_6', 'kRSJapanese', 'kRSKanWa', 'kRSKangXi', 'kRSKorean', 'kRSUnicode', 'kSemanticVariant', 'kSimplifiedVariant', 'kSpecializedSemanticVariant', 'kTang', 'kTotalStrokes', 'kTraditionalVariant', 'kVietnamese', 'kXHC1983', 'kZVariant']

Mapping of field names from unihan-etl (UNIHAN database)

cihai.data.unihan.constants.UNIHAN_ETL_DEFAULT_OPTIONS = {'expand': False, 'fields': ['kAccountingNumeric', 'kCangjie', 'kCantonese', 'kCheungBauer', 'kCihaiT', 'kCompatibilityVariant', 'kDefinition', 'kFenn', 'kFourCornerCode', 'kFrequency', 'kGradeLevel', 'kHDZRadBreak', 'kHKGlyph', 'kHangul', 'kHanyuPinlu', 'kHanyuPinyin', 'kJapaneseKun', 'kJapaneseOn', 'kKorean', 'kMandarin', 'kOtherNumeric', 'kPhonetic', 'kPrimaryNumeric', 'kRSAdobe_Japan1_6', 'kRSJapanese', 'kRSKanWa', 'kRSKangXi', 'kRSKorean', 'kRSUnicode', 'kSemanticVariant', 'kSimplifiedVariant', 'kSpecializedSemanticVariant', 'kTang', 'kTotalStrokes', 'kTraditionalVariant', 'kVietnamese', 'kXHC1983', 'kZVariant'], 'format': 'python', 'input_files': ['Unihan_DictionaryLikeData.txt', 'Unihan_IRGSources.txt', 'Unihan_NumericValues.txt', 'Unihan_RadicalStrokeCounts.txt', 'Unihan_Readings.txt', 'Unihan_Variants.txt']}

Default settings passed to unihan-etl

Variants plugin

class cihai.data.unihan.dataset.UnihanVariants

Bases: DatasetPlugin

bootstrap()

Conversion

cihai.conversion.euc_to_unicode(hexstr)

Return EUC-CN (GB2312) hex to a Python unicode.

Parameters:

hexstr (bytes) –

Returns:

Python unicode e.g. u'\\u4e00' / ‘一’.

Return type:

unicode

Examples

>>> u'\u4e00'.encode('gb2312').decode('utf-8')
'\u04bb'

>>> (b'\\x' + b'd2' + b'\\x' + b'bb').replace('\\x', '') \  
... .decode('hex').decode('utf-8')
u'\u04bb'

Note: bytes don’t have a .replace:

>>> gb_enc = gb_enc.replace('\\x', '').decode('hex')  
>>> gb_enc.decode('string_escape')  # Won't work with Python 3.x.  

cihai.conversion.euc_to_utf8(euchex)

Convert EUC hex (e.g. “d2bb”) to UTF8 hex (e.g. “e4 b8 80”).

cihai.conversion.gb2312_to_euc(gb2312hex)

Convert GB2312-1980 hex (internal representation) to EUC-CN hex (the “external encoding”)

cihai.conversion.kuten_to_gb2312(kuten)

Convert GB kuten / quwei form (94 zones * 94 points) to GB2312-1980 / ISO-2022-CN hex (internal representation)

cihai.conversion.python_to_euc(uni_char, as_bytes=False)

Return EUC character from a Python Unicode character.

Converts a one character Python unicode string (e.g. u’u4e00’) to the corresponding EUC hex (‘d2bb’).

cihai.conversion.python_to_ucn(uni_char, as_bytes=False)

Return UCN character from Python Unicode character.

Converts a one character Python unicode string (e.g. u’u4e00’) to the corresponding Unicode UCN (‘U+4E00’).

cihai.conversion.ucn_to_unicode(ucn)

Convert a Unicode Universal Character Number (e.g. “U+4E00” or “4E00”) to Python unicode (u’u4e00’)

cihai.conversion.ucnstring_to_python(ucn_string)

Return string with Unicode UCN (e.g. “U+4E00”) to native Python Unicode (u’u4e00’).

cihai.conversion.ucnstring_to_unicode(ucn_string)

Return ucnstring as Unicode.

Exceptions

When using cihai via Python, you can catch Cihai-specific exceptions via these. All Cihai-specific exceptions are catchable via CihaiException since its the base exception.

Exceptions raised from the Cihai library.

exception cihai.exc.CihaiException

Bases: Exception

Base Cihai Exception class.

args

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception cihai.exc.ImportStringError(import_name, exception)

Bases: ImportError, CihaiException

Provides information about a failed import_string() attempt.

Notes

This is from werkzeug.utils d36aaf1 on August 20 2022, LICENSE BSD. https://github.com/pallets/werkzeug

Changes: - Deferred load import import_string from cihai.util - Format with black

import_name: str

String in dotted notation that failed to be imported.

args

msg

exception message

name

module name

path

module path

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

exception: BaseException

Wrapped exception.

Utilities

Utility and helper methods for cihai.

cihai.utils.supports_wide()

Return affirmative if python interpreter supports wide characters.

Returns:

True if python supports wide character sets

Return type:

bool

cihai.utils.import_string(import_name, silent=False)

Imports an object based on a string.

This is useful if you want to use import paths as endpoints or something similar. An import path can be specified either in dotted notation (xml.sax.saxutils.escape) or with a colon as object delimiter (xml.sax.saxutils:escape).

If silent is True the return value will be None if the import fails.

Parameters:

• import_name (string) – the dotted name for the object to import.

• silent (bool) – if set to True import errors are ignored and None is returned instead.

Return type:

imported object

Raises:

cihai.exc.ImportStringError (ImportError, cihai.exc.CihaiException) –

Notes

This is from werkzeug.utils d36aaf1 on May 23, 2022, LICENSE BSD. https://github.com/pallets/werkzeug

Changes: - Exception raised is cihai.exc.ImportStringError - Format with black

---