Coverage for dariah_topics/postprocessing.py : 71%
Hot-keys on this page
r m x p toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
|
""" Postprocessing Text Data, Saving Matrices, Corpora and LDA Models *****************************************************************
Functions of this module are for **postprocessing purpose**. You can save \ `document-term matrices <https://en.wikipedia.org/wiki/Document-term_matrix>`_, \ `tokenized corpora <https://en.wikipedia.org/wiki/Tokenization_(lexical_analysis)>`_ \ and `LDA models <https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation>`_, \ access topics, topic probabilites for documents, and word probabilities \ for each topic. All matrix variants provided in :func:`preprocessing.create_document_term_matrix()`_ \ are supported, as well as `lda <https://pypi.python.org/pypi/lda>`_, `Gensim <https://radimrehurek.com/gensim/>`_ \ and `MALLET <http://mallet.cs.umass.edu/topics.php>`_ models or output, respectively. \ Recurrent variable names are based on the following conventions:
* ``topics`` means a pandas DataFrame containing the top words for each \ topic and any Dirichlet parameters. * ``document_topics`` means a pandas DataFrame containing topic proportions per \ document, at the end of the iterations. * ``word_weights`` means unnormalized weights for every topic and word type. * ``keys`` means the top *n* tokens of a topic.
Contents ******** * :func:`doc2bow()` * :func:`save_document_term_matrix()` writes a document-term matrix to a `CSV <https://en.wikipedia.org/wiki/Comma-separated_values>`_ file or to a `Matrix Market <http://math.nist.gov/MatrixMarket/formats.html#MMformat>`_ file, respectively. * :func:`save_model()` saves a LDA model (except MALLET models, which will be saved \ by specifying a parameter of :func:`mallet.create_mallet_model()`). * :func:`save_tokenized_corpus()` writes tokens of a tokenized corpus to plain text \ files per document. * :func:`show_document_topics()` shows topic probabilities for each document. * :func:`show_topics()` shows topics generated by a LDA model. * :func:`show_word_weights()` shows word probabilities for each topic. """
"""Creates a `doc2bow` pandas Series for Gensim.
With this function you can create a `doc2bow` pandas Series as input for Gensim, e.g. \ to instantiate the :class:`gensim.models.LdaModel` class or get topic distributions \ with :func:`gensim.models.LdaModel.get_document_topics()`.
Args: document_term_matrix (pandas.DataFrame): A document-term matrix **designed for large corpora**.
Returns: List of lists containing tuples.
Todo: * Improve efficiency.
Example: >>> from cophi_toolbox import preprocessing >>> tokenized_corpus = [['this', 'is', 'document', 'one'], ['this', 'is', 'document', 'two']] >>> document_labels = ['document_one', 'document_two'] >>> document_term_matrix, _, _ = preprocessing.create_document_term_matrix(tokenized_corpus, document_labels, True) >>> isinstance(doc2bow(document_term_matrix), pd.Series) True """
"""Saves document-term matrix.
Writes a ``document_term_matrix`` and, in case of a large corpus matrix, \ ``document_ids`` and ``type_ids``, which have to be specified, to comma-separated \ values (CSV) files. Furthermore, if ``document_term_matrix`` is designed for \ large corpora and ``matrix_market`` is True, the matrix will be saved in the \ `Matrix Market format <http://math.nist.gov/MatrixMarket/formats.html#MMformat>`_ (`.mm`). \ Libraries like `scipy <https://www.scipy.org>`_ and `gensim <https://radimrehurek.com/gensim/>`_ \ are able to read and process the Matrix Market format. Use the function :func:`preprocessing.create_document_term_matrix()` to create a document-term matrix.
Args: document_term_matrix (pandas.DataFrame): Document-term matrix with rows corresponding to ``document_labels`` and columns corresponding to types (unique tokens in the corpus). The single values of the matrix are type frequencies. Will be saved as ``document_term_matrix.csv`` or ``document_term_matrix.mm``, respectively. path (str): Path to the output directory. document_ids (dict, optional): Dictionary containing ``document_labels`` as keys and an unique identifier as value. Only required, if ``document_term_matrix`` is designed for large corpora. Will be saved as ``document_ids.csv``. Defaults to None. type_ids (dict, optional): Dictionary containing types as keys and an unique identifier as value. Only required, if ``document_term_matrix`` is designed for large corpora. Will be saved as ``type_ids.csv``. Defaults to None. matrix_market (bool, optional): If True, matrix will be saved in Matrix Market format. Only for the large corpus variant of ``document_term_matrix`` available. Defaults to False.
Returns: None.
Example: >>> from cophi_toolbox import preprocessing >>> import os >>> path = 'tmp' >>> tokenized_corpus = [['this', 'is', 'document', 'one'], ['this', 'is', 'document', 'two']] >>> document_labels = ['document_one', 'document_two'] >>> document_term_matrix = preprocessing.create_document_term_matrix(tokenized_corpus, document_labels) >>> save_document_term_matrix(document_term_matrix=document_term_matrix, path=path) >>> preprocessing.read_document_term_matrix(os.path.join(path, 'document_term_matrix.csv')) #doctest +NORMALIZE_WHITESPACE this is document two one document_one 1.0 1.0 1.0 0.0 1.0 document_two 1.0 1.0 1.0 1.0 0.0 >>> document_term_matrix, document_ids, type_ids = preprocessing.create_document_term_matrix(tokenized_corpus, document_labels, True) >>> save_document_term_matrix(document_term_matrix, path, document_ids, type_ids) >>> isinstance(preprocessing.read_document_term_matrix(os.path.join(path, 'document_term_matrix.csv')), pd.DataFrame) True """ log.info("Creating directory {} ...".format(path)) os.makedirs(path) else: raise ValueError("You have to pass document_ids and type_ids as parameters.") _save_matrix_market(document_term_matrix, path)
"""Saves a LDA model.
With this function you can save a LDA model using :module:`pickle`. If you want \ to save MALLET models, you have to specify a parameter of the function :func:`mallet.create_mallet_model()`.
Args: model: Fitted LDA model produced by `Gensim <https://radimrehurek.com/gensim/>`_ or `lda <https://pypi.python.org/pypi/lda>`_. filepath (str): Path to LDA model, e.g. ``/home/models/model.pickle``.
Returns: None.
Example: >>> from lda import LDA >>> from gensim.models import LdaModel >>> from dariah_topics import preprocessing >>> save_model(LDA, 'model.pickle') >>> preprocessing.read_model('model.pickle') == LDA True >>> save_model(LdaModel, 'model.pickle') >>> preprocessing.read_model('model.pickle') == LdaModel True """
"""Writes a tokenized corpus to text files.
With this function you can write tokens of a `tokenized_corpus` to plain text \ files per document to ``path``. Every file will be named after its ``document_label``. \ Depending on the used tokenizer, ``tokenized_corpus`` does normally not contain \ any punctuations or one-letter words. Use the function :func:`preprocessing.tokenize()` to tokenize a corpus.
Args: tokenized_corpus (list): Tokenized corpus containing one or more iterables containing tokens. document_labels (list): Name of each `tokenized_document` in `tokenized_corpus`. path (str): Path to the output directory.
Returns: None
Example: >>> tokenized_corpus = [['this', 'is', 'a', 'tokenized', 'document']] >>> document_labels = ['document_label'] >>> path = 'tmp' >>> save_tokenized_corpus(tokenized_corpus, document_labels, path) >>> with open(os.path.join(path, 'document_label.txt'), 'r', encoding='utf-8') as file: ... file.read() 'this\\nis\\na\\ntokenized\\ndocument' """ log.info("Creating directory {} ...".format(path)) os.makedirs(path)
"""Shows topic distribution for each document.
With this function you can show the topic distributions for all documents in a pandas DataFrame. \ For each topic, the top ``num_keys`` keys will be considered. If you have a * `lda <https://pypi.python.org/pypi/lda>`_ model, you have to pass the model \ as ``model`` and the document-term matrix vocabulary as ``vocabulary``. * `Gensim <https://radimrehurek.com/gensim/>`_ model, you have to pass only the model \ as ``model``. * `MALLET <http://mallet.cs.umass.edu/topics.php>`_ based workflow, you have to\ pass only the ``doc_topics_file``.
Args: topics (pandas.DataFrame, optional): Only for lda models. A pandas DataFrame containing all topics. Produced by visualization.show_topics() model (optional): lda or Gensim model. document_labels (list, optional): An list of all document labels. doc_topics_file (str, optional): Only for MALLET. Path to the doc-topics file. doc2bow (list, optional): A list of lists containing tuples of ``type_id`` and frequency. num_keys (int, optional): Number of top keys for each topic. dec (int, optional): Number of decimal places in the document-topics-value
Returns: A pandas DataFrame with rows corresponding to topics and columns corresponding to documents.
Example: """
"""Shows topics of LDA model.
With this function you can show all topics of a LDA model in a pandas DataFrame. \ For each topic, the top ``num_keys`` keys will be considered. If you have a * `lda <https://pypi.python.org/pypi/lda>`_ model, you have to pass the model \ as ``model`` and the document-term matrix vocabulary as ``vocabulary``. * `Gensim <https://radimrehurek.com/gensim/>`_ model, you have to pass only the model \ as ``model``. * `MALLET <http://mallet.cs.umass.edu/topics.php>`_ based workflow, you have to\ pass only the ``topic_keys_file``.
Args: model (optional): lda or Gensim model. vocabulary (list, optional): Only for lda. The vocabulary of the document-term matrix. topic_keys_file (str): Only for MALLET. Path to the topic keys file. num_keys (int, optional): Number of top keys for each topic.
Returns: A pandas DataFrame with rows corresponding to topics and columns corresponding to keys.
Example: """
"""Read Mallet word_weigths file
Description: Reads Mallet word_weigths into pandas DataFrame.
Args: word_weigts_file: Word_weights_file created with Mallet
Returns: Pandas DataFrame
Todo: * Adapt for ``lda`` and ``gensim`` output.
Example: >>> import tempfile >>> with tempfile.NamedTemporaryFile(suffix='.txt') as tmpfile: ... tmpfile.write(b'0\\tthis\\t0.5\\n0\\tis\\t0.4\\n0\\ta\\t0.3\\n0\\tdocument\\t0.2') and True ... tmpfile.flush() ... show_word_weights(tmpfile.name, 2) #doctest: +NORMALIZE_WHITESPACE True document token weight 0 0 this 0.5 1 0 is 0.4
"""
"""Collects data into fixed-length chunks or blocks.
This private function is wrapped in :func:`_show_mallet_document_topics()`.
Args: n (int): Length of chunks or blocks iterable (object): Iterable object fillvalue (boolean): If iterable can not be devided into evenly-sized chunks fill chunks with value.
Returns: n-sized chunks
""" args = [iter(iterable)] * n return itertools.zip_longest(*args, fillvalue=fillvalue)
"""Creates a document-topic-matrix.
Description: With this function you can create a doc-topic-maxtrix for gensim output.
Args: corpus (mmCorpus): Gensim corpus. model: Gensim LDA model doc_labels (list): List of document labels.
Returns: Doc_topic-matrix as DataFrame
Example: >>> from gensim.models import LdaModel >>> from gensim.corpora import Dictionary >>> document_labels = ['document_one', 'document_two'] >>> tokenized_corpus = [['this', 'is', 'the', 'first', 'document'], ['this', 'is', 'the', 'second', 'document']] >>> id2word = Dictionary(tokenized_corpus) >>> corpus = [id2word.doc2bow(document) for document in tokenized_corpus] >>> model = LdaModel(corpus=corpus, id2word=id2word, iterations=1, passes=1, num_topics=2) >>> topics = _show_gensim_topics(model, 5) >>> index = [' '.join(keys[:2]) for keys in topics.values] >>> isinstance(_show_gensim_document_topics(corpus, model, document_labels, index), pd.DataFrame) True """
"""Converts gensim output to DataFrame.
Description: With this function you can convert gensim output (usually a list of tuples) to a DataFrame, a more convenient datastructure.
Args: model: Gensim LDA model. num_keys (int): Number of top keywords for topic.
Returns: DataFrame.
ToDo:
Example: >>> from gensim.models import LdaModel >>> from gensim.corpora import Dictionary >>> tokenized_corpus = [['this', 'is', 'the', 'first', 'document'], ['this', 'is', 'the', 'second', 'document']] >>> id2word = Dictionary(tokenized_corpus) >>> corpus = [id2word.doc2bow(document) for document in tokenized_corpus] >>> model = LdaModel(corpus=corpus, id2word=id2word, iterations=1, passes=1, num_topics=2) >>> isinstance(_show_gensim_topics(model, 5), pd.DataFrame) True """
"""Creates a doc_topic_matrix for lda output.
Description: With this function you can convert lda output to a DataFrame, a more convenient datastructure. Use 'lda2DataFrame()' to get topics.
Note:
Args: model: Gensim LDA model. topics: DataFrame. doc_labels (list[str]): List of doc labels as string.
Returns: DataFrame
Example: >>> import lda >>> from cophi_toolbox import preprocessing >>> tokenized_corpus = [['this', 'is', 'the', 'first', 'document'], ['this', 'is', 'the', 'second', 'document']] >>> document_labels = ['document_one', 'document_two'] >>> document_term_matrix = preprocessing.create_document_term_matrix(tokenized_corpus, document_labels) >>> vocabulary = document_term_matrix.columns >>> model = lda.LDA(n_topics=2, n_iter=1) >>> model = model.fit(document_term_matrix.as_matrix().astype(int)) >>> topics = _show_lda_topics(model, vocabulary, num_keys=5) >>> index = [' '.join(keys[:3]) for keys in topics.values] >>> isinstance(_show_lda_document_topics(model, document_labels, index), pd.DataFrame) True """
"""Converts lda output to a DataFrame
Description: With this function you can convert lda output to a DataFrame, a more convenient datastructure.
Note:
Args: model: LDA model. vocab (list[str]): List of strings containing corpus vocabulary. num_keys (int): Number of top keywords for topic
Returns: DataFrame
Example: >>> import lda >>> from cophi_toolbox import preprocessing >>> tokenized_corpus = [['this', 'is', 'the', 'first', 'document'], ['this', 'is', 'the', 'second', 'document']] >>> document_labels = ['document_one', 'document_two'] >>> document_term_matrix = preprocessing.create_document_term_matrix(tokenized_corpus, document_labels) >>> vocabulary = document_term_matrix.columns >>> model = lda.LDA(n_topics=2, n_iter=1) >>> model = model.fit(document_term_matrix.as_matrix().astype(int)) >>> isinstance(_show_lda_topics(model, vocabulary, num_keys=5), pd.DataFrame) True """
"""Shows document-topic-mapping. Args: outfolder (str): Folder for MALLET output. doc_topics (str): Name of MALLET's doc_topic file. Defaults to 'doc_topics.txt'. index (list): array-like topic labels
ToDo: Prettify docnames
Example: >>> import tempfile >>> index = ['first topic', 'second topic'] >>> with tempfile.NamedTemporaryFile(suffix='.txt') as tmpfile: ... tmpfile.write(b'0\\tdocument_one.txt\\t0.1\\t0.2\\n1\\tdocument_two.txt\\t0.4\\t0.5') and True ... tmpfile.flush() ... _show_mallet_document_topics(tmpfile.name, index, True) #doctest: +NORMALIZE_WHITESPACE True document_one document_two first topic 0.1 0.4 second topic 0.2 0.5 """ lines = file.readlines() for line in lines: documet_number, document_label, *values = line.rstrip().split('\t') document_labels.append(os.path.splitext(os.path.basename(document_label))[0]) for topic, share in _grouper(2, values): triple = (document_label, int(topic), float(share)) topics.append(int(topic)) document_topics_triples.append(triple) else: else: document_topics_triples = sorted(document_topics_triples, key=operator.itemgetter(0, 1)) document_labels = sorted(document_labels) num_documents = len(document_labels) num_topics = len(topics) document_topics = np.zeros((num_documents, num_topics)) for triple in document_topics_triples: document_label, topic, share = triple index_num = document_labels.index(document_label) document_topics[index_num, topic] = share return pd.DataFrame(document_topics, index=index, columns=columns.T)
"""Show topic-key-mapping.
Args: outfolder (str): Folder for Mallet output, topicsKeyFile (str): Name of Mallets' topic_key file, default "topic_keys"
#topic-model-mallet Note: FBased on DARIAH-Tutorial -> https://de.dariah.eu/tatom/topic_model_mallet.html
ToDo: Prettify index
Example: >>> import tempfile >>> with tempfile.NamedTemporaryFile(suffix='.txt') as tmpfile: ... tmpfile.write(b'0\\t0.5\\tthis is the first document\\n1\\t0.5\\tthis is the second document') and True ... tmpfile.flush() ... _show_mallet_topics(tmpfile.name) True Key 0 Key 1 Key 2 Key 3 Key 4 Topic 0 this is the first document Topic 1 this is the second document """
""" Writes a `document_term_matrix` designed for large corpora to `Matrix Market <http://math.nist.gov/MatrixMarket/formats.html#MMformat>`_ file (`.mm`). Libraries like `scipy <https://www.scipy.org>`_ and `gensim <https://radimrehurek.com/gensim/>`_ are able to read and process the Matrix Market format. This private function is wrapped in `save_document_term_matrix()`.
**Use the function `preprocessing.create_document_term_matrix()` to create a document-term matrix.**
Args: document_term_matrix (pandas.DataFrame): Document-term matrix with only one column corresponding to type frequencies and a pandas MultiIndex with `document_ids` for level 0 and `type_ids` for level 1. Will be saved as `document_term_matrix.mm`. path (str): Path to the output directory.
Returns: None.
Example: """ num_docs = document_term_matrix.index.get_level_values('document_id').max() num_types = document_term_matrix.index.get_level_values('type_id').max() sum_counts = document_term_matrix[0].sum() header = "{} {} {}\n".format(num_docs, num_types, sum_counts)
with open(os.path.join(path, 'document_term_matrix.mm'), 'w', encoding='utf-8') as file: file.write("%%MatrixMarket matrix coordinate real general\n") file.write(header) document_term_matrix.to_csv(file, sep=' ', header=None) return None
if vocabulary is not None and topic_word_weights_file is None: key_weights = _show_lda_key_weights(model, vocabulary, topic_no, num_keys) elif vocabulary is None and topic_word_weights_file is None: key_weights = _show_gensim_key_weights(model, topic_no, num_keys) elif topic_word_weights_file is not None: key_weights = _show_mallet_key_weights(topic_word_weights_file, topic_no) if sort_ascending is None: return pd.Series(key_weights)[:num_keys] else: return pd.Series(key_weights).sort_values(ascending=sort_ascending)[:num_keys]
return {key: weight for key, weight in zip(vocabulary[:num_keys], model.components_[topic_no][:num_keys])}
return dict(model.show_topic(topic_no, num_keys))
key_weights = pd.read_table(topic_word_weights_file, sep='\t', header=None, names=['topic_id', 'key', 'weight']) key_weights = key_weights[key_weights['topic_id'] == topic_no].drop('topic_id', axis=1) return key_weights.set_index('key')['weight'].to_dict()
return np.array(values)[np.argsort(distribution)][:-length-1:-1] |