libsent/src/ngram/ngram_read_arpa.c

ARPA形式のN-gramファイルを読み込む. [詳細]

#include <sent/stddefs.h>
#include <sent/ngram2.h>

ngram_read_arpa.cのインクルード依存関係図

ソースコードを見る。

関数
static WORD_ID	lookup_word (NGRAM_INFO *ndata, char *str)
void	set_unknown_id (NGRAM_INFO *ndata)
	Set unknown word ID to the N-gram data.
static void	set_total_info (FILE *fp, NGRAM_INFO *ndata)
static void	set_and_check_total_info (FILE *fp, NGRAM_INFO *ndata)
static void	set_unigram (FILE *fp, NGRAM_INFO *ndata)
static void	add_unigram (FILE *fp, NGRAM_INFO *ndata)
static void	set_bigram (FILE *fp, NGRAM_INFO *ndata)
static void	add_bigram_rl (FILE *fp, NGRAM_INFO *ndata)
static void	set_trigram (FILE *fp, NGRAM_INFO *ndata)
boolean	ngram_read_arpa (FILE *fp, NGRAM_INFO *ndata, int direction)
void	ngram_compact_bigram_context (NGRAM_INFO *ndata)
変数
static char	buf [800]
	Local buffer for reading.
static char	pbuf [800]
	Local buffer for error string.
static boolean	LR_2gram_read = FALSE
	TRUE if LR 2gram has already been read.

---

説明

ARPA形式のN-gramファイルを読み込む.

作者:

Akinobu LEE

日付:

Wed Feb 16 16:52:24 2005

ARPA形式のN-gramファイルを用いる場合，2-gram と逆向き 3-gram を それぞれ別々のファイルから読み込みます．

参照:

ngram2.h

Revision

1.6

ngram_read_arpa.c で定義されています。

---

関数

static WORD_ID lookup_word	(	NGRAM_INFO *	ndata,
		char *	str
	)			[static]

Get N-gram word/class id of a string, and terminate program if not found.

引数:

	ndata	[in] N-gram data
	str	[in] name string of N-gram entry

戻り値:

the entry ID.

ngram_read_arpa.c の 52 行で定義されています。

参照元 add_bigram_rl(), add_unigram(), set_bigram(), と set_trigram().

void set_unknown_id

(

NGRAM_INFO *

ndata

)

Set unknown word ID to the N-gram data.

In CMU-Cam SLM toolkit, OOV words are always mapped to UNK, which always appear at the very beginning of N-gram entry, so we fix the unknown word ID at "0".

引数:

ndata

[out] N-gram data to set unknown word ID.

ngram_read_arpa.c の 72 行で定義されています。

参照元 ngram_read_arpa(), と ngram_read_bin().

static void set_total_info	(	FILE *	fp,
		NGRAM_INFO *	ndata
	)			[static]

Set number of N-gram entries, for reading the first LR 2-gram.

引数:

	fp	[in] file pointer
	ndata	[out] N-gram data to set it.

ngram_read_arpa.c の 97 行で定義されています。

参照元 ngram_read_arpa().

static void set_and_check_total_info	(	FILE *	fp,
		NGRAM_INFO *	ndata
	)			[static]

Read number of N-gram entries of the second RL 3-gram, and check if those values are exactly the same as the previous LR values.

引数:

	fp	[in] file pointer
	ndata	[i/o] N-gram data

ngram_read_arpa.c の 121 行で定義されています。

参照元 ngram_read_arpa().

static void set_unigram	(	FILE *	fp,
		NGRAM_INFO *	ndata
	)			[static]

Read word/class entry names and 1-gram data from LR 2-gram file.

引数:

	fp	[in] file pointer
	ndata	[out] N-gram to set the read data.

ngram_read_arpa.c の 155 行で定義されています。

参照元 ngram_read_arpa().

static void add_unigram	(	FILE *	fp,
		NGRAM_INFO *	ndata
	)			[static]

Read 1-gram data from RL 3-gram file. Only the back-off weights are stored.

引数:

	fp	[in] file pointer
	ndata	[out] N-gram to store the read data.

ngram_read_arpa.c の 220 行で定義されています。

参照元 ngram_read_arpa().

static void set_bigram	(	FILE *	fp,
		NGRAM_INFO *	ndata
	)			[static]

Read 2-gram data from LR 2-gram file.

引数:

	fp	[in] file pointer
	ndata	[out] N-gram to set the read data.

ngram_read_arpa.c の 260 行で定義されています。

参照元 ngram_read_arpa().

static void add_bigram_rl	(	FILE *	fp,
		NGRAM_INFO *	ndata
	)			[static]

Read reverse 2-gram data from RL 3-gram file, and set RL 2-gram probabilities and back-off values for RL 3-gram to the corresponding LR 2-gram data.

引数:

	fp	[in] file pointer
	ndata	[i/o] N-gram to set the read data.

ngram_read_arpa.c の 337 行で定義されています。

参照元 ngram_read_arpa().

static void set_trigram	(	FILE *	fp,
		NGRAM_INFO *	ndata
	)			[static]

Read reverse 3-gram data from RL 3-gram file and store them.

引数:

	fp	[in] file pointer
	ndata	[i/o] N-gram to set the read data.

ngram_read_arpa.c の 374 行で定義されています。

参照元 ngram_read_arpa().

boolean ngram_read_arpa	(	FILE *	fp,
		NGRAM_INFO *	ndata,
		int	direction
	)

Read in one ARPA N-gram file, either LR 2-gram or RL 3-gram.

引数:

	fp	[in] file pointer
	ndata	[out] N-gram data to store the read data
	direction	[in] specify whether this is LR 2-gram or RL 3-gram

戻り値:

TRUE on success, FALSE on failure.

ngram_read_arpa.c の 518 行で定義されています。

参照元 init_ngram_arpa().

void ngram_compact_bigram_context

(

NGRAM_INFO *

ndata

)

Compact the 2-gram context information.

引数:

ndata

[i/o] N-gram data

ngram_read_arpa.c の 630 行で定義されています。

参照元 ngram_read_arpa(), と ngram_read_bin().