Gensim word2vec 模型输出 1000 维 ndarray 但 ndarray 维度的最大数量为 32 – 如何？

Gensim word2vec model outputs 1000 dimension ndarray but the maximum number of ndarray dimensions is 32 - how?

我正在尝试使用这个 1000 维维基百科 word2vec 模型来分析一些文档。

使用自省我发现一个单词的向量表示是一个 1000 维的 numpy.ndarray，但是每当我尝试创建一个 ndarray 来查找最近的单词时，我都会得到一个值错误：

ValueError: maximum supported dimension for an ndarray is 32, found 1000
doc = [model[word] for word in text if word in model.vocab]

out = []

n = len(doc[0])

print(n)

print(len(model["hello"]))

print(type(doc[0]))

for i in range(n):

  sum = 0

  for d in doc:

    sum += d[i]

  out.append(sum/n)

out = np.ndarray(out)

1000

1000

<class 'numpy.ndarray'>

ValueError: maximum supported dimension for an ndarray is 32, found 1000
result = np.ndarray(len(d))

for i in range(len(d)):

  result[i] = d[i]
result = np.ndarray(len(d))

result[:] = d # same result as previous 3-lines w/ loop
result = np.array(d) # same result as previous 2-lines

# get a list of available word-vetors

doc = [model[word] for word in text if word in model.vocab]

# average all those vectors

out = np.mean(doc, axis=0)

我可以通过在线查看 32 确实是 ndarray 支持的最大维数 - 那么给出了什么？ gensim 如何能够输出 1000 维的 ndarray？

这里是一些示例代码：

ValueError: maximum supported dimension for an ndarray is 32, found 1000
doc = [model[word] for word in text if word in model.vocab]

out = []

n = len(doc[0])

print(n)

print(len(model["hello"]))

print(type(doc[0]))

for i in range(n):

  sum = 0

  for d in doc:

    sum += d[i]

  out.append(sum/n)

out = np.ndarray(out)

1000

1000

<class 'numpy.ndarray'>

ValueError: maximum supported dimension for an ndarray is 32, found 1000
result = np.ndarray(len(d))

for i in range(len(d)):

  result[i] = d[i]
result = np.ndarray(len(d))

result[:] = d # same result as previous 3-lines w/ loop
result = np.array(d) # same result as previous 2-lines

# get a list of available word-vetors

doc = [model[word] for word in text if word in model.vocab]

# average all those vectors

out = np.mean(doc, axis=0)

输出：

ValueError: maximum supported dimension for an ndarray is 32, found 1000
doc = [model[word] for word in text if word in model.vocab]

out = []

n = len(doc[0])

print(n)

print(len(model["hello"]))

print(type(doc[0]))

for i in range(n):

  sum = 0

  for d in doc:

    sum += d[i]

  out.append(sum/n)

out = np.ndarray(out)

1000

1000

<class 'numpy.ndarray'>

ValueError: maximum supported dimension for an ndarray is 32, found 1000
result = np.ndarray(len(d))

for i in range(len(d)):

  result[i] = d[i]
result = np.ndarray(len(d))

result[:] = d # same result as previous 3-lines w/ loop
result = np.array(d) # same result as previous 2-lines

# get a list of available word-vetors

doc = [model[word] for word in text if word in model.vocab]

# average all those vectors

out = np.mean(doc, axis=0)

这里的目标是计算语料库中所有单词的平均向量，其格式可用于在模型中查找附近的单词，因此欢迎提出任何替代建议。

您正在调用 numpy 的 ndarray() 构造函数，其中包含一个包含 1000 个数字的列表——您手动计算的 1000 个维度中每个维度的平均值。

ndarray() 函数期望它的参数是构造的矩阵的形状，所以它试图创建一个形状为 (d[0], d[1], ..., d[999]) 的新矩阵——然后该矩阵中的每个单独的值都将用1000-int 坐标集。而且，确实 numpy 数组只能有 32 个独立维度。

但是，即使您将提供给 ndarray() 的列表减少到只有 32 个数字，您仍然会遇到问题，因为您的 32 个数字是浮点值，而 ndarray() 期望整数计数。 (你会得到一个 TypeError。)

沿着您尝试采用的方法——这不是我们将在下面介绍的最佳方法——你真的想创建一个具有 1000 个浮点维度的单个向量。也就是说，1000 个类似单元格的值 - 不是 d[0] * d[1] * ... * d[999] 单独的类似单元格的值。

因此，按照您最初的方法的粗略解决方法可能是将您的最后一行替换为：

ValueError: maximum supported dimension for an ndarray is 32, found 1000
doc = [model[word] for word in text if word in model.vocab]

out = []

n = len(doc[0])

print(n)

print(len(model["hello"]))

print(type(doc[0]))

for i in range(n):

  sum = 0

  for d in doc:

    sum += d[i]

  out.append(sum/n)

out = np.ndarray(out)

1000

1000

<class 'numpy.ndarray'>

ValueError: maximum supported dimension for an ndarray is 32, found 1000
result = np.ndarray(len(d))

for i in range(len(d)):

  result[i] = d[i]
result = np.ndarray(len(d))

result[:] = d # same result as previous 3-lines w/ loop
result = np.array(d) # same result as previous 2-lines

# get a list of available word-vetors

doc = [model[word] for word in text if word in model.vocab]

# average all those vectors

out = np.mean(doc, axis=0)

但是有很多方法可以逐步提高效率、紧凑性和惯用性——我将在下面提到其中的一些方法，尽管最好的方法从根本上来说是不必要的。

首先，您可以使用 Python 的括号索引赋值选项，而不是上面代码中的赋值循环：

ValueError: maximum supported dimension for an ndarray is 32, found 1000
doc = [model[word] for word in text if word in model.vocab]

out = []

n = len(doc[0])

print(n)

print(len(model["hello"]))

print(type(doc[0]))

for i in range(n):

  sum = 0

  for d in doc:

    sum += d[i]

  out.append(sum/n)

out = np.ndarray(out)

1000

1000

<class 'numpy.ndarray'>

ValueError: maximum supported dimension for an ndarray is 32, found 1000
result = np.ndarray(len(d))

for i in range(len(d)):

  result[i] = d[i]
result = np.ndarray(len(d))

result[:] = d # same result as previous 3-lines w/ loop
result = np.array(d) # same result as previous 2-lines

# get a list of available word-vetors

doc = [model[word] for word in text if word in model.vocab]

# average all those vectors

out = np.mean(doc, axis=0)

但实际上，numpy\\的array()函数本质上可以从给定的列表中创建必要的numpy-native ndarray，所以完全不用ndarray()，你可以只使用array():

ValueError: maximum supported dimension for an ndarray is 32, found 1000
doc = [model[word] for word in text if word in model.vocab]

out = []

n = len(doc[0])

print(n)

print(len(model["hello"]))

print(type(doc[0]))

for i in range(n):

  sum = 0

  for d in doc:

    sum += d[i]

  out.append(sum/n)

out = np.ndarray(out)

1000

1000

<class 'numpy.ndarray'>

ValueError: maximum supported dimension for an ndarray is 32, found 1000
result = np.ndarray(len(d))

for i in range(len(d)):

  result[i] = d[i]
result = np.ndarray(len(d))

result[:] = d # same result as previous 3-lines w/ loop
result = np.array(d) # same result as previous 2-lines

# get a list of available word-vetors

doc = [model[word] for word in text if word in model.vocab]

# average all those vectors

out = np.mean(doc, axis=0)

但进一步，numpy 的许多原生处理数组(和类似数组的列表)的函数已经包括在一个步骤中执行多个向量的平均值的事情(甚至循环都隐藏在里面非常高效的编译代码或 CPU 大容量向量操作)。例如，有一个 mean() 函数可以平均数字列表、数字的多维数组或对齐的向量集等等。

这允许更快、更清晰、单行的方法，可以将整个原始代码替换为以下内容：

ValueError: maximum supported dimension for an ndarray is 32, found 1000
doc = [model[word] for word in text if word in model.vocab]

out = []

n = len(doc[0])

print(n)

print(len(model["hello"]))

print(type(doc[0]))

for i in range(n):

  sum = 0

  for d in doc:

    sum += d[i]

  out.append(sum/n)

out = np.ndarray(out)

1000

1000

<class 'numpy.ndarray'>

ValueError: maximum supported dimension for an ndarray is 32, found 1000
result = np.ndarray(len(d))

for i in range(len(d)):

  result[i] = d[i]
result = np.ndarray(len(d))

result[:] = d # same result as previous 3-lines w/ loop
result = np.array(d) # same result as previous 2-lines

# get a list of available word-vetors

doc = [model[word] for word in text if word in model.vocab]

# average all those vectors

out = np.mean(doc, axis=0)

(如果没有 axis 参数，它会将所有插槽中的所有单个维度值平均在一起，形成一个最终的平均数。)