fix: 堆炸

This commit is contained in:
camera-2018
2023-04-25 19:49:04 +08:00
parent 7b8e8a23f4
commit 78828f70da
7 changed files with 187 additions and 48 deletions

View File

@@ -55,9 +55,9 @@ one-hot向量的维度是词汇表的大小500000
## dense word vectors表达单词
如果我们可以使用某种方法为每个单词构建一个合适的dense vector如下图那么通过点积等数学计算就可以获得单词之间的某种联系
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片1.png" alt="在这里插入图片描述" style="zoom:50%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片1.png" />
# Word2vec
@@ -70,9 +70,9 @@ one-hot向量的维度是词汇表的大小500000
我们先引入上下文context的概念当单词 w 出现在文本中时,其**上下文context**是出现在w附近的一组单词在固定大小的窗口内如下图
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片2.png" alt="在这里插入图片描述" style="zoom:60%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片2.png" />
这些上下文单词context words决定了banking的意义
@@ -96,15 +96,15 @@ Word2vec包含两个模型**Skip-gram与CBOW**。下面,我们先讲**Skip-
下图展示了以“into”为中心词窗口大小为2的情况下它的上下文词。以及相对应的$P(o|c)$
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片3.png" alt="在这里插入图片描述" style="zoom:50%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片3.png" />
我们滑动窗口再以banking为中心词
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片4.png"在这里插入图片描述" style="zoom:50%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片4.png"在这里插入图片描述" />
那么,如果我们在整个语料库上不断地滑动窗口,我们可以得到所有位置的$P(o|c)$,我们希望在所有位置上**最大化单词o在单词c周围出现了这一事实**,由极大似然法,可得:
@@ -114,15 +114,15 @@ $$
此式还可以依图3写为
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片5.png" alt="在这里插入图片描述" style="zoom:50%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片5.png" />
加log,加负号,缩放大小可得:
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片7.png" alt="在这里插入图片描述" style="zoom:50%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片7.png" />
上式即为**skip-gram的损失函数**,最小化损失函数,就可以得到合适的词向量
@@ -140,9 +140,9 @@ $$
又P(o|c)是一个概率,所以我们在整个语料库上使用**softmax**将点积的值映射到概率如图6
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片6.png" alt="在这里插入图片描述" style="zoom:70%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片6.png" />
注:注意到上图,中心词词向量为$v_{c}$,而上下文词词向量为$u_{o}$。也就是说每个词会对应两个词向量,**在词w做中心词时使用$v_{w}$作为词向量,而在它做上下文词时,使用$u_{w}$作为词向量**。这样做的原因是为了求导等操作时计算上的简便。当整个模型训练完成后,我们既可以使用$v_{w}$作为词w的词向量也可以使用$u_{w}$作为词w的词向量亦或是将二者平均。在下一部分的模型结构中我们将更清楚地看到两个词向量究竟在模型的哪个位置。
@@ -152,9 +152,9 @@ $$
## Word2vec模型结构
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片8.png" alt="在这里插入图片描述" style="zoom:50%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片8.png" />
如图八所示这是一个输入为1 X V维的one-hot向量V为整个词汇表的长度这个向量只有一个1值其余为0值表示一个词单隐藏层**隐藏层的维度为N这里是一个超参数这个参数由我们定义也就是词向量的维度**输出为1 X V维的softmax层的模型。
@@ -174,15 +174,15 @@ $W^{I}$为V X N的参数矩阵$W^{O}$为N X V的参数矩阵。
如上文所述Skip-gram为给定中心词预测周围的词即求P(o|c),如下图所示:
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片image-20220424105817437.png" alt="在这里插入图片描述" style="zoom:50%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片image-20220424105817437.png" />
而CBOW为给定周围的词预测中心词即求P(c|o),如下图所示:
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片10.png" alt="在这里插入图片描述" style="zoom:50%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片10.png" />
注意在使用CBOW时上文所给出的模型结构并没有变在这里我们输入多个上下文词o在隐藏层**将这多个上下文词经过第一个参数矩阵的计算得到的词向量相加作为隐藏单元的值**。其余均不变,$W^{O}$中的每一列依然为为词汇表中的每一个单词的词向量u。
@@ -193,9 +193,9 @@ $W^{I}$为V X N的参数矩阵$W^{O}$为N X V的参数矩阵。
我们再看一眼通过softmax得到的$P(o|c)$,如图:
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片image-20220424105958191.png" alt="在这里插入图片描述" style="zoom:50%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片image-20220424105958191.png" />
可以看到,$P(o|c)$的分母需要在整个单词表上做乘积和exp运算这无疑是非常消耗计算资源的Word2vec的作者针对这个问题做出了改进。
@@ -208,9 +208,9 @@ $W^{I}$为V X N的参数矩阵$W^{O}$为N X V的参数矩阵。
我们首先给出负采样的损失函数:
<div align=center>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片12.png" alt="在这里插入图片描述" style="zoom:40%;" />
</div>
<img src="http://ryluo.oss-cn-chengdu.aliyuncs.com/图片12.png" />
其中$\sigma$为sigmoid函数$1/(1+e^{-x})$, $u_{o}$为实际样本中的上下文词的词向量,而$u_{k}$为我们在单词表中随机选出按一定的规则随机选出具体可参阅参考资料的K个单词。

View File

@@ -1,5 +1,3 @@
# Graph Convolutional Neural Networks for Web-Scale Recommender Systems
该论文是斯坦福大学和Pinterest公司与2018年联合发表与KDD上的一篇关于GCN成功应用于工业级推荐系统的工作。该论文提到的PinSage模型是在GraphSAGE的理论基础进行了更改以适用于实际的工业场景。下面将简单介绍一下GraphSAGE的原理以及Pinsage的核心和细节。
@@ -128,7 +126,7 @@ PinSage在训练时采用的是 Margin Hinge Loss 损失函数,主要的思想
- easy 负样本这里对于mini-batch内的所有pair(训练样本对)会共享500负样本这500个样本从batch之外的所有节点中随机采样得到。这么做可以减少在每个mini-batch中因计算所有节点的embedding所需的时间文中指出这和为每个item采样一定数量负样本无差异。
- hard 负样本这里使用hard 负样本的原因是根据实际场景的问题出发模型需要从20亿的物品item集合中识别出最相似的1000个即模型需要从2百万 item 中识别出最相似的那一个 item。也就是说模型的区分能力不够细致为了解决这个问题加入了一些hard样本。对于hard 负样本,应该是与 q 相似 以及和 i 不相似的物品,具体地的生成方式是将图上的节点计算相对节点 q 的个性化PageRank分值根据分值的排序随机从2000~5000的位置选取节点作为负样本。
负样本的构建是召回模型的中关键的内容,在各家公司的工作都予以体现,具体的大家可以参考 Facebook 发表的[《Embedding-based Retrieval in Facebook Search》]([https://arxiv.org/pdf/2006.11632v1.pdf](https://links.jianshu.com/go?to=https%3A%2F%2Farxiv.org%2Fpdf%2F2006.11632v1.pdf))
负样本的构建是召回模型的中关键的内容,在各家公司的工作都予以体现,具体的大家可以参考 Facebook 发表的[《Embedding-based Retrieval in Facebook Search》](https://arxiv.org/pdf/2006.11632v1.pdf)
**渐进式训练(Curriculum training)**

View File

@@ -150,6 +150,6 @@ def DCN(linear_feature_columns, dnn_feature_columns):
**参考资料**
* 《深度学习推荐系统》 --- 王喆
* [Deep&Cross模型原论文](https://arxiv.org/abs/1708.05123)
* [AI上推荐 之 Wide&Deep与Deep&Cross模型(记忆与泛化并存的华丽转身)]()
* [Wide&Deep模型的进阶---Cross&Deep模型](https://mp.weixin.qq.com/s/DkoaMaXhlgQv1NhZHF-7og)
* AI上推荐 之 Wide&Deep与Deep&Cross模型记忆与泛化并存的华丽转身)
* [Wide&Deep模型的进阶---Cross&Deep模型](https://mp.weixin.qq.com/s/DkoaMaXhlgQv1NhZHF-7og)

View File

@@ -143,5 +143,5 @@ class FM(Layer):
**参考资料**
* [FM推荐算法中的瑞士军刀](https://zhuanlan.zhihu.com/p/343174108)
* [FM算法解析](https://zhuanlan.zhihu.com/p/37963267)
* [FM论文原文]([https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf](https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf))
* [FM论文原文](https://www.csie.ntu.edu.tw/~b97053/paper/Rendle2010FM.pdf)
* [AI上推荐 之 FM和FFM](https://blog.csdn.net/wuzhongqiang/article/details/108719417)

View File

@@ -162,7 +162,7 @@ def DIEN(feature_columns, behavior_feature_list, behavior_seq_feature_list, neg_
> 下面这个图失效了
<div align=center>
<img src="C:\Users\ryl\Desktop\DW_Rec\组队学习内容\代码\imgs\DIEN.png" alt="image-20210218180755462" style="zoom:70%;" />
此处无图
</div>
## 思考

142
package-lock.json generated
View File

@@ -1,5 +1,5 @@
{
"name": "srt-wiki",
"name": "hdu-cs-wiki",
"lockfileVersion": 2,
"requires": true,
"packages": {
@@ -12,6 +12,7 @@
"markdown-it": "^13.0.1"
},
"devDependencies": {
"cross-env": "^7.0.3",
"markdown-it-mathjax3": "^4.3.2",
"vitepress": "^1.0.0-alpha.73"
}
@@ -1722,6 +1723,38 @@
"integrity": "sha512-nXBEVpmUnNRhz83cHd9JRQC52cTMcuXAmR56+9dSMpRdpeA4I1PX6yjmhd71Eyc/wXNsdBdUDIj1QTIeZpU5Tg==",
"hasInstallScript": true
},
"node_modules/cross-env": {
"version": "7.0.3",
"resolved": "https://registry.npmmirror.com/cross-env/-/cross-env-7.0.3.tgz",
"integrity": "sha512-+/HKd6EgcQCJGh2PSjZuUitQBQynKor4wrFbRg4DtAgS1aWO+gU52xpH7M9ScGgXSYmAVS9bIJ8EzuaGw0oNAw==",
"dev": true,
"dependencies": {
"cross-spawn": "^7.0.1"
},
"bin": {
"cross-env": "src/bin/cross-env.js",
"cross-env-shell": "src/bin/cross-env-shell.js"
},
"engines": {
"node": ">=10.14",
"npm": ">=6",
"yarn": ">=1"
}
},
"node_modules/cross-spawn": {
"version": "7.0.3",
"resolved": "https://registry.npmmirror.com/cross-spawn/-/cross-spawn-7.0.3.tgz",
"integrity": "sha512-iRDPJKUPVEND7dHPO8rkbOnPpyDygcDFtWjpeWNCgy8WP2rXcxXL8TskReQl6OrB2G7+UJrags1q15Fudc7G6w==",
"dev": true,
"dependencies": {
"path-key": "^3.1.0",
"shebang-command": "^2.0.0",
"which": "^2.0.1"
},
"engines": {
"node": ">= 8"
}
},
"node_modules/crypto": {
"version": "1.0.1",
"resolved": "https://registry.npmmirror.com/crypto/-/crypto-1.0.1.tgz",
@@ -2117,6 +2150,12 @@
"node": ">= 0.4"
}
},
"node_modules/isexe": {
"version": "2.0.0",
"resolved": "https://registry.npmmirror.com/isexe/-/isexe-2.0.0.tgz",
"integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==",
"dev": true
},
"node_modules/isomorphic.js": {
"version": "0.2.5",
"resolved": "https://registry.npmmirror.com/isomorphic.js/-/isomorphic.js-0.2.5.tgz",
@@ -2617,6 +2656,15 @@
"resolved": "https://registry.npmmirror.com/path-browserify/-/path-browserify-1.0.1.tgz",
"integrity": "sha512-b7uo2UCUOYZcnF/3ID0lulOJi/bafxa1xPe7ZPsammBSpjSWQkjNxlt635YGS2MiR9GjvuXCtz2emr3jbsz98g=="
},
"node_modules/path-key": {
"version": "3.1.1",
"resolved": "https://registry.npmmirror.com/path-key/-/path-key-3.1.1.tgz",
"integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==",
"dev": true,
"engines": {
"node": ">=8"
}
},
"node_modules/picocolors": {
"version": "1.0.0",
"resolved": "https://registry.npmmirror.com/picocolors/-/picocolors-1.0.0.tgz",
@@ -2833,6 +2881,27 @@
"object-assign": "^4.1.1"
}
},
"node_modules/shebang-command": {
"version": "2.0.0",
"resolved": "https://registry.npmmirror.com/shebang-command/-/shebang-command-2.0.0.tgz",
"integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==",
"dev": true,
"dependencies": {
"shebang-regex": "^3.0.0"
},
"engines": {
"node": ">=8"
}
},
"node_modules/shebang-regex": {
"version": "3.0.0",
"resolved": "https://registry.npmmirror.com/shebang-regex/-/shebang-regex-3.0.0.tgz",
"integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==",
"dev": true,
"engines": {
"node": ">=8"
}
},
"node_modules/shiki": {
"version": "0.14.1",
"resolved": "https://registry.npmmirror.com/shiki/-/shiki-0.14.1.tgz",
@@ -3191,6 +3260,21 @@
"webidl-conversions": "^3.0.0"
}
},
"node_modules/which": {
"version": "2.0.2",
"resolved": "https://registry.npmmirror.com/which/-/which-2.0.2.tgz",
"integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==",
"dev": true,
"dependencies": {
"isexe": "^2.0.0"
},
"bin": {
"node-which": "bin/node-which"
},
"engines": {
"node": ">= 8"
}
},
"node_modules/wicked-good-xpath": {
"version": "1.3.0",
"resolved": "https://registry.npmmirror.com/wicked-good-xpath/-/wicked-good-xpath-1.3.0.tgz",
@@ -4714,6 +4798,26 @@
"resolved": "https://registry.npmmirror.com/core-js-pure/-/core-js-pure-3.30.1.tgz",
"integrity": "sha512-nXBEVpmUnNRhz83cHd9JRQC52cTMcuXAmR56+9dSMpRdpeA4I1PX6yjmhd71Eyc/wXNsdBdUDIj1QTIeZpU5Tg=="
},
"cross-env": {
"version": "7.0.3",
"resolved": "https://registry.npmmirror.com/cross-env/-/cross-env-7.0.3.tgz",
"integrity": "sha512-+/HKd6EgcQCJGh2PSjZuUitQBQynKor4wrFbRg4DtAgS1aWO+gU52xpH7M9ScGgXSYmAVS9bIJ8EzuaGw0oNAw==",
"dev": true,
"requires": {
"cross-spawn": "^7.0.1"
}
},
"cross-spawn": {
"version": "7.0.3",
"resolved": "https://registry.npmmirror.com/cross-spawn/-/cross-spawn-7.0.3.tgz",
"integrity": "sha512-iRDPJKUPVEND7dHPO8rkbOnPpyDygcDFtWjpeWNCgy8WP2rXcxXL8TskReQl6OrB2G7+UJrags1q15Fudc7G6w==",
"dev": true,
"requires": {
"path-key": "^3.1.0",
"shebang-command": "^2.0.0",
"which": "^2.0.1"
}
},
"crypto": {
"version": "1.0.1",
"resolved": "https://registry.npmmirror.com/crypto/-/crypto-1.0.1.tgz",
@@ -5043,6 +5147,12 @@
"has-tostringtag": "^1.0.0"
}
},
"isexe": {
"version": "2.0.0",
"resolved": "https://registry.npmmirror.com/isexe/-/isexe-2.0.0.tgz",
"integrity": "sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==",
"dev": true
},
"isomorphic.js": {
"version": "0.2.5",
"resolved": "https://registry.npmmirror.com/isomorphic.js/-/isomorphic.js-0.2.5.tgz",
@@ -5440,6 +5550,12 @@
"resolved": "https://registry.npmmirror.com/path-browserify/-/path-browserify-1.0.1.tgz",
"integrity": "sha512-b7uo2UCUOYZcnF/3ID0lulOJi/bafxa1xPe7ZPsammBSpjSWQkjNxlt635YGS2MiR9GjvuXCtz2emr3jbsz98g=="
},
"path-key": {
"version": "3.1.1",
"resolved": "https://registry.npmmirror.com/path-key/-/path-key-3.1.1.tgz",
"integrity": "sha512-ojmeN0qd+y0jszEtoY48r0Peq5dwMEkIlCOu6Q5f41lfkswXuKtYrhgoTpLnyIcHm24Uhqx+5Tqm2InSwLhE6Q==",
"dev": true
},
"picocolors": {
"version": "1.0.0",
"resolved": "https://registry.npmmirror.com/picocolors/-/picocolors-1.0.0.tgz",
@@ -5619,6 +5735,21 @@
"object-assign": "^4.1.1"
}
},
"shebang-command": {
"version": "2.0.0",
"resolved": "https://registry.npmmirror.com/shebang-command/-/shebang-command-2.0.0.tgz",
"integrity": "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA==",
"dev": true,
"requires": {
"shebang-regex": "^3.0.0"
}
},
"shebang-regex": {
"version": "3.0.0",
"resolved": "https://registry.npmmirror.com/shebang-regex/-/shebang-regex-3.0.0.tgz",
"integrity": "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A==",
"dev": true
},
"shiki": {
"version": "0.14.1",
"resolved": "https://registry.npmmirror.com/shiki/-/shiki-0.14.1.tgz",
@@ -5921,6 +6052,15 @@
"webidl-conversions": "^3.0.0"
}
},
"which": {
"version": "2.0.2",
"resolved": "https://registry.npmmirror.com/which/-/which-2.0.2.tgz",
"integrity": "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA==",
"dev": true,
"requires": {
"isexe": "^2.0.0"
}
},
"wicked-good-xpath": {
"version": "1.3.0",
"resolved": "https://registry.npmmirror.com/wicked-good-xpath/-/wicked-good-xpath-1.3.0.tgz",

View File

@@ -1,8 +1,8 @@
{
"scripts": {
"docs:dev": "vitepress dev",
"docs:build": "vitepress build",
"docs:preview": "vitepress preview"
"docs:dev": "cross-env NODE_OPTIONS='--max-old-space-size=8192' vitepress dev",
"docs:build": "cross-env NODE_OPTIONS='--max-old-space-size=8192' vitepress build",
"docs:preview": "cross-env NODE_OPTIONS='--max-old-space-size=8192' vitepress preview"
},
"dependencies": {
"@codemirror/lang-python": "^6.1.2",
@@ -12,6 +12,7 @@
"markdown-it": "^13.0.1"
},
"devDependencies": {
"cross-env": "^7.0.3",
"markdown-it-mathjax3": "^4.3.2",
"vitepress": "^1.0.0-alpha.73"
}