Skip to content
项目
群组
代码片段
帮助
当前项目
正在载入...
登录 / 注册
切换导航面板
M
mtbookv2
概览
Overview
Details
Activity
Cycle Analytics
版本库
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
问题
0
Issues
0
列表
Board
标记
里程碑
合并请求
0
Merge Requests
0
CI / CD
CI / CD
流水线
作业
日程表
图表
维基
Wiki
代码片段
Snippets
成员
Collapse sidebar
Close sidebar
活动
图像
聊天
创建新问题
作业
提交
Issue Boards
Open sidebar
NiuTrans
mtbookv2
Commits
007176c8
Commit
007176c8
authored
Nov 09, 2020
by
曹润柘
Browse files
Options
Browse Files
Download
Plain Diff
合并分支 'caorunzhe' 到 'master'
Caorunzhe 查看合并请求
!384
parents
9a1eeda6
3557eb23
全部展开
隐藏空白字符变更
内嵌
并排
正在显示
6 个修改的文件
包含
90 行增加
和
90 行删除
+90
-90
Chapter10/chapter10.tex
+1
-1
Chapter11/Figures/figure-use-cnn-in-nmt.tex
+38
-38
Chapter11/Figures/figure-use-cnn-in-sentence-classification.tex
+49
-49
Chapter11/chapter11.tex
+1
-1
Chapter12/chapter12.tex
+1
-1
bibliography.bib
+0
-0
没有找到文件。
Chapter10/chapter10.tex
查看文件 @
007176c8
...
...
@@ -1257,7 +1257,7 @@ L(\mathbi{Y},\widehat{\mathbi{Y}}) = \sum_{j=1}^n L_{\textrm{ce}}(\mathbi{y}_j,\
\vspace
{
0.5em
}
\item
注意力机制的使用是机器翻译乃至整个自然语言处理近几年获得成功的重要因素之一
\upcite
{
bahdanau2014neural,DBLP:journals/corr/LuongPM15
}
。早期，有研究者尝试将注意力机制和统计机器翻译的词对齐进行统一
\upcite
{
WangNeural,He2016ImprovedNM,li-etal-2019-word
}
。最近，也有大量的研究工作对注意力机制进行改进，比如，使用自注意力机制构建翻译模型等
\upcite
{
vaswani2017attention
}
。而对注意力模型的改进也成为了自然语言处理中的热点问题之一。在
{
\chapterfifteen
}
会对机器翻译中不同注意力模型进行进一步讨论。
\vspace
{
0.5em
}
\item
一般来说，神经机器翻译的计算过程是没有人工干预的，翻译流程也无法用人类的知识直接进行解释，因此一个有趣的方向是在神经机器翻译中引入先验知识，使得机器翻译的行为更“像”人。比如，可以使用句法树来引入人类的语言学知识
\upcite
{
Yang2017TowardsBH,Wang2019TreeTI
}
，基于句法的神经机器翻译也包含大量的树结构的神经网络建模
\upcite
{
DBLP:journals/corr/abs-1809-01854,DBLP:journals/corr/abs-1808-09374
}
。此外，也可以把用户定义的词典或者翻译记忆加入到翻译过程来
\upcite
{
DBLP:journals/corr/ZhangZ16c,zhang-etal-2017-prior,duan-etal-2020-bilingual,cao-xiong-2018-encoding
}
，使得用户的约束可以直接反映到机器翻译的结果上来。先验知识的种类还有很多，包括词对齐
\upcite
{
li-etal-2019-word
}
、 篇章信息
\upcite
{
Werlen2018DocumentLevelNM,DBLP:journals/corr/abs-1805-10163
}
等等，都是神经机器翻译中能够使用的信息。
\item
一般来说，神经机器翻译的计算过程是没有人工干预的，翻译流程也无法用人类的知识直接进行解释，因此一个有趣的方向是在神经机器翻译中引入先验知识，使得机器翻译的行为更“像”人。比如，可以使用句法树来引入人类的语言学知识
\upcite
{
Yang2017TowardsBH,Wang2019TreeTI
}
，基于句法的神经机器翻译也包含大量的树结构的神经网络建模
\upcite
{
DBLP:journals/corr/abs-1809-01854,DBLP:journals/corr/abs-1808-09374
}
。此外，也可以把用户定义的词典或者翻译记忆加入到翻译过程来
\upcite
{
DBLP:journals/corr/ZhangZ16c,zhang-etal-2017-prior,duan-etal-2020-bilingual,cao-xiong-2018-encoding
}
，使得用户的约束可以直接反映到机器翻译的结果上来。先验知识的种类还有很多，包括词对齐
\upcite
{
li-etal-2019-word
,DBLP:conf/emnlp/MiWI16,DBLP:conf/coling/LiuUFS16
}
、 篇章信息
\upcite
{
Werlen2018DocumentLevelNM,DBLP:journals/corr/abs-1805-10163,DBLP:conf/acl/LiLWJXZLL20
}
等等，都是神经机器翻译中能够使用的信息。
\end{itemize}
Chapter11/Figures/figure-use-cnn-in-nmt.tex
查看文件 @
007176c8
%\newlength{\bc}
\setlength
{
\bc
}{
0.4cm
}
%\newlength{\bc
c
}
\setlength
{
\bc
c
}{
0.4cm
}
\begin{tikzpicture}
\begin{scope}
%\tikzstyle{every node}=[scale=0.8]
\tikzstyle
{
line
}
= [dash pattern=on 2pt off 1pt,line width=0.5pt]
\tikzstyle
{
cir
}
= [thin,fill=blue!8,draw,circle,minimum size =0.5em,drop shadow=
{
shadow xshift=0.15em, shadow yshift=-0.1em
}
]
\tikzstyle
{
word
}
= [inner sep=0pt, font=
\scriptsize
,minimum height=
\bc
]
\tikzstyle
{
word
}
= [inner sep=0pt, font=
\scriptsize
,minimum height=
\bc
c
]
\draw
[fill=red!8,line width=0.2pt]
(0cm,0cm+1*
\bc
) rectangle (0cm+4*
\bc
,0cm+7*
\b
c
);
\draw
[fill=cyan!14,line width=0.2pt]
(0cm,0cm) rectangle (0cm+4*
\bc
,0cm+1*
\b
c
);
\draw
[fill=cyan!14,line width=0.2pt]
(0cm,0cm+7*
\bc
) rectangle (0cm+4*
\bc
,0cm+8*
\b
c
);
\draw
[step=\bc
]
(0cm,0cm) grid (0cm+4*
\bc
,0cm+8*
\b
c
);
%\draw[line width=0.7pt] (0cm,0cm) rectangle (0cm+4*\bc
,0cm+8*\b
c);
\draw
[red!50,line width=1.8pt]
(0cm,0cm+5*
\bc
) rectangle (0cm+4*
\bc
,0cm+8*
\b
c
);
\draw
[ugreen!50,line width=1.8pt]
(0cm,0cm+1*
\bc
) rectangle (0cm+4*
\bc
,0cm+4*
\b
c
);
\draw
[fill=red!8,line width=0.2pt]
(0cm,0cm+1*
\bc
c
) rectangle (0cm+4*
\bcc
,0cm+7*
\bc
c
);
\draw
[fill=cyan!14,line width=0.2pt]
(0cm,0cm) rectangle (0cm+4*
\bc
c
,0cm+1*
\bc
c
);
\draw
[fill=cyan!14,line width=0.2pt]
(0cm,0cm+7*
\bc
c
) rectangle (0cm+4*
\bcc
,0cm+8*
\bc
c
);
\draw
[step=\bc
c]
(0cm,0cm) grid (0cm+4*
\bcc
,0cm+8*
\bc
c
);
%\draw[line width=0.7pt] (0cm,0cm) rectangle (0cm+4*\bc
c,0cm+8*\bc
c);
\draw
[red!50,line width=1.8pt]
(0cm,0cm+5*
\bc
c
) rectangle (0cm+4*
\bcc
,0cm+8*
\bc
c
);
\draw
[ugreen!50,line width=1.8pt]
(0cm,0cm+1*
\bc
c
) rectangle (0cm+4*
\bcc
,0cm+4*
\bc
c
);
\draw
[fill=blue!8,xshift=5.0cm,yshift=1.0cm,line width=0.2pt]
(0cm,0cm) rectangle (0cm+1*
\bc
,0cm+6*
\b
c
);
\draw
[step=\bc
,xshift=5.0cm,yshift=1.0cm]
(0cm,0cm) grid (0cm+1*
\bc
,0cm+6*
\b
c
);
%\draw[line width=0.7pt,xshift=5.0cm,yshift=1.0cm] (0cm,0cm) rectangle (0cm+1*\bc
,0cm+6*\b
c);
\draw
[fill=blue!8,xshift=5.0cm,yshift=1.0cm,line width=0.2pt]
(0cm,0cm) rectangle (0cm+1*
\bc
c
,0cm+6*
\bc
c
);
\draw
[step=\bc
c,xshift=5.0cm,yshift=1.0cm]
(0cm,0cm) grid (0cm+1*
\bcc
,0cm+6*
\bc
c
);
%\draw[line width=0.7pt,xshift=5.0cm,yshift=1.0cm] (0cm,0cm) rectangle (0cm+1*\bc
c,0cm+6*\bc
c);
\draw
[fill=blue!8,xshift=5.2cm,yshift=0.8cm,line width=0.2pt]
(0cm,0cm) rectangle (0cm+1*
\bc
,0cm+6*
\b
c
);
\draw
[step=\bc
,xshift=5.2cm,yshift=0.8cm]
(0cm,0cm) grid (0cm+1*
\bc
,0cm+6*
\b
c
);
%\draw[line width=0.7pt,xshift=5.2cm,yshift=0.8cm] (0cm,0cm) rectangle (0cm+1*\bc
,0cm+6*\b
c);
\draw
[ugreen!50,line width=2pt,xshift=5.2cm,yshift=0.8cm]
(0cm,0cm+1*
\bc
) rectangle (0cm+1*
\bc
,0cm+2*
\b
c
);
\draw
[fill=blue!8,xshift=5.2cm,yshift=0.8cm,line width=0.2pt]
(0cm,0cm) rectangle (0cm+1*
\bc
c
,0cm+6*
\bc
c
);
\draw
[step=\bc
c,xshift=5.2cm,yshift=0.8cm]
(0cm,0cm) grid (0cm+1*
\bcc
,0cm+6*
\bc
c
);
%\draw[line width=0.7pt,xshift=5.2cm,yshift=0.8cm] (0cm,0cm) rectangle (0cm+1*\bc
c,0cm+6*\bc
c);
\draw
[ugreen!50,line width=2pt,xshift=5.2cm,yshift=0.8cm]
(0cm,0cm+1*
\bc
c
) rectangle (0cm+1*
\bcc
,0cm+2*
\bc
c
);
\draw
[fill=blue!8,xshift=5.4cm,yshift=0.6cm,line width=0.2pt]
(0cm,0cm) rectangle (0cm+1*
\bc
,0cm+6*
\b
c
);
\draw
[step=\bc
,xshift=5.4cm,yshift=0.6cm]
(0cm,0cm) grid (0cm+1*
\bc
,0cm+6*
\b
c
);
%\draw[line width=0.7pt,xshift=5.4cm,yshift=0.6cm] (0cm,0cm) rectangle (0cm+1*\bc
,0cm+6*\b
c);
\draw
[fill=blue!8,xshift=5.4cm,yshift=0.6cm,line width=0.2pt]
(0cm,0cm) rectangle (0cm+1*
\bc
c
,0cm+6*
\bc
c
);
\draw
[step=\bc
c,xshift=5.4cm,yshift=0.6cm]
(0cm,0cm) grid (0cm+1*
\bcc
,0cm+6*
\bc
c
);
%\draw[line width=0.7pt,xshift=5.4cm,yshift=0.6cm] (0cm,0cm) rectangle (0cm+1*\bc
c,0cm+6*\bc
c);
\draw
[fill=blue!8,xshift=5.6cm,yshift=0.4cm,line width=0.2pt]
(0cm,0cm) rectangle (0cm+1*
\bc
,0cm+6*
\b
c
);
\draw
[step=\bc
,xshift=5.6cm,yshift=0.4cm]
(0cm,0cm) grid (0cm+1*
\bc
,0cm+6*
\b
c
);
%\draw[line width=0.7pt,xshift=5.6cm,yshift=0.4cm] (0cm,0cm) rectangle (0cm+1*\bc
,0cm+6*\b
c);
\draw
[red!50,line width=2pt,xshift=5.6cm,yshift=0.4cm]
(0cm,0cm+5*
\bc
) rectangle (0cm+1*
\bc
,0cm+6*
\b
c
);
\draw
[fill=blue!8,xshift=5.6cm,yshift=0.4cm,line width=0.2pt]
(0cm,0cm) rectangle (0cm+1*
\bc
c
,0cm+6*
\bc
c
);
\draw
[step=\bc
c,xshift=5.6cm,yshift=0.4cm]
(0cm,0cm) grid (0cm+1*
\bcc
,0cm+6*
\bc
c
);
%\draw[line width=0.7pt,xshift=5.6cm,yshift=0.4cm] (0cm,0cm) rectangle (0cm+1*\bc
c,0cm+6*\bc
c);
\draw
[red!50,line width=2pt,xshift=5.6cm,yshift=0.4cm]
(0cm,0cm+5*
\bc
c
) rectangle (0cm+1*
\bcc
,0cm+6*
\bc
c
);
\draw
[red!50,line width=0.5pt]
(0cm+4*
\bc
,0cm+8*
\bc
) -- ([xshift=5.6cm,yshift=0.4cm]0cm,0cm+6*
\b
c
);
\draw
[red!50,line width=0.5pt]
(0cm+4*
\bc
,0cm+5*
\bc
) -- ([xshift=5.6cm,yshift=0.4cm]0cm,0cm+5*
\b
c
);
\draw
[red!50,line width=0.5pt]
(0cm+4*
\bc
c
,0cm+8*
\bcc
) -- ([xshift=5.6cm,yshift=0.4cm]0cm,0cm+6*
\bc
c
);
\draw
[red!50,line width=0.5pt]
(0cm+4*
\bc
c
,0cm+5*
\bcc
) -- ([xshift=5.6cm,yshift=0.4cm]0cm,0cm+5*
\bc
c
);
\draw
[ugreen!50,line width=0.5pt]
(0cm+4*
\bc
,0cm+4*
\bc
) -- ([xshift=5.2cm,yshift=0.8cm]0cm,0cm+2*
\b
c
);
\draw
[ugreen!50,line width=0.5pt]
(0cm+4*
\bc
,0cm+1*
\bc
) -- ([xshift=5.2cm,yshift=0.8cm]0cm,0cm+1*
\b
c
);
\draw
[ugreen!50,line width=0.5pt]
(0cm+4*
\bc
c
,0cm+4*
\bcc
) -- ([xshift=5.2cm,yshift=0.8cm]0cm,0cm+2*
\bc
c
);
\draw
[ugreen!50,line width=0.5pt]
(0cm+4*
\bc
c
,0cm+1*
\bcc
) -- ([xshift=5.2cm,yshift=0.8cm]0cm,0cm+1*
\bc
c
);
\node
[word]
(w1) at (-0.5cm, 3.0cm)
{$
<
$
p
$
>
$}
;
\node
[word]
(w2) at ([yshift=-
\bc
]w1)
{
今天
}
;
\node
[word]
(w3) at ([yshift=-
\bc
]w2)
{
是
}
;
\node
[word]
(w4) at ([yshift=-
\bc
]w3)
{
个
}
;
\node
[word]
(w5) at ([yshift=-
\bc
]w4)
{
好
}
;
\node
[word]
(w6) at ([yshift=-
\bc
]w5)
{
日子
}
;
\node
[word]
(w7) at ([yshift=-
\bc
]w6)
{
。
}
;
\node
[word]
(w8) at ([yshift=-
\bc
]w7)
{$
<
$
p
$
>
$}
;
\node
[word]
(w2) at ([yshift=-
\bc
c
]w1)
{
今天
}
;
\node
[word]
(w3) at ([yshift=-
\bc
c
]w2)
{
是
}
;
\node
[word]
(w4) at ([yshift=-
\bc
c
]w3)
{
个
}
;
\node
[word]
(w5) at ([yshift=-
\bc
c
]w4)
{
好
}
;
\node
[word]
(w6) at ([yshift=-
\bc
c
]w5)
{
日子
}
;
\node
[word]
(w7) at ([yshift=-
\bc
c
]w6)
{
。
}
;
\node
[word]
(w8) at ([yshift=-
\bc
c
]w7)
{$
<
$
p
$
>
$}
;
\node
[inner xsep=2pt,inner ysep=0pt,font=\footnotesize]
(c1) at (0.8cm, 3.5cm)
{$
O
$}
;
\draw
[-latex]
(c1.west) -- ([xshift=-0.4cm]c1.west);
...
...
@@ -59,9 +59,9 @@
\draw
[-latex]
(c2.east) -- ([xshift=0.2cm, yshift=-0.2cm]c2.east);
%%%%%%%%%%%%%%%%%%%%%
\node
[word]
(sub) at ([xshift=9.5*
\bc
,yshift=3.5*
\b
c
]w1)
{$
O
$
\ \
：输入通道数
}
;
\node
[word]
(sub2) at ([yshift=-0.5*
\bc
]sub.south)
{$
N
$
\ \
：卷积核数量
}
;
\node
[word]
(sub3) at ([xshift=-1.35em,yshift=-0.5*
\bc
]sub2.south)
{$
<
$
p
$
>
$
：填充
}
;
\node
[word]
(sub) at ([xshift=9.5*
\bc
c
,yshift=3.5*
\bc
c
]w1)
{$
O
$
\ \
：输入通道数
}
;
\node
[word]
(sub2) at ([yshift=-0.5*
\bc
c
]sub.south)
{$
N
$
\ \
：卷积核数量
}
;
\node
[word]
(sub3) at ([xshift=-1.35em,yshift=-0.5*
\bc
c
]sub2.south)
{$
<
$
p
$
>
$
：填充
}
;
\end{scope}
...
...
Chapter11/Figures/figure-use-cnn-in-sentence-classification.tex
查看文件 @
007176c8
%\newlength{\bc}
\setlength
{
\bc
}{
0.4cm
}
%\newlength{\bc
c
}
\setlength
{
\bc
c
}{
0.4cm
}
\begin{tikzpicture}
\begin{scope}
%\tikzstyle{every node}=[scale=0.8]
\tikzstyle
{
line
}
= [dash pattern=on 2pt off 1pt,line width=0.6pt]
\tikzstyle
{
cir
}
= [thin,fill=blue!8,draw,circle,minimum size =0.5em,drop shadow=
{
shadow xshift=0.15em, shadow yshift=-0.1em
}
]
\tikzstyle
{
word
}
= [inner sep=0pt, font=
\footnotesize
,minimum height=
\bc
]
\tikzstyle
{
word
}
= [inner sep=0pt, font=
\footnotesize
,minimum height=
\bc
c
]
\draw
[fill=blue!8,xshift=0.3cm,yshift=0.5cm,line width=0.6pt]
(0cm,0cm) rectangle (0cm+6*
\bc
,0cm+9*
\b
c
);
\draw
[ugreen!60,step=\bc
,xshift=0.3cm,yshift=0.5cm,gray]
(0cm,0cm) grid (0cm+6*
\bc
,0cm+9*
\b
c
);
%\draw[line width=0.7pt,xshift=0.3cm,yshift=0.5cm] (0cm,0cm) rectangle (0cm+6*\bc
,0cm+9*\b
c);
\draw
[red!60,line width=2pt,xshift=0.3cm,yshift=0.5cm]
(0cm,0cm+2*
\bc
) rectangle (0cm+6*
\bc
,0cm+4*
\b
c
);
\draw
[fill=blue!8,xshift=0.3cm,yshift=0.5cm,line width=0.6pt]
(0cm,0cm) rectangle (0cm+6*
\bc
c
,0cm+9*
\bc
c
);
\draw
[ugreen!60,step=\bc
c,xshift=0.3cm,yshift=0.5cm,gray]
(0cm,0cm) grid (0cm+6*
\bcc
,0cm+9*
\bc
c
);
%\draw[line width=0.7pt,xshift=0.3cm,yshift=0.5cm] (0cm,0cm) rectangle (0cm+6*\bc
c,0cm+9*\bc
c);
\draw
[red!60,line width=2pt,xshift=0.3cm,yshift=0.5cm]
(0cm,0cm+2*
\bc
c
) rectangle (0cm+6*
\bcc
,0cm+4*
\bc
c
);
\draw
[thick,fill=blue!8,line width=0.6pt]
(0cm,0cm) rectangle (0cm+6*
\bc
,0cm+9*
\b
c
);
\draw
[step=\bc
,gray]
(0cm,0cm) grid (0cm+6*
\bc
,0cm+9*
\b
c
);
%\draw[line width=0.7pt] (0cm,0cm) rectangle (0cm+6*\bc
,0cm+9*\b
c);
\draw
[red!60,line width=2pt]
(0cm,0cm) rectangle (0cm+6*
\bc
,0cm+2*
\b
c
);
\draw
[ugreen!60,line width=2pt]
(0cm,0cm+3*
\bc
) rectangle (0cm+6*
\bc
,0cm+6*
\b
c
);
\draw
[red!60,line width=2pt]
(0cm,0cm+7*
\bc
) rectangle (0cm+6*
\bc
,0cm+9*
\b
c
);
\draw
[thick,fill=blue!8,line width=0.6pt]
(0cm,0cm) rectangle (0cm+6*
\bc
c
,0cm+9*
\bc
c
);
\draw
[step=\bc
c,gray]
(0cm,0cm) grid (0cm+6*
\bcc
,0cm+9*
\bc
c
);
%\draw[line width=0.7pt] (0cm,0cm) rectangle (0cm+6*\bc
c,0cm+9*\bc
c);
\draw
[red!60,line width=2pt]
(0cm,0cm) rectangle (0cm+6*
\bc
c
,0cm+2*
\bc
c
);
\draw
[ugreen!60,line width=2pt]
(0cm,0cm+3*
\bc
c
) rectangle (0cm+6*
\bcc
,0cm+6*
\bc
c
);
\draw
[red!60,line width=2pt]
(0cm,0cm+7*
\bc
c
) rectangle (0cm+6*
\bcc
,0cm+9*
\bc
c
);
\draw
[fill=blue!8,xshift=5.0cm,yshift=1.3cm,line width=0.6pt]
(0cm,0cm) rectangle (0cm+1*
\bc
,0cm+6*
\b
c
);
\draw
[step=\bc
,gray,xshift=5.0cm,yshift=1.3cm]
(0cm,0cm) grid (0cm+1*
\bc
,0cm+6*
\b
c
);
%\draw[xshift=5.0cm,yshift=1.3cm,line width=0.7pt] (0cm,0cm) rectangle (0cm+1*\bc
,0cm+6*\b
c);
\draw
[ugreen!60,line width=2pt,xshift=5.0cm,yshift=1.3cm]
(0cm,0cm+2*
\bc
) rectangle (0cm+1*
\bc
,0cm+3*
\b
c
);
\draw
[fill=blue!8,xshift=5.0cm,yshift=1.3cm,line width=0.6pt]
(0cm,0cm) rectangle (0cm+1*
\bc
c
,0cm+6*
\bc
c
);
\draw
[step=\bc
c,gray,xshift=5.0cm,yshift=1.3cm]
(0cm,0cm) grid (0cm+1*
\bcc
,0cm+6*
\bc
c
);
%\draw[xshift=5.0cm,yshift=1.3cm,line width=0.7pt] (0cm,0cm) rectangle (0cm+1*\bc
c,0cm+6*\bc
c);
\draw
[ugreen!60,line width=2pt,xshift=5.0cm,yshift=1.3cm]
(0cm,0cm+2*
\bc
c
) rectangle (0cm+1*
\bcc
,0cm+3*
\bc
c
);
\draw
[gray,fill=blue!8,line width=0.6pt](8cm,2.6cm) -- (8.4cm, 2.6cm) -- (9cm,1cm) -- (8.6cm, 1cm) -- (8cm,2.6cm);
\draw
[gray](8.15cm,2.2cm) -- (8.55cm,2.2cm);
...
...
@@ -35,48 +35,48 @@
\draw
[gray,fill=blue!8,line width=0.6pt](11cm,2.2cm) -- (11.4cm, 2.2cm) -- (11.7cm,1.4cm) -- (11.3cm, 1.4cm) -- (11cm,2.2cm);
\draw
[gray](11.15cm,1.8cm) -- (11.55cm,1.8cm);
\draw
[ugreen!60,line]
([xshift=5.0cm,yshift=1.3cm]0cm+1*
\bc
,0cm+6*
\b
c
) -- (8cm,2.6cm);
\draw
[ugreen!60,line]
([xshift=5.0cm,yshift=1.3cm]0cm+1*
\bc
,0cm) -- (8.15cm,2.2cm);
\draw
[ugreen!60,line]
([xshift=5.0cm,yshift=1.3cm]0cm+1*
\bc
c
,0cm+6*
\bc
c
) -- (8cm,2.6cm);
\draw
[ugreen!60,line]
([xshift=5.0cm,yshift=1.3cm]0cm+1*
\bc
c
,0cm) -- (8.15cm,2.2cm);
\draw
[fill=blue!8,xshift=5.2cm,yshift=1.0cm,line width=0.6pt]
(0cm,0cm) rectangle (0cm+1*
\bc
,0cm+6*
\b
c
);
\draw
[step=\bc
,gray,xshift=5.2cm,yshift=1.0cm]
(0cm,0cm) grid (0cm+1*
\bc
,0cm+6*
\b
c
);
%\draw[line width=0.7pt,xshift=5.2cm,yshift=1.0cm] (0cm,0cm) rectangle (0cm+1*\bc
,0cm+6*\b
c);
\draw
[fill=blue!8,xshift=5.2cm,yshift=1.0cm,line width=0.6pt]
(0cm,0cm) rectangle (0cm+1*
\bc
c
,0cm+6*
\bc
c
);
\draw
[step=\bc
c,gray,xshift=5.2cm,yshift=1.0cm]
(0cm,0cm) grid (0cm+1*
\bcc
,0cm+6*
\bc
c
);
%\draw[line width=0.7pt,xshift=5.2cm,yshift=1.0cm] (0cm,0cm) rectangle (0cm+1*\bc
c,0cm+6*\bc
c);
\draw
[fill=blue!8,xshift=5.4cm,yshift=0.3cm,line width=0.6pt]
(0cm,0cm) rectangle (0cm+1*
\bc
,0cm+7*
\b
c
);
\draw
[step=\bc
,gray,xshift=5.4cm,yshift=0.3cm]
(0cm,0cm) grid (0cm+1*
\bc
,0cm+7*
\b
c
);
%\draw[line width=0.7pt,xshift=5.4cm,yshift=0.3cm] (0cm,0cm) rectangle (0cm+1*\bc
,0cm+7*\b
c);
\draw
[fill=blue!8,xshift=5.4cm,yshift=0.3cm,line width=0.6pt]
(0cm,0cm) rectangle (0cm+1*
\bc
c
,0cm+7*
\bc
c
);
\draw
[step=\bc
c,gray,xshift=5.4cm,yshift=0.3cm]
(0cm,0cm) grid (0cm+1*
\bcc
,0cm+7*
\bc
c
);
%\draw[line width=0.7pt,xshift=5.4cm,yshift=0.3cm] (0cm,0cm) rectangle (0cm+1*\bc
c,0cm+7*\bc
c);
\draw
[fill=blue!8,xshift=5.6cm,yshift=0cm,line width=0.6pt]
(0cm,0cm) rectangle (0cm+1*
\bc
,0cm+7*
\b
c
);
\draw
[step=\bc
,gray,xshift=5.6cm,yshift=0cm]
(0cm,0cm) grid (0cm+1*
\bc
,0cm+7*
\b
c
);
%\draw[line width=0.7pt,xshift=5.6cm,yshift=0cm] (0cm,0cm) rectangle (0cm+1*\bc
,0cm+7*\b
c);
\draw
[red!60,line width=2pt,xshift=5.6cm,yshift=0cm]
(0cm,0cm) rectangle (0cm+1*
\bc
,0cm+1*
\b
c
);
\draw
[red!60,line width=2pt,xshift=5.6cm,yshift=0cm]
(0cm,0cm+2*
\bc
) rectangle (0cm+1*
\bc
,0cm+3*
\b
c
);
\draw
[red!60,line width=2pt,xshift=5.6cm,yshift=0cm]
(0cm,0cm+6*
\bc
) rectangle (0cm+1*
\bc
,0cm+7*
\b
c
);
\draw
[fill=blue!8,xshift=5.6cm,yshift=0cm,line width=0.6pt]
(0cm,0cm) rectangle (0cm+1*
\bc
c
,0cm+7*
\bc
c
);
\draw
[step=\bc
c,gray,xshift=5.6cm,yshift=0cm]
(0cm,0cm) grid (0cm+1*
\bcc
,0cm+7*
\bc
c
);
%\draw[line width=0.7pt,xshift=5.6cm,yshift=0cm] (0cm,0cm) rectangle (0cm+1*\bc
c,0cm+7*\bc
c);
\draw
[red!60,line width=2pt,xshift=5.6cm,yshift=0cm]
(0cm,0cm) rectangle (0cm+1*
\bc
c
,0cm+1*
\bc
c
);
\draw
[red!60,line width=2pt,xshift=5.6cm,yshift=0cm]
(0cm,0cm+2*
\bc
c
) rectangle (0cm+1*
\bcc
,0cm+3*
\bc
c
);
\draw
[red!60,line width=2pt,xshift=5.6cm,yshift=0cm]
(0cm,0cm+6*
\bc
c
) rectangle (0cm+1*
\bcc
,0cm+7*
\bc
c
);
\draw
[line]
(8.4cm, 2.6cm) -- (11cm,2.2cm);
\draw
[line]
(9cm,1cm) -- (11.3cm, 1.4cm);
\draw
[red!60,line]
([xshift=5.6cm,yshift=0cm]0cm+1*
\bc
,0cm+7*
\b
c
) -- (8.45cm,1.4cm);
\draw
[red!60,line]
([xshift=5.6cm,yshift=0cm]0cm+1*
\bc
,0cm) -- (8.6cm, 1cm);
\draw
[red!60,line]
([xshift=5.6cm,yshift=0cm]0cm+1*
\bc
c
,0cm+7*
\bc
c
) -- (8.45cm,1.4cm);
\draw
[red!60,line]
([xshift=5.6cm,yshift=0cm]0cm+1*
\bc
c
,0cm) -- (8.6cm, 1cm);
\draw
[red!60,line]
(0cm+6*
\bc
,0cm+9*
\bc
) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+7*
\b
c
);
\draw
[red!60,line]
(0cm+6*
\bc
,0cm+7*
\bc
) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+6*
\b
c
);
\draw
[red!60,line]
(0cm+6*
\bc
,0cm+2*
\bc
) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+1*
\b
c
);
\draw
[red!60,line]
(0cm+6*
\bc
,0cm) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm);
\draw
[ugreen!60,line]
(0cm+6*
\bc
,0cm+6*
\bc
) -- ([xshift=5.0cm,yshift=1.3cm]0cm,0cm+3*
\b
c
);
\draw
[ugreen!60,line]
(0cm+6*
\bc
,0cm+3*
\bc
) -- ([xshift=5.0cm,yshift=1.3cm]0cm,0cm+2*
\b
c
);
\draw
[red!60,line]
([xshift=0.3cm,yshift=0.5cm]0cm+6*
\bc
,0cm+4*
\bc
) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+3*
\b
c
);
\draw
[red!60,line]
([xshift=0.3cm,yshift=0.5cm]0cm+6*
\bc
,0cm+2*
\bc
) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+2*
\b
c
);
\draw
[red!60,line]
(0cm+6*
\bc
c
,0cm+9*
\bcc
) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+7*
\bc
c
);
\draw
[red!60,line]
(0cm+6*
\bc
c
,0cm+7*
\bcc
) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+6*
\bc
c
);
\draw
[red!60,line]
(0cm+6*
\bc
c
,0cm+2*
\bcc
) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+1*
\bc
c
);
\draw
[red!60,line]
(0cm+6*
\bc
c
,0cm) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm);
\draw
[ugreen!60,line]
(0cm+6*
\bc
c
,0cm+6*
\bcc
) -- ([xshift=5.0cm,yshift=1.3cm]0cm,0cm+3*
\bc
c
);
\draw
[ugreen!60,line]
(0cm+6*
\bc
c
,0cm+3*
\bcc
) -- ([xshift=5.0cm,yshift=1.3cm]0cm,0cm+2*
\bc
c
);
\draw
[red!60,line]
([xshift=0.3cm,yshift=0.5cm]0cm+6*
\bc
c
,0cm+4*
\bcc
) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+3*
\bc
c
);
\draw
[red!60,line]
([xshift=0.3cm,yshift=0.5cm]0cm+6*
\bc
c
,0cm+2*
\bcc
) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+2*
\bc
c
);
\node
[word]
(w1) at (-0.5cm, 3.4cm)
{
wait
}
;
\node
[word]
(w2) at ([yshift=-
\bc
]w1)
{
for
}
;
\node
[word]
(w3) at ([yshift=-
\bc
]w2)
{
the
}
;
\node
[word]
(w4) at ([yshift=-
\bc
]w3)
{
video
}
;
\node
[word]
(w5) at ([yshift=-
\bc
]w4)
{
and
}
;
\node
[word]
(w6) at ([yshift=-
\bc
]w5)
{
do
}
;
\node
[word]
(w7) at ([yshift=-
\bc
]w6)
{
n't
}
;
\node
[word]
(w8) at ([yshift=-
\bc
]w7)
{
rent
}
;
\node
[word]
(w9) at ([yshift=-
\bc
]w8)
{
it
}
;
\node
[word]
(w2) at ([yshift=-
\bc
c
]w1)
{
for
}
;
\node
[word]
(w3) at ([yshift=-
\bc
c
]w2)
{
the
}
;
\node
[word]
(w4) at ([yshift=-
\bc
c
]w3)
{
video
}
;
\node
[word]
(w5) at ([yshift=-
\bc
c
]w4)
{
and
}
;
\node
[word]
(w6) at ([yshift=-
\bc
c
]w5)
{
do
}
;
\node
[word]
(w7) at ([yshift=-
\bc
c
]w6)
{
n't
}
;
\node
[word]
(w8) at ([yshift=-
\bc
c
]w7)
{
rent
}
;
\node
[word]
(w9) at ([yshift=-
\bc
c
]w8)
{
it
}
;
\node
[draw,rectangle callout,callout relative pointer={(0.28,-0.6)}]
at (-0.3cm,4.6cm)
{
\textrm
{
卷积核
}}
;
\node
[draw,rectangle callout,callout relative pointer={(0.1,-0.5)}]
at (5cm,4.6cm)
{
\textrm
{
特征图
}}
;
...
...
Chapter11/chapter11.tex
查看文件 @
007176c8
...
...
@@ -16,7 +16,7 @@
\renewcommand\figurename
{
图
}
%将figure改为图
\renewcommand\tablename
{
表
}
%将figure改为图
\chapterimage
{
fig-NEU-2.jpg
}
% Chapter heading image
\newlength
{
\bc
}
\newlength
{
\bc
c
}
%----------------------------------------------------------------------------------------
% CHAPTER 11
...
...
Chapter12/chapter12.tex
查看文件 @
007176c8
...
...
@@ -587,7 +587,7 @@ Transformer Deep（48层） & 30.2 & 43.1 & 194$\times 10^
\begin{itemize}
\vspace
{
0.5em
}
\item
近两年，有研究已经发现注意力机制可以捕捉一些语言现象
\upcite
{
DBLP:journals/corr/abs-1905-09418
}
，比如，在Transformer 的多头注意力中，不同头往往会捕捉到不同的信息，比如，有些头对低频词更加敏感，有些头更适合词意消歧，甚至有些头可以捕捉句法信息。此外，由于注意力机制增加了模型的复杂性，而且随着网络层数的增多，神经机器翻译中也存在大量的冗余，因此研发轻量的注意力模型也是具有实践意义的方向
\upcite
{
Xiao2019SharingAW,DBLP:journals/corr/abs-1805-00631,Lin2020WeightDT
}
。
\item
近两年，有研究已经发现注意力机制可以捕捉一些语言现象
\upcite
{
DBLP:journals/corr/abs-1905-09418
}
，比如，在Transformer 的多头注意力中，不同头往往会捕捉到不同的信息，比如，有些头对低频词更加敏感，有些头更适合词意消歧，甚至有些头可以捕捉句法信息。此外，由于注意力机制增加了模型的复杂性，而且随着网络层数的增多，神经机器翻译中也存在大量的冗余，因此研发轻量的注意力模型也是具有实践意义的方向
\upcite
{
Xiao2019SharingAW,DBLP:journals/corr/abs-1805-00631,Lin2020WeightDT
,DBLP:conf/iclr/WuLLLH20,Kitaev2020ReformerTE,DBLP:journals/corr/abs-2005-00743,dai-etal-2019-transformer,DBLP:journals/corr/abs-2004-05150,DBLP:conf/iclr/RaePJHL20
}
。
\vspace
{
0.5em
}
\item
神经机器翻译依赖成本较高的GPU设备，因此对模型的裁剪和加速也是很多系统研发人员所感兴趣的方向。比如，从工程上，可以考虑减少运算强度，比如使用低精度浮点数
\upcite
{
Ott2018ScalingNM
}
或者整数
\upcite
{
DBLP:journals/corr/abs-1906-00532,Lin2020TowardsF8
}
进行计算，或者引入缓存机制来加速模型的推断
\upcite
{
Vaswani2018Tensor2TensorFN
}
；也可以通过对模型参数矩阵的剪枝来减小整个模型的体积
\upcite
{
DBLP:journals/corr/SeeLM16
}
；另一种方法是知识蒸馏
\upcite
{
Hinton2015Distilling,kim-rush-2016-sequence
}
。 利用大模型训练小模型，这样往往可以得到比单独训练小模型更好的效果
\upcite
{
DBLP:journals/corr/ChenLCL17
}
。
\vspace
{
0.5em
}
...
...
bibliography.bib
查看文件 @
007176c8
差异被折叠。
点击展开。
编写
预览
Markdown
格式
0%
重试
或
添加新文件
添加附件
取消
您添加了
0
人
到此讨论。请谨慎行事。
请先完成此评论的编辑！
取消
请
注册
或者
登录
后发表评论
