Skip to content

M
mtbookv2
Commit 6ccd1667 by 孟霞
Options

合并分支 'master' 到 'mengxia' 

Master

查看合并请求 !917
parents 0e2895d3 bcc8f3d9
正在显示 包含 631 行增加 和 569 行删除
Chapter1/Figures/figure-example-smt.tex
...@@ -52,7 +52,7 @@ ...@@ -52,7 +52,7 @@
} }
{ {
\begin{scope}[xshift=1.7in] \begin{scope}[xshift=2in]
{\scriptsize {\scriptsize
\node [anchor=north west] (phrase1) at (0,0) {$\textrm{Pr}(\textrm{我} \to \textrm{I}) = 0.7$}; \node [anchor=north west] (phrase1) at (0,0) {$\textrm{Pr}(\textrm{我} \to \textrm{I}) = 0.7$};
\node [anchor=north west] (phrase2) at ([yshift=0.1em]phrase1.south west) {$\textrm{Pr}(\textrm{我} \to \textrm{me}) = 0.3$}; \node [anchor=north west] (phrase2) at ([yshift=0.1em]phrase1.south west) {$\textrm{Pr}(\textrm{我} \to \textrm{me}) = 0.3$};
...@@ -76,7 +76,7 @@ ...@@ -76,7 +76,7 @@
} }
{ {
\begin{scope}[xshift=1.7in,yshift=-1.55in] \begin{scope}[xshift=2in,yshift=-1.55in]
{\scriptsize {\scriptsize
\node [anchor=north west] (ngram1) at (0,0) {$\textrm{Pr}(\textrm{I}) = 0.0001$}; \node [anchor=north west] (ngram1) at (0,0) {$\textrm{Pr}(\textrm{I}) = 0.0001$};
\node [anchor=north west] (ngram2) at ([yshift=0.0em]ngram1.south west) {$\textrm{Pr}(\textrm{I}\ \textrm{am}) = 0.623$}; \node [anchor=north west] (ngram2) at ([yshift=0.0em]ngram1.south west) {$\textrm{Pr}(\textrm{I}\ \textrm{am}) = 0.623$};
...@@ -96,14 +96,14 @@ ...@@ -96,14 +96,14 @@
} }
{ {
\draw[->,thick,ublue] (bidata.east)--([xshift=2.2em]bidata.east) node[pos=0.5,above] (simexample) {\color{red}{\scriptsize{\scriptsize{学习}}}}; \draw[->,thick,ublue] (bidata.east)--([xshift=4.2em]bidata.east) node[pos=0.5,above] (simexample) {\color{red}{\scriptsize{\scriptsize{学习}}}};
} }
{ {
\draw[->,thick,ublue] (monodata.east)--([xshift=2.2em]monodata.east) node[pos=0.5,above] (simexample) {\color{red}{\scriptsize{\scriptsize{学习}}}}; \draw[->,thick,ublue] (monodata.east)--([xshift=4.2em]monodata.east) node[pos=0.5,above] (simexample) {\color{red}{\scriptsize{\scriptsize{学习}}}};
} }
\begin{scope}[xshift=3.6in] \begin{scope}[xshift=4in]
{\footnotesize {\footnotesize
{ {
\node[anchor=center] (srcsentence) at (0,0) {我 对 你 感到 满意}; \node[anchor=center] (srcsentence) at (0,0) {我 对 你 感到 满意};
......
Chapter1/chapter1.tex
...@@ -54,14 +54,6 @@ ...@@ -54,14 +54,6 @@
\parinterval 从机器翻译系统的组成上来看，通常可以抽象为两个部分，如图\ref{fig:1-2}所示： \parinterval 从机器翻译系统的组成上来看，通常可以抽象为两个部分，如图\ref{fig:1-2}所示：
\begin{itemize}
\vspace{0.5em}
\item {\small\bfnew{资源}}：如果把机器翻译系统比作一辆汽车，资源就好比是可以使汽车运行的“汽油”，它包括很多内容，如翻译规则、双（单）语数据、知识库等翻译知识，且这些“知识”都是计算机可读的。值得一提的是,如果没有翻译资源的支持，任何机器翻译系统都无法运行起来。
\vspace{0.5em}
\item {\small\bfnew{系统}}：机器翻译算法的程序实现被称作系统，也就是机器翻译研究人员开发的软件。无论是翻译规则、翻译模板还是统计模型中的参数都需要通过机器翻译系统进行读取和使用。
\vspace{0.5em}
\end{itemize}
%---------------------------------------------- %----------------------------------------------
\begin{figure}[htp] \begin{figure}[htp]
\centering \centering
...@@ -71,6 +63,14 @@ ...@@ -71,6 +63,14 @@
\end{figure} \end{figure}
%------------------------------------------- %-------------------------------------------
\begin{itemize}
\vspace{0.5em}
\item {\small\bfnew{资源}}：如果把机器翻译系统比作一辆汽车，资源就好比是可以使汽车运行的“汽油”，它包括很多内容，如翻译规则、双（单）语数据、知识库等翻译知识，且这些“知识”都是计算机可读的。值得一提的是,如果没有翻译资源的支持，任何机器翻译系统都无法运行起来。
\vspace{0.5em}
\item {\small\bfnew{系统}}：机器翻译算法的程序实现被称作系统，也就是机器翻译研究人员开发的软件。无论是翻译规则、翻译模板还是统计模型中的参数都需要通过机器翻译系统进行读取和使用。
\vspace{0.5em}
\end{itemize}
\parinterval 构建一个强大的机器翻译系统需要“资源”和“系统”两方面共同作用。在资源方面，随着语料库语言学的发展，已经有大量的高质量的双语和单语数据（称为语料）被整理并且被电子化存储，因此可以说具备了研发机器翻译系统所需要的语料基础。特别是像英语、汉语等世界主流语种，相关语料资源已经非常丰富，这也大大加速了相关研究的进展。当然，对于一些稀缺资源语种或者特殊的领域，语料库中的语料仍然匮乏，但是这些并不影响机器翻译领域整体的发展速度。因此在现有语料库的基础上，很多研究者把精力集中在“系统”研发上。 \parinterval 构建一个强大的机器翻译系统需要“资源”和“系统”两方面共同作用。在资源方面，随着语料库语言学的发展，已经有大量的高质量的双语和单语数据（称为语料）被整理并且被电子化存储，因此可以说具备了研发机器翻译系统所需要的语料基础。特别是像英语、汉语等世界主流语种，相关语料资源已经非常丰富，这也大大加速了相关研究的进展。当然，对于一些稀缺资源语种或者特殊的领域，语料库中的语料仍然匮乏，但是这些并不影响机器翻译领域整体的发展速度。因此在现有语料库的基础上，很多研究者把精力集中在“系统”研发上。
%---------------------------------------------------------------------------------------- %----------------------------------------------------------------------------------------
...@@ -129,7 +129,7 @@ ...@@ -129,7 +129,7 @@
\parinterval 1957年，Noam Chomsky在\emph{Syntactic Structures}中描述了转换生成语法\upcite{chomsky1957syntactic}，并使用数学方法来研究自然语言，建立了包括上下文有关语法、上下文无关语法等4种类型的语法。这些工作最终为今天计算机中广泛使用的“形式语言”奠定了基础。而他的思想也深深地影响了同时期的语言学和自然语言处理领域的学者。特别的是，早期基于规则的机器翻译中也大量使用了这些思想。 \parinterval 1957年，Noam Chomsky在\emph{Syntactic Structures}中描述了转换生成语法\upcite{chomsky1957syntactic}，并使用数学方法来研究自然语言，建立了包括上下文有关语法、上下文无关语法等4种类型的语法。这些工作最终为今天计算机中广泛使用的“形式语言”奠定了基础。而他的思想也深深地影响了同时期的语言学和自然语言处理领域的学者。特别的是，早期基于规则的机器翻译中也大量使用了这些思想。
\parinterval 虽然在这段时间，使用机器进行翻译的议题越加火热，但是事情并不总是一帆风顺，怀疑论者对机器翻译一直存有质疑，并很容易找出一些机器翻译无法解决的问题。自然地，人们也期望能够客观地评估一下机器翻译的可行性。当时美国基金资助组织委任自动语言处理咨询会承担了这项任务。经过近两年的调查与分析，该委员会于1966年11月公布了一个题为\emph{LANGUAGE AND MACHINES}的报告（图\ref{fig:1-5}），即ALPAC报告。该报告全面否定了机器翻译的可行性，为机器翻译的研究泼了一盆冷水。 \parinterval 虽然在这段时间，使用机器进行翻译的议题越加火热，但是事情并不总是一帆风顺，怀疑论者对机器翻译一直存有质疑，并很容易找出一些机器翻译无法解决的问题。自然地，人们也期望能够客观地评估一下机器翻译的可行性。当时美国基金资助组织委任自动语言处理咨询会承担了这项任务。经过近两年的调查与分析，该委员会于1966年11月公布了一个题为\emph{LANGUAGE AND MACHINES}的报告（图\ref{fig:1-4}），即ALPAC报告。该报告全面否定了机器翻译的可行性，为机器翻译的研究泼了一盆冷水。
%---------------------------------------------- %----------------------------------------------
\begin{figure}[htp] \begin{figure}[htp]
...@@ -258,8 +258,6 @@ ...@@ -258,8 +258,6 @@
\parinterval 规则就像语言中的“If-then”语句，如果满足条件，则执行相应的语义动作。比如，可以将待翻译句子中的某个词，使用目标语言单词进行替换，但是这种替换并非随意的，而是在语言学知识的指导下进行的。 \parinterval 规则就像语言中的“If-then”语句，如果满足条件，则执行相应的语义动作。比如，可以将待翻译句子中的某个词，使用目标语言单词进行替换，但是这种替换并非随意的，而是在语言学知识的指导下进行的。
\parinterval 图\ref{fig:1-8}展示了一个使用转换法进行翻译的实例。这里，利用一个简单的汉译英规则库完成对句子“我对你感到满意”的翻译。当翻译“我”时，从规则库中找到规则1，该规则表示遇到单词“我”就翻译为“I”；类似地，也可以从规则库中找到规则4，该规则表示翻译调序，即将单词“you”放到“be satisfied with”后面。这种通过规则表示单词之间的对应关系也为统计机器翻译方法提供了思路。如统计机器翻译中，基于短语的翻译模型使用短语对对原文进行替换，详细描述可以参考{\chapterseven}。
%---------------------------------------------- %----------------------------------------------
\begin{figure}[htp] \begin{figure}[htp]
\centering \centering
...@@ -269,6 +267,8 @@ ...@@ -269,6 +267,8 @@
\end{figure} \end{figure}
%------------------------------------------- %-------------------------------------------
\parinterval 图\ref{fig:1-8}展示了一个使用转换法进行翻译的实例。这里，利用一个简单的汉译英规则库完成对句子“我对你感到满意”的翻译。当翻译“我”时，从规则库中找到规则1，该规则表示遇到单词“我”就翻译为“I”；类似地，也可以从规则库中找到规则4，该规则表示翻译调序，即将单词“you”放到“be satisfied with”后面。这种通过规则表示单词之间的对应关系也为统计机器翻译方法提供了思路。如统计机器翻译中，基于短语的翻译模型使用短语对对原文进行替换，详细描述可以参考{\chapterseven}。
\parinterval 在上述例子中可以发现，规则不仅仅可以翻译句子之间单词的对应，如规则1，还可以表示句法甚至语法之间的对应，如规则6。因此基于规则的方法可以分成多个层次，如图\ref{fig:1-9}所示。图中不同的层次表示采用不同的知识来书写规则，进而完成机器翻译过程。对于翻译问题，可以构建不同层次的基于规则的机器翻译系统。这里包括四个层次，分别为：词汇转换、句法转换、语义转换和中间语言层。其中，上层可以继承下层的翻译知识，比如说句法转换层会利用词汇转换层知识。早期基于规则的方法属于词汇转换层。 \parinterval 在上述例子中可以发现，规则不仅仅可以翻译句子之间单词的对应，如规则1，还可以表示句法甚至语法之间的对应，如规则6。因此基于规则的方法可以分成多个层次，如图\ref{fig:1-9}所示。图中不同的层次表示采用不同的知识来书写规则，进而完成机器翻译过程。对于翻译问题，可以构建不同层次的基于规则的机器翻译系统。这里包括四个层次，分别为：词汇转换、句法转换、语义转换和中间语言层。其中，上层可以继承下层的翻译知识，比如说句法转换层会利用词汇转换层知识。早期基于规则的方法属于词汇转换层。
%---------------------------------------------- %----------------------------------------------
......
Chapter10/Figures/figure-3-base-problom-of-p.tex
...@@ -12,7 +12,7 @@ ...@@ -12,7 +12,7 @@
% RNN Encoder % RNN Encoder
\coordinate (eemb0) at (0,0); \coordinate (eemb0) at (0,0);
\foreach \x [count=\y from 0] in {1,2,...,3} \foreach \x [count=\y from 0] in {1,2,...,3}
\node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=west] (eemb\x) at ([xshift=0.4\base]eemb\y.east) {\tiny{$\textrm{e}_x()$}}; \node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=west] (eemb\x) at ([xshift=0.6\base]eemb\y.east) {\tiny{$\textrm{e}_x()$}};
\foreach \x in {1,2,...,3} \foreach \x in {1,2,...,3}
\node[rnnnode,fill=blue!30!white,anchor=south] (enc\x) at ([yshift=0.3\base]eemb\x.north) {}; \node[rnnnode,fill=blue!30!white,anchor=south] (enc\x) at ([yshift=0.3\base]eemb\x.north) {};
\node[] (enclabel1) at (enc1) {\tiny{$\mathbi{h}_{m-2}$}}; \node[] (enclabel1) at (enc1) {\tiny{$\mathbi{h}_{m-2}$}};
...@@ -73,7 +73,7 @@ ...@@ -73,7 +73,7 @@
\draw[-latex'] (enc3.north) .. controls +(north:0.3\base) and +(east:\base) .. (bridge) .. controls +(west:2.7\base) and +(west:0.3\base) .. (dec1.west); \draw[-latex'] (enc3.north) .. controls +(north:0.3\base) and +(east:\base) .. (bridge) .. controls +(west:2.7\base) and +(west:0.3\base) .. (dec1.west);
{ {
\node [anchor=east] (line1) at ([xshift=-3em,yshift=0.5em]softmax1.west) {\scriptsize{基于RNN的隐层状态$\mathbi{s}_i$}}; \node [anchor=east] (line1) at ([xshift=-4em,yshift=0.5em]softmax1.west) {\scriptsize{基于RNN的隐层状态$\mathbi{s}_i$}};
\node [anchor=north west] (line2) at ([yshift=0.3em]line1.south west) {\scriptsize{预测目标词的概率}}; \node [anchor=north west] (line2) at ([yshift=0.3em]line1.south west) {\scriptsize{预测目标词的概率}};
\node [anchor=north west] (line3) at ([yshift=0.3em]line2.south west) {\scriptsize{通常，用Softmax函数}}; \node [anchor=north west] (line3) at ([yshift=0.3em]line2.south west) {\scriptsize{通常，用Softmax函数}};
\node [anchor=north west] (line4) at ([yshift=0.3em]line3.south west) {\scriptsize{实现 $\funp{P}(y_i|...)$}}; \node [anchor=north west] (line4) at ([yshift=0.3em]line3.south west) {\scriptsize{实现 $\funp{P}(y_i|...)$}};
...@@ -87,7 +87,7 @@ ...@@ -87,7 +87,7 @@
} }
{ {
\node [anchor=west] (line21) at ([xshift=1.3em,yshift=1.5em]enc3.east) {\scriptsize{源语编码器最后一个}}; \node [anchor=west] (line21) at ([xshift=3.3em,yshift=1.5em]enc3.east) {\scriptsize{源语编码器最后一个}};
\node [anchor=north west] (line22) at ([yshift=0.3em]line21.south west) {\scriptsize{循环单元的输出被}}; \node [anchor=north west] (line22) at ([yshift=0.3em]line21.south west) {\scriptsize{循环单元的输出被}};
\node [anchor=north west] (line23) at ([yshift=0.3em]line22.south west) {\scriptsize{看作是句子的表示,}}; \node [anchor=north west] (line23) at ([yshift=0.3em]line22.south west) {\scriptsize{看作是句子的表示,}};
\node [anchor=north west] (line24) at ([yshift=0.3em]line23.south west) {\scriptsize{记为$\mathbi{C}$}}; \node [anchor=north west] (line24) at ([yshift=0.3em]line23.south west) {\scriptsize{记为$\mathbi{C}$}};
...@@ -97,21 +97,21 @@ ...@@ -97,21 +97,21 @@
{ {
\node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=red!10,drop shadow,draw=red,minimum width =9em] [fit = (line1) (line2) (line3) (line4)] (box1) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=red!10,drop shadow,draw=red,minimum width =9em] [fit = (line1) (line2) (line3) (line4)] (box1) {};
\node [rectangle,inner sep=0.2em,rounded corners=1pt,very thick,dotted,draw=red] [fit = (softmax1) (softmax2) (softmax3)] (box4) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,very thick,dotted,draw=red] [fit = (softmax1) (softmax2) (softmax3)] (box4) {};
\draw [->,dotted,very thick,red] ([yshift=1em,xshift=2.5em]box1.east) -- ([yshift=1em,xshift=0.1em]box1.east); \draw [->,dotted,very thick,red] ([yshift=1em,xshift=3.5em]box1.east) -- ([yshift=1em,xshift=0.1em]box1.east);
} }
{ {
\node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=green!10,drop shadow,draw=ugreen,minimum width =9em] [fit = (line11) (line12) (line13) (line14)] (box2) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=green!10,drop shadow,draw=ugreen,minimum width =9em] [fit = (line11) (line12) (line13) (line14)] (box2) {};
\node [rectangle,inner sep=0.2em,rounded corners=1pt,very thick,dotted,draw=ugreen] [fit = (eemb1) (eemb2) (eemb3)] (box5) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,very thick,dotted,draw=ugreen] [fit = (eemb1) (eemb2) (eemb3)] (box5) {};
\node [rectangle,inner sep=0.2em,rounded corners=1pt,very thick,dotted,draw=ugreen] [fit = (demb1) (demb2) (demb3)] (box6) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,very thick,dotted,draw=ugreen] [fit = (demb1) (demb2) (demb3)] (box6) {};
\draw [->,dotted,very thick,ugreen] ([yshift=-1.3em,xshift=2.5em]box2.east) -- ([yshift=-1.3em,xshift=0.1em]box2.east); \draw [->,dotted,very thick,ugreen] ([yshift=-1.7em,xshift=3.5em]box2.east) -- ([yshift=-1.7em,xshift=0.1em]box2.east);
\draw [->,dotted,very thick,ugreen] ([xshift=0.1em]box6.west) .. controls +(west:1) and +(east:1) .. ([yshift=1.0em]box2.east) ; \draw [->,dotted,very thick,ugreen] ([xshift=0.1em]box6.west) .. controls +(west:1) and +(east:1) .. ([yshift=1.0em]box2.east) ;
} }
{ {
\node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=purple!10,drop shadow,draw=purple] [fit = (line21) (line22) (line23) (line24)] (box3) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=purple!10,drop shadow,draw=purple] [fit = (line21) (line22) (line23) (line24)] (box3) {};
\node [rectangle,inner sep=0.2em,rounded corners=1pt,very thick,dotted,draw=purple] [fit = (enc3)] (box7) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,very thick,dotted,draw=purple] [fit = (enc3)] (box7) {};
\draw [->,dotted,very thick,purple] ([xshift=0.1em]box7.east) -- ([xshift=0.8em]box7.east) ; \draw [->,dotted,very thick,purple] ([xshift=0.1em]box7.east) -- ([xshift=2.8em]box7.east) ;
} }
\end{pgfonlayer} \end{pgfonlayer}
......
Chapter10/Figures/figure-a-working-example-of-neural-machine-translation.tex
...@@ -10,11 +10,11 @@ ...@@ -10,11 +10,11 @@
% RNN Encoder % RNN Encoder
\coordinate (eemb0) at (0,0); \coordinate (eemb0) at (0,0);
\foreach \x [count=\y from 0] in {1,2,...,4} \foreach \x [count=\y from 0] in {1,2,...,4}
\node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=west] (eemb\x) at ([xshift=0.4\base]eemb\y.east) {}; \node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=west] (eemb\x) at ([xshift=0.7\base]eemb\y.east) {};
\foreach \x in {1,2,...,4} \foreach \x in {1,2,...,4}
\node[rnnnode,fill=blue!30!white,anchor=south] (enc\x) at ([yshift=0.5\base]eemb\x.north) {}; \node[rnnnode,fill=blue!30!white,anchor=south] (enc\x) at ([yshift=0.5\base]eemb\x.north) {};
\node[wordnode,left=0.4\base of enc1,font=\footnotesize] (init) {0}; \node[wordnode,left=0.4\base of enc1,font=\footnotesize] (init) {0};
\node[wordnode,anchor=east] (init2) at ([xshift=-3.0em]init.west){}; \node[wordnode,anchor=east] (init2) at ([xshift=-3em]init.west){};
{ {
\node[rnnnode,fill=purple] (repr) at (enc4) {}; \node[rnnnode,fill=purple] (repr) at (enc4) {};
\node[wordnode] (label) at ([yshift=2.5em]enc4.north) { \node[wordnode] (label) at ([yshift=2.5em]enc4.north) {
...@@ -35,7 +35,7 @@ ...@@ -35,7 +35,7 @@
% RNN Decoder % RNN Decoder
\foreach \x in {1,2,...,4} \foreach \x in {1,2,...,4}
\node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=south] (demb\x) at ([xshift=9.5em,yshift=-3.9em]enc\x.north) {}; \node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=south] (demb\x) at ([xshift=12.5em,yshift=-3.9em]enc\x.north) {};
\foreach \x in {1,2,...,4} \foreach \x in {1,2,...,4}
\node[rnnnode,fill=blue!30!white,anchor=south] (dec\x) at ([yshift=0.5\base]demb\x.north) {}; \node[rnnnode,fill=blue!30!white,anchor=south] (dec\x) at ([yshift=0.5\base]demb\x.north) {};
\foreach \x in {1,2,...,4} \foreach \x in {1,2,...,4}
......
Chapter10/Figures/figure-attention-of-source-and-target-words.tex
...@@ -11,7 +11,7 @@ ...@@ -11,7 +11,7 @@
%\setlength{\mystep}{1.6em} %\setlength{\mystep}{1.6em}
\foreach \x in {1,2,...,6} \foreach \x in {1,2,...,6}
\node[] (s\x) at (\x * 1.6em,0) {}; \node[] (s\x) at (\x * 2em,0) {};
\node [] (ws1) at (s1) {\scriptsize{这}}; \node [] (ws1) at (s1) {\scriptsize{这}};
\node [] (ws2) at (s2) {\scriptsize{是}}; \node [] (ws2) at (s2) {\scriptsize{是}};
...@@ -21,7 +21,7 @@ ...@@ -21,7 +21,7 @@
\node [] (ws6) at (s6) {\scriptsize{句子}}; \node [] (ws6) at (s6) {\scriptsize{句子}};
\foreach \x in {1,2,...,6} \foreach \x in {1,2,...,6}
\node[] (t\x) at (\x * 1.6em + 2.4in,0) {}; \node[] (t\x) at (\x * 2em + 2.8in,0) {};
\node [] (wt1) at (t1) {\scriptsize{This}}; \node [] (wt1) at (t1) {\scriptsize{This}};
\node [] (wt2) at (t2) {\scriptsize{is}}; \node [] (wt2) at (t2) {\scriptsize{is}};
...@@ -30,9 +30,9 @@ ...@@ -30,9 +30,9 @@
\node [] (wt5) at (t5) {\scriptsize{long}}; \node [] (wt5) at (t5) {\scriptsize{long}};
\node [] (wt6) at ([xshift=1em]t6) {\scriptsize{sentence}}; \node [] (wt6) at ([xshift=1em]t6) {\scriptsize{sentence}};
\node [anchor=south west,fill=red!30,minimum width=1.6in,minimum height=1.5em] (encoder) at ([yshift=1.0em]ws1.north west) {\footnotesize{Encoder}}; \node [anchor=south west,fill=red!30,minimum width=1.9in,minimum height=1.5em] (encoder) at ([yshift=1.0em,xshift=-0.4em]ws1.north west) {\footnotesize{Encoder}};
\node [anchor=west,fill=blue!30,minimum width=1.9in,minimum height=1.5em] (decoder) at ([xshift=4.5em]encoder.east) {\footnotesize{Decoder}}; \node [anchor=west,fill=green!30,minimum height=1.5em] (representation) at ([xshift=2em]encoder.east) {\footnotesize{表示}};
\node [anchor=west,fill=green!30,minimum height=1.5em] (representation) at ([xshift=1em]encoder.east) {\footnotesize{表示}}; \node [anchor=west,fill=blue!30,minimum width=1.9in,minimum height=1.5em] (decoder) at ([xshift=2em]representation.east) {\footnotesize{Decoder}};
\draw [->,thick] ([xshift=1pt]encoder.east)--([xshift=-1pt]representation.west); \draw [->,thick] ([xshift=1pt]encoder.east)--([xshift=-1pt]representation.west);
\draw [->,thick] ([xshift=1pt]representation.east)--([xshift=-1pt]decoder.west); \draw [->,thick] ([xshift=1pt]representation.east)--([xshift=-1pt]decoder.west);
......
Chapter10/Figures/figure-beam-search-process.tex
...@@ -5,7 +5,7 @@ ...@@ -5,7 +5,7 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{rnnnode} = [minimum height=1.1em,minimum width=3.5em,inner sep=2pt,rounded corners=1pt,draw,fill=red!20]; \tikzstyle{rnnnode} = [minimum height=1.1em,minimum width=4em,inner sep=2pt,rounded corners=1pt,draw,fill=red!20];
\tikzstyle{wnode} = [minimum height=1.0em,minimum width=3.5em,inner sep=2pt,rounded corners=1pt,draw,fill=white]; \tikzstyle{wnode} = [minimum height=1.0em,minimum width=3.5em,inner sep=2pt,rounded corners=1pt,draw,fill=white];
...@@ -137,13 +137,13 @@ ...@@ -137,13 +137,13 @@
} }
{ {
\node [anchor=east] (vocab) at ([xshift=-5em]s1.west) {\tiny{$\begin{bmatrix} \textrm{Have} & 0.50 \\ \textrm{I} & 0.02 \\ \textrm{it} & 0.03 \\ \textrm{has} & 0.30 \\ \textrm{you} & 0.01 \\ \textrm{the} & 0.01 \\ \textrm{a} & 0.01 \\ \textrm{an} & 0.02 \\ \textrm{he} & 0.03 \\ \textrm{she} & 0.01 \\ \textrm{are} & 0.00 \\ \textrm{am} & 0.01 \\ ... & ... \end{bmatrix}$}}; \node [anchor=east] (vocab) at ([xshift=-8em]s1.west) {\tiny{$\begin{bmatrix} \textrm{Have} & 0.50 \\ \textrm{I} & 0.02 \\ \textrm{it} & 0.03 \\ \textrm{has} & 0.30 \\ \textrm{you} & 0.01 \\ \textrm{the} & 0.01 \\ \textrm{a} & 0.01 \\ \textrm{an} & 0.02 \\ \textrm{he} & 0.03 \\ \textrm{she} & 0.01 \\ \textrm{are} & 0.00 \\ \textrm{am} & 0.01 \\ ... & ... \end{bmatrix}$}};
\node [anchor=south] (vocablabel) at (vocab.north) {\scriptsize{单词的概率分布}}; \node [anchor=south] (vocablabel) at (vocab.north) {\scriptsize{单词的概率分布}};
\draw [->,red,very thick,dotted] (o1.west) .. controls +(west:1) and +(east:2) .. ([yshift=1em]vocab.south east); \draw [->,red,very thick,dotted] (o1.west) .. controls +(west:1) and +(east:2) .. ([yshift=1em]vocab.south east);
} }
{ {
\node [anchor=east,inner sep=1pt] (vocabtopn) at ([xshift=-0.5em,yshift=-0.5em]wo1.west) {\scriptsize{$\begin{bmatrix} \textrm{Have} \\ \textrm{has} \\ \textrm{it} \end{bmatrix}$}}; \node [anchor=east,inner sep=1pt] (vocabtopn) at ([xshift=-1.5em,yshift=-0.5em]wo1.west) {\scriptsize{$\begin{bmatrix} \textrm{Have} \\ \textrm{has} \\ \textrm{it} \end{bmatrix}$}};
\draw [->] ([yshift=-1.6em,xshift=-0.4em]vocab.north east) .. controls +(east:1) and +(west:1) .. ([xshift=0.1em,yshift=0.4em]vocabtopn.west) node [pos=0.3,below] (topnlabel) {\scriptsize{top-3}}; \draw [->] ([yshift=-1.6em,xshift=-0.4em]vocab.north east) .. controls +(east:1) and +(west:1) .. ([xshift=0.1em,yshift=0.4em]vocabtopn.west) node [pos=0.3,below] (topnlabel) {\scriptsize{top-3}};
{ {
......
Chapter10/Figures/figure-bi-rnn.tex
...@@ -13,7 +13,7 @@ ...@@ -13,7 +13,7 @@
% RNN Encoder % RNN Encoder
\coordinate (eemb0) at (0,0); \coordinate (eemb0) at (0,0);
\foreach \x [count=\y from 0] in {1,2,...,10} \foreach \x [count=\y from 0] in {1,2,...,10}
\node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=west] (eemb\x) at ([xshift=0.4\base]eemb\y.east) {}; \node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=west] (eemb\x) at ([xshift=0.7\base]eemb\y.east) {};
\foreach \x in {1,2,...,10} \foreach \x in {1,2,...,10}
\node[rnnnode,fill=blue!30!white,anchor=south] (backenc\x) at ([yshift=0.5\base]eemb\x.north) {}; \node[rnnnode,fill=blue!30!white,anchor=south] (backenc\x) at ([yshift=0.5\base]eemb\x.north) {};
\foreach \x in {1,2,...,10} \foreach \x in {1,2,...,10}
......
Chapter10/Figures/figure-calculation-process-of-context-vector-c.tex
...@@ -8,25 +8,25 @@ ...@@ -8,25 +8,25 @@
\begin{scope} \begin{scope}
\node [anchor=west,draw,fill=red!20!white,inner sep=3pt,minimum width=2em,minimum height=1.2em] (h1) at (0,0) {\scriptsize{$\mathbi{h}_1$}}; \node [anchor=west,draw,fill=red!30!white,inner sep=3pt,minimum width=2em,minimum height=1.2em] (h1) at (0,0) {\scriptsize{$\mathbi{h}_1$}};
\node [anchor=west,draw,fill=red!20!white,inner sep=3pt,minimum width=2em,minimum height=1.2em] (h2) at ([xshift=1em]h1.east) {\scriptsize{$\mathbi{h}_2$}}; \node [anchor=west,draw,fill=red!30!white,inner sep=3pt,minimum width=2em,minimum height=1.2em] (h2) at ([xshift=2em]h1.east) {\scriptsize{$\mathbi{h}_2$}};
\node [anchor=west,inner sep=0pt,minimum width=3em] (h3) at ([xshift=0.5em]h2.east) {\scriptsize{...}}; \node [anchor=west,inner sep=0pt,minimum width=3em] (h3) at ([xshift=1em]h2.east) {\scriptsize{...}};
\node [anchor=west,draw,fill=red!20!white,inner sep=3pt,minimum width=2em,minimum height=1.2em] (h4) at ([xshift=0.5em]h3.east) {\scriptsize{$\mathbi{h}_m$}}; \node [anchor=west,draw,fill=red!30!white,inner sep=3pt,minimum width=2em,minimum height=1.2em] (h4) at ([xshift=1em]h3.east) {\scriptsize{$\mathbi{h}_m$}};
\node [anchor=south,circle,minimum size=1.0em,draw,ublue,thick] (sum) at ([yshift=2em]h2.north east) {}; \node [anchor=south,circle,minimum size=1.0em,draw,ublue,thick] (sum) at ([yshift=2em]h2.north east) {};
\draw [thick,-,ublue] (sum.north) -- (sum.south); \draw [thick,-,ublue] (sum.north) -- (sum.south);
\draw [thick,-,ublue] (sum.west) -- (sum.east); \draw [thick,-,ublue] (sum.west) -- (sum.east);
\node [anchor=south,draw,fill=green!20!white,inner sep=3pt,minimum width=2em,minimum height=1.2em] (th1) at ([yshift=2em,xshift=-1em]sum.north west) {\scriptsize{$\mathbi{s}_{j-1}$}}; \node [anchor=south,draw,fill=green!30!white,inner sep=3pt,minimum width=2em,minimum height=1.2em] (th1) at ([yshift=2em,xshift=-1em]sum.north west) {\scriptsize{$\mathbi{s}_{j-1}$}};
\node [anchor=west,draw,fill=green!20!white,inner sep=3pt,minimum width=2em,minimum height=1.2em] (th2) at ([xshift=2em]th1.east) {\scriptsize{$\mathbi{s}_{j}$}}; \node [anchor=west,draw,fill=green!30!white,inner sep=3pt,minimum width=2em,minimum height=1.2em] (th2) at ([xshift=2.5em]th1.east) {\scriptsize{$\mathbi{s}_{j}$}};
\draw [->] (h1.north) .. controls +(north:0.8) and +(west:1) .. (sum.190) node [pos=0.2,left] {\scriptsize{$\alpha_{1,j}$}}; \draw [->] (h1.north) .. controls +(north:0.8) and +(west:1) .. (sum.190) node [pos=0.2,left] {\scriptsize{$\alpha_{1,j}$}};
\draw [->] (h2.north) .. controls +(north:0.6) and +(220:0.2) .. (sum.220) node [pos=0.2,right] {\scriptsize{$\alpha_{2,j}$}}; \draw [->] (h2.north) .. controls +(north:0.6) and +(220:0.2) .. (sum.220) node [pos=0.2,right] {\scriptsize{$\alpha_{2,j}$}};
\draw [->] (h4.north) .. controls +(north:0.8) and +(east:1) .. (sum.-10) node [pos=0.1,left] (alphan) {\scriptsize{$\alpha_{m,j}$}}; \draw [->] (h4.north) .. controls +(north:0.8) and +(east:1) .. (sum.-10) node [pos=0.1,left] (alphan) {\scriptsize{$\alpha_{m,j}$}};
\draw [->] ([xshift=-1.5em]th1.west) -- ([xshift=-0.1em]th1.west); \draw [->] ([xshift=-2em]th1.west) -- ([xshift=-0.1em]th1.west);
\draw [->] ([xshift=0.1em]th1.east) -- ([xshift=-0.1em]th2.west); \draw [->] ([xshift=0.1em]th1.east) -- ([xshift=-0.1em]th2.west);
\draw [->] ([xshift=0.1em]th2.east) -- ([xshift=1.5em]th2.east); \draw [->] ([xshift=0.1em]th2.east) -- ([xshift=2em]th2.east);
\draw [->] (sum.north) .. controls +(north:0.8) and +(west:0.2) .. ([yshift=-0.4em,xshift=-0.1em]th2.west) node [pos=0.2,right] (ci) {\scriptsize{$\mathbi{C}_{j}$}}; \draw [->] (sum.north) .. controls +(north:0.8) and +(west:0.2) .. ([yshift=-0.4em,xshift=-0.1em]th2.west) node [pos=0.2,right] (ci) {\scriptsize{$\mathbi{C}_{j}$}};
\node [anchor=south,inner sep=1pt] (output) at ([yshift=0.8em]th2.north) {\scriptsize{输出层}}; \node [anchor=south,inner sep=1pt] (output) at ([yshift=0.8em]th2.north) {\scriptsize{输出层}};
...@@ -39,7 +39,7 @@ ...@@ -39,7 +39,7 @@
\node [anchor=north] (enc42) at ([yshift=0.5em]enc4.south) {\scriptsize{(位置$m$)}}; \node [anchor=north] (enc42) at ([yshift=0.5em]enc4.south) {\scriptsize{(位置$m$)}};
{ {
\node [anchor=west] (math1) at ([xshift=5em,yshift=1em]th2.east) {$\mathbi{C}_j = \sum_{i} \alpha_{i,j} \mathbi{h}_i $ \ \ \ \ }; \node [anchor=west] (math1) at ([xshift=7em,yshift=1em]th2.east) {$\mathbi{C}_j = \sum_{i} \alpha_{i,j} \mathbi{h}_i $ \ \ \ \ };
} }
{ {
\node [anchor=north west] (math2) at ([yshift=-2em,xshift=-0.2em]math1.south west) {$\alpha_{i,j} = \frac{\exp(\beta_{i,j})}{\sum_{i'} \exp(\beta_{i',j})}$}; \node [anchor=north west] (math2) at ([yshift=-2em,xshift=-0.2em]math1.south west) {$\alpha_{i,j} = \frac{\exp(\beta_{i,j})}{\sum_{i'} \exp(\beta_{i',j})}$};
...@@ -48,10 +48,10 @@ ...@@ -48,10 +48,10 @@
\begin{pgfonlayer}{background} \begin{pgfonlayer}{background}
{ {
\node [rectangle,inner sep=0.4em,rounded corners=1pt,fill=blue!10,drop shadow,minimum width=9.5em] [fit = (math1)] (box1) {}; \node [rectangle,inner sep=0.4em,rounded corners=1pt,fill=blue!20,drop shadow,minimum width=9.5em] [fit = (math1)] (box1) {};
} }
{ {
\node [rectangle,inner sep=0.4em,rounded corners=1pt,fill=orange!10,drop shadow,minimum width=9.5em] [fit = (math2) (math3)] (box2) {}; \node [rectangle,inner sep=0.4em,rounded corners=1pt,fill=orange!20,drop shadow,minimum width=9.5em] [fit = (math2) (math3)] (box2) {};
} }
\end{pgfonlayer} \end{pgfonlayer}
......
Chapter10/Figures/figure-data-parallel-process.tex
...@@ -17,7 +17,7 @@ ...@@ -17,7 +17,7 @@
\node [samplenode,anchor=south west,font=\scriptsize] (batch\i) at ([shift={(-1em,-0.5em)}]batch\j.south west) {句子 \k}; \node [samplenode,anchor=south west,font=\scriptsize] (batch\i) at ([shift={(-1em,-0.5em)}]batch\j.south west) {句子 \k};
\draw [decorate,decoration={brace}] (batch1.south east) to node [auto,rotate=30,anchor=north,font=\scriptsize] {batch大小} (batch3.south east); \draw [decorate,decoration={brace}] (batch1.south east) to node [auto,rotate=30,anchor=north,font=\scriptsize] {batch大小} (batch3.south east);
\node [samplenode,anchor=west,font=\scriptsize] (sample2) at ([xshift=4em]batch2.east) {句子2}; \node [samplenode,anchor=west,font=\scriptsize] (sample2) at ([xshift=5em]batch2.east) {句子2};
\node [samplenode,anchor=south,font=\scriptsize] (sample3) at ([yshift=3em]sample2.north) {句子3}; \node [samplenode,anchor=south,font=\scriptsize] (sample3) at ([yshift=3em]sample2.north) {句子3};
\node [samplenode,anchor=north,font=\scriptsize] (sample1) at ([yshift=-3em]sample2.south) {句子1}; \node [samplenode,anchor=north,font=\scriptsize] (sample1) at ([yshift=-3em]sample2.south) {句子1};
...@@ -26,18 +26,18 @@ ...@@ -26,18 +26,18 @@
\foreach \i in {1,2,3} \foreach \i in {1,2,3}
{ {
\coordinate (start) at ([xshift=2em]sample\i.east); \coordinate (start) at ([xshift=3em]sample\i.east);
\node [wordnode,anchor=west] (rnn0) at (start) {$0$}; \node [wordnode,anchor=west] (rnn0) at (start) {$0$};
\foreach \j [count=\k from 0] in {1,2,3} \foreach \j [count=\k from 0] in {1,2,3}
{ {
\node [rnnnode,anchor=west] (rnn\j) at ([xshift=1em]rnn\k.east) {}; \node [rnnnode,anchor=west] (rnn\j) at ([xshift=1.5em]rnn\k.east) {};
\draw [-latex'] (rnn\k) to (rnn\j); \draw [-latex'] (rnn\k) to (rnn\j);
\coordinate (in\j) at ([yshift=-1em]rnn\j.south); \coordinate (in\j) at ([yshift=-1em]rnn\j.south);
\draw [-latex'] (in\j) to (rnn\j.south); \draw [-latex'] (in\j) to (rnn\j.south);
\coordinate (out\j) at ([yshift=1em]rnn\j.north); \coordinate (out\j) at ([yshift=1em]rnn\j.north);
\draw [-latex'] (rnn\j.north) to (out\j); \draw [-latex'] (rnn\j.north) to (out\j);
} }
\node [wordnode,anchor=west] (rnn4) at ([xshift=1em]rnn3.east) {$\cdots$}; \node [wordnode,anchor=west] (rnn4) at ([xshift=1.5em]rnn3.east) {$\cdots$};
\draw [-latex'] (rnn3) to (rnn4); \draw [-latex'] (rnn3) to (rnn4);
\node [draw,densely dashed,thick,rounded corners=0.3em,fit=(start) (in3) (out3) (rnn4),label={[font=\footnotesize,rotate=90,anchor=north]0:设备\i}] (rnn) {}; \node [draw,densely dashed,thick,rounded corners=0.3em,fit=(start) (in3) (out3) (rnn4),label={[font=\footnotesize,rotate=90,anchor=north]0:设备\i}] (rnn) {};
\draw [->,double] ([xshift=3pt]sample\i.east) -- ([xshift=-3pt]rnn.west); \draw [->,double] ([xshift=3pt]sample\i.east) -- ([xshift=-3pt]rnn.west);
......
Chapter10/Figures/figure-decode-the-word-probability-distribution-at-the-first-position.tex
...@@ -42,7 +42,7 @@ ...@@ -42,7 +42,7 @@
} }
{ {
\node [anchor=east] (vocab) at ([xshift=-5em]s1.west) {\tiny{$\begin{bmatrix} \textrm{Have} & 0.50 \\ \textrm{I} & 0.02 \\ \textrm{it} & 0.03 \\ \textrm{has} & 0.30 \\ \textrm{you} & 0.01 \\ \textrm{the} & 0.01 \\ \textrm{a} & 0.01 \\ \textrm{an} & 0.02 \\ \textrm{he} & 0.03 \\ \textrm{she} & 0.01 \\ \textrm{are} & 0.00 \\ \textrm{am} & 0.01 \\ ... & ... \end{bmatrix}$}}; \node [anchor=east] (vocab) at ([xshift=-7em]s1.west) {\tiny{$\begin{bmatrix} \textrm{Have} & 0.50 \\ \textrm{I} & 0.02 \\ \textrm{it} & 0.03 \\ \textrm{has} & 0.30 \\ \textrm{you} & 0.01 \\ \textrm{the} & 0.01 \\ \textrm{a} & 0.01 \\ \textrm{an} & 0.02 \\ \textrm{he} & 0.03 \\ \textrm{she} & 0.01 \\ \textrm{are} & 0.00 \\ \textrm{am} & 0.01 \\ ... & ... \end{bmatrix}$}};
\node [anchor=south] (vocablabel) at (vocab.north) {\scriptsize{单词的概率分布}}; \node [anchor=south] (vocablabel) at (vocab.north) {\scriptsize{单词的概率分布}};
\draw [->,red,very thick,dotted] (o1.west) .. controls +(west:1) and +(east:2) .. ([yshift=1em]vocab.south east); \draw [->,red,very thick,dotted] (o1.west) .. controls +(west:1) and +(east:2) .. ([yshift=1em]vocab.south east);
} }
......
Chapter10/Figures/figure-decoding-process-based-on-greedy-method.tex
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{rnnnode} = [minimum height=1.1em,minimum width=2.1em,inner sep=2pt,rounded corners=1pt,draw,fill=red!20]; \tikzstyle{rnnnode} = [minimum height=1.1em,minimum width=2.7em,inner sep=2pt,rounded corners=1pt,draw,fill=red!20];
\node [rnnnode,anchor=west] (h1) at (0,0) {\tiny{$\mathbi{h}_1$}}; \node [rnnnode,anchor=west] (h1) at (0,0) {\tiny{$\mathbi{h}_1$}};
\node [anchor=west] (h2) at ([xshift=1em]h1.east) {\tiny{...}}; \node [anchor=west] (h2) at ([xshift=1em]h1.east) {\tiny{...}};
...@@ -139,7 +139,7 @@ ...@@ -139,7 +139,7 @@
\draw [->] ([yshift=-0.3em]s1.west) .. controls +(west:2) and +(-50:0.3) .. (c2.-40); \draw [->] ([yshift=-0.3em]s1.west) .. controls +(west:2) and +(-50:0.3) .. (c2.-40);
} }
{ {
\draw [->] (c2.0) -- ([xshift=1.2in]c2.0) -- ([yshift=0.3em,xshift=-1.2em]s2.west) -- ([yshift=0.3em,xshift=-0.1em]s2.west); \draw [->] (c2.0) -- ([xshift=1.4in]c2.0) -- ([yshift=0.3em,xshift=-1em]s2.west) -- ([yshift=0.3em,xshift=-0.1em]s2.west);
} }
{ {
......
Chapter10/Figures/figure-double-layer-rnn.tex
...@@ -10,7 +10,7 @@ ...@@ -10,7 +10,7 @@
% RNN Encoder % RNN Encoder
\coordinate (eemb0) at (0,0); \coordinate (eemb0) at (0,0);
\foreach \x [count=\y from 0] in {1,2,...,10} \foreach \x [count=\y from 0] in {1,2,...,10}
\node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=west] (eemb\x) at ([xshift=0.4\base]eemb\y.east) {}; \node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=west] (eemb\x) at ([xshift=0.7\base]eemb\y.east) {};
\foreach \x in {1,2,...,10} \foreach \x in {1,2,...,10}
\node[rnnnode,fill=blue!30!white,anchor=south] (enc1\x) at ([yshift=0.3\base]eemb\x.north) {}; \node[rnnnode,fill=blue!30!white,anchor=south] (enc1\x) at ([yshift=0.3\base]eemb\x.north) {};
\foreach \x in {1,2,...,10} \foreach \x in {1,2,...,10}
...@@ -121,10 +121,10 @@ ...@@ -121,10 +121,10 @@
} }
\coordinate (bridge) at ([yshift=1.4\base]enc16.north west); \coordinate (bridge) at ([yshift=1.4\base]enc16.north west);
\draw[-latex'] (enc210.north) .. controls +(north:0.4\base) and +(east:1.5\base) .. (bridge) .. controls +(west:8.0\base) and +(south west:0.8\base) .. (dec21.west); \draw[-latex'] (enc210.north) .. controls +(north:0.6\base) and +(east:2\base) .. (bridge) .. controls +(west:8.0\base) and +(south west:2.8\base) .. (dec21.west);
\coordinate (bridge) at ([yshift=1.6\base]enc16.north west); \coordinate (bridge) at ([yshift=1.6\base]enc16.north west);
\draw[-latex'] (enc110.east) .. controls +(east:0.5\base) and +(east:8\base) .. (bridge) .. controls +(west:7.5\base) and +(south west:0.1\base) .. (dec11.west); \draw[-latex'] ([yshift=-0.5em]enc110.east) .. controls +(north:1.5\base) and +(east:8\base) .. (bridge) .. controls +(west:7.5\base) and +(south west:1.2\base) .. (dec11.west);
% stack RNN % stack RNN
\begin{pgfonlayer}{background} \begin{pgfonlayer}{background}
......
Chapter10/Figures/figure-encoder-decoder-with-attention.tex
...@@ -11,7 +11,7 @@ ...@@ -11,7 +11,7 @@
%%% a simple encoder-decoder model %%% a simple encoder-decoder model
\begin{scope} \begin{scope}
\foreach \x in {1,2,...,6} \foreach \x in {1,2,...,6}
\node[] (s\x) at (\x * 1.6em,0) {}; \node[] (s\x) at (\x * 2em,0) {};
\node [] (ws1) at (s1) {\scriptsize{这}}; \node [] (ws1) at (s1) {\scriptsize{这}};
\node [] (ws2) at (s2) {\scriptsize{是}}; \node [] (ws2) at (s2) {\scriptsize{是}};
...@@ -21,7 +21,7 @@ ...@@ -21,7 +21,7 @@
\node [] (ws6) at (s6) {\scriptsize{句子}}; \node [] (ws6) at (s6) {\scriptsize{句子}};
\foreach \x in {1,2,...,6} \foreach \x in {1,2,...,6}
\node[] (t\x) at (\x * 1.6em + 2.4in,0) {}; \node[] (t\x) at (\x * 2em + 2.8in,0) {};
\node [] (wt1) at (t1) {\scriptsize{This}}; \node [] (wt1) at (t1) {\scriptsize{This}};
\node [] (wt2) at (t2) {\scriptsize{is}}; \node [] (wt2) at (t2) {\scriptsize{is}};
...@@ -30,9 +30,9 @@ ...@@ -30,9 +30,9 @@
\node [] (wt5) at (t5) {\scriptsize{long}}; \node [] (wt5) at (t5) {\scriptsize{long}};
\node [] (wt6) at ([xshift=1em]t6) {\scriptsize{sentence}}; \node [] (wt6) at ([xshift=1em]t6) {\scriptsize{sentence}};
\node [anchor=south west,fill=red!30,minimum width=1.6in,minimum height=1.5em] (encoder) at ([yshift=1.0em]ws1.north west) {\footnotesize{Encoder}}; \node [anchor=south west,fill=red!30,minimum width=1.9in,minimum height=1.5em] (encoder) at ([yshift=1.0em,xshift=-0.4em]ws1.north west) {\footnotesize{Encoder}};
\node [anchor=west,fill=blue!30,minimum width=1.9in,minimum height=1.5em] (decoder) at ([xshift=4.5em]encoder.east) {\footnotesize{Decoder}}; \node [anchor=west,fill=green!30,minimum height=1.5em] (representation) at ([xshift=2em]encoder.east) {\footnotesize{表示}};
\node [anchor=west,fill=green!30,minimum height=1.5em] (representation) at ([xshift=1em]encoder.east) {\footnotesize{表示}}; \node [anchor=west,fill=blue!30,minimum width=1.9in,minimum height=1.5em] (decoder) at ([xshift=2em]representation.east) {\footnotesize{Decoder}};
\draw [->,thick] ([xshift=1pt]encoder.east)--([xshift=-1pt]representation.west); \draw [->,thick] ([xshift=1pt]encoder.east)--([xshift=-1pt]representation.west);
\draw [->,thick] ([xshift=1pt]representation.east)--([xshift=-1pt]decoder.west); \draw [->,thick] ([xshift=1pt]representation.east)--([xshift=-1pt]decoder.west);
...@@ -42,7 +42,7 @@ ...@@ -42,7 +42,7 @@
\foreach \x in {1,2,...,5} \foreach \x in {1,2,...,5}
\draw[<-] ([yshift=0.1em]t\x.north) -- ([yshift=1.2em]t\x.north); \draw[<-] ([yshift=0.1em]t\x.north) -- ([yshift=1.2em]t\x.north);
\draw[<-] ([yshift=0.1em,xshift=1em]t6.north) -- ([yshift=1.2em,xshift=1em]t6.north); \draw[<-,thick] ([yshift=0.1em,xshift=1em]t6.north) -- ([yshift=1.2em,xshift=1em]t6.north);
\node [anchor=north] (cap) at ([xshift=2em,yshift=-2.5em]encoder.south east) {\small{(a) 简单的编码器-解码器框架}}; \node [anchor=north] (cap) at ([xshift=2em,yshift=-2.5em]encoder.south east) {\small{(a) 简单的编码器-解码器框架}};
\end{scope} \end{scope}
...@@ -50,7 +50,7 @@ ...@@ -50,7 +50,7 @@
%%% a encoder-decoder model with attention %%% a encoder-decoder model with attention
\begin{scope}[yshift=-1.7in] \begin{scope}[yshift=-1.7in]
\foreach \x in {1,2,...,6} \foreach \x in {1,2,...,6}
\node[] (s\x) at (\x * 1.6em,0) {}; \node[] (s\x) at (\x * 2em,0) {};
\node [] (ws1) at (s1) {\scriptsize{这}}; \node [] (ws1) at (s1) {\scriptsize{这}};
\node [] (ws2) at (s2) {\scriptsize{是}}; \node [] (ws2) at (s2) {\scriptsize{是}};
...@@ -60,7 +60,7 @@ ...@@ -60,7 +60,7 @@
\node [] (ws6) at (s6) {\scriptsize{句子}}; \node [] (ws6) at (s6) {\scriptsize{句子}};
\foreach \x in {1,2,...,6} \foreach \x in {1,2,...,6}
\node[] (t\x) at (\x * 1.6em + 2.4in,0) {}; \node[] (t\x) at (\x * 2em + 2.8in,0) {};
\node [] (wt1) at (t1) {\scriptsize{This}}; \node [] (wt1) at (t1) {\scriptsize{This}};
\node [] (wt2) at (t2) {\scriptsize{is}}; \node [] (wt2) at (t2) {\scriptsize{is}};
...@@ -69,8 +69,8 @@ ...@@ -69,8 +69,8 @@
\node [] (wt5) at (t5) {\scriptsize{long}}; \node [] (wt5) at (t5) {\scriptsize{long}};
\node [] (wt6) at ([xshift=1em]t6) {\scriptsize{sentence}}; \node [] (wt6) at ([xshift=1em]t6) {\scriptsize{sentence}};
\node [anchor=south west,fill=red!30,minimum width=1.6in,minimum height=1.5em] (encoder) at ([yshift=1.0em]ws1.north west) {\footnotesize{Encoder}}; \node [anchor=south west,fill=red!30,minimum width=1.9in,minimum height=1.5em] (encoder) at ([yshift=1.0em,xshift=-0.4em]ws1.north west) {\footnotesize{Encoder}};
\node [anchor=west,fill=blue!30,minimum width=1.9in,minimum height=1.5em] (decoder) at ([xshift=4.5em]encoder.east) {\footnotesize{Decoder}}; \node [anchor=west,fill=blue!30,minimum width=1.9in,minimum height=1.5em] (decoder) at ([xshift=6.4em]encoder.east) {\footnotesize{Decoder}};
\foreach \x in {1,2,...,6} \foreach \x in {1,2,...,6}
\draw[->] ([yshift=0.1em]s\x.north) -- ([yshift=1.2em]s\x.north); \draw[->] ([yshift=0.1em]s\x.north) -- ([yshift=1.2em]s\x.north);
...@@ -78,11 +78,11 @@ ...@@ -78,11 +78,11 @@
\foreach \x in {1,2,...,5} \foreach \x in {1,2,...,5}
\draw[<-] ([yshift=0.1em]t\x.north) -- ([yshift=1.2em]t\x.north); \draw[<-] ([yshift=0.1em]t\x.north) -- ([yshift=1.2em]t\x.north);
\draw[<-] ([yshift=0.1em,xshift=1em]t6.north) -- ([yshift=1.2em,xshift=1em]t6.north); \draw[<-,thick] ([yshift=0.1em,xshift=1em]t6.north) -- ([yshift=1.2em,xshift=1em]t6.north);
\draw [->] ([yshift=3em]s6.north) -- ([yshift=4em]s6.north) -- ([yshift=4em]t1.north) node [pos=0.5,fill=green!30,inner sep=2pt] (c1) {\scriptsize{表示$\mathbi{C}_1$}} -- ([yshift=3em]t1.north) ; \draw [->,thick] ([yshift=3em]s6.north) -- ([yshift=4em]s6.north) -- ([yshift=4em]t1.north) node [pos=0.5,fill=green!30,inner sep=2pt] (c1) {\scriptsize{表示$\mathbi{C}_1$}} -- ([yshift=3em]t1.north) ;
\draw [->] ([yshift=3em]s5.north) -- ([yshift=5.3em]s5.north) -- ([yshift=5.3em]t2.north) node [pos=0.5,fill=green!30,inner sep=2pt] (c2) {\scriptsize{表示$\mathbi{C}_2$}} -- ([yshift=3em]t2.north) ; \draw [->,thick] ([yshift=3em]s5.north) -- ([yshift=5.3em]s5.north) -- ([yshift=5.3em]t2.north) node [pos=0.5,fill=green!30,inner sep=2pt] (c2) {\scriptsize{表示$\mathbi{C}_2$}} -- ([yshift=3em]t2.north) ;
\draw [->] ([yshift=3.5em]s3.north) -- ([yshift=6.6em]s3.north) -- ([yshift=6.6em]t4.north) node [pos=0.5,fill=green!30,inner sep=2pt] (c3) {\scriptsize{表示$\mathbi{C}_i$}} -- ([yshift=3.5em]t4.north) ; \draw [->,thick] ([yshift=3.5em]s3.north) -- ([yshift=6.6em]s3.north) -- ([yshift=6.6em]t4.north) node [pos=0.5,fill=green!30,inner sep=2pt] (c3) {\scriptsize{表示$\mathbi{C}_i$}} -- ([yshift=3.5em]t4.north) ;
\node [anchor=north] (smore) at ([yshift=3.5em]s3.north) {...}; \node [anchor=north] (smore) at ([yshift=3.5em]s3.north) {...};
\node [anchor=north] (tmore) at ([yshift=3.5em]t4.north) {...}; \node [anchor=north] (tmore) at ([yshift=3.5em]t4.north) {...};
......
Chapter10/Figures/figure-model-structure-based-on-recurrent-neural-network-translation.tex
...@@ -11,7 +11,7 @@ ...@@ -11,7 +11,7 @@
% RNN Encoder % RNN Encoder
\coordinate (eemb0) at (0,0); \coordinate (eemb0) at (0,0);
\foreach \x [count=\y from 0] in {1,2,...,10} \foreach \x [count=\y from 0] in {1,2,...,10}
\node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=west] (eemb\x) at ([xshift=0.4\base]eemb\y.east) {}; \node[rnnnode,minimum height=0.5\base,fill=green!30!white,anchor=west] (eemb\x) at ([xshift=0.7\base]eemb\y.east) {};
\foreach \x in {1,2,...,10} \foreach \x in {1,2,...,10}
\node[rnnnode,fill=blue!30!white,anchor=south] (enc\x) at ([yshift=0.5\base]eemb\x.north) {}; \node[rnnnode,fill=blue!30!white,anchor=south] (enc\x) at ([yshift=0.5\base]eemb\x.north) {};
\node[wordnode,left=0.4\base of enc1] (init) {$0$}; \node[wordnode,left=0.4\base of enc1] (init) {$0$};
...@@ -114,7 +114,7 @@ ...@@ -114,7 +114,7 @@
% legend % legend
\begin{scope}[shift={(10\base,2.5\base)}] \begin{scope}[shift={(10\base,2.5\base)}]
\node[rnnnode,minimum height=0.5\base,fill=green!30!white,label={[label distance=3pt,font=\scriptsize]0:词嵌入层}] (emb) at (0,0) {}; \node[rnnnode,minimum height=0.5\base,fill=green!30!white,label={[label distance=3pt,font=\scriptsize]0:词嵌入层}] (emb) at (3,0) {};
\node[rnnnode,fill=blue!30!white,anchor=north west,label={[label distance=3pt,font=\scriptsize]0:循环单元}] (rnn) at ([yshift=2\base]emb.south west) {}; \node[rnnnode,fill=blue!30!white,anchor=north west,label={[label distance=3pt,font=\scriptsize]0:循环单元}] (rnn) at ([yshift=2\base]emb.south west) {};
\node[rnnnode,minimum height=0.5\base,fill=red!30!white,anchor=north west,label={[label distance=3pt,font=\scriptsize]0:输出层}] (softmax) at ([yshift=2\base]rnn.south west) {}; \node[rnnnode,minimum height=0.5\base,fill=red!30!white,anchor=north west,label={[label distance=3pt,font=\scriptsize]0:输出层}] (softmax) at ([yshift=2\base]rnn.south west) {};
\node [anchor=north west] (softmax2) at ([xshift=0.6\base]softmax.south west) {\scriptsize{Softmax}}; \node [anchor=north west] (softmax2) at ([xshift=0.6\base]softmax.south west) {\scriptsize{Softmax}};
......
Chapter10/Figures/figure-numbers-of-wmt-systems.tex
...@@ -5,7 +5,7 @@ ...@@ -5,7 +5,7 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope}[local bounding box=WMT] \begin{scope}[local bounding box=WMT]
\draw[->,thick] (0.4,0) to (9.5,0); \draw[->,thick] (0.4,0) to (11.5,0);
\draw[->,thick] (0.4,-0) to (0.4,4.3); \draw[->,thick] (0.4,-0) to (0.4,4.3);
\draw[thick] (0.4,2) to (0.6,2); \draw[thick] (0.4,2) to (0.6,2);
\draw[thick] (0.4,4) to (0.6,4); \draw[thick] (0.4,4) to (0.6,4);
...@@ -13,38 +13,38 @@ ...@@ -13,38 +13,38 @@
\node[font=\scriptsize] at (0,4) {20}; \node[font=\scriptsize] at (0,4) {20};
% 2015 % 2015
\node[minimum width=0.5cm,thick,minimum height=7*0.2cm,fill=blue!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (smt2015) at (1.5*0.7,0.5pt) {}; \node[minimum width=0.7cm,thick,minimum height=7*0.2cm,fill=blue!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (smt2015) at (1.5*0.7,0.5pt) {};
\node[minimum width=0.5cm,thick,minimum height=2*0.2cm,fill=red!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (nmt2015) at (smt2015.south east) {}; \node[minimum width=0.7cm,thick,minimum height=2*0.2cm,fill=red!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (nmt2015) at (smt2015.south east) {};
\node[font=\scriptsize,anchor=north] () at ([yshift=-0.2em]smt2015.south east) {2015}; \node[font=\scriptsize,anchor=north] () at ([yshift=-0.2em]smt2015.south east) {2015};
% 2016 % 2016
\node[minimum width=0.5cm,thick,minimum height=3*0.2cm,fill=blue!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (smt2016) at ($(nmt2015.south east)+(0.7,0)$) {}; \node[minimum width=0.7cm,thick,minimum height=3*0.2cm,fill=blue!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (smt2016) at ($(nmt2015.south east)+(0.7,0)$) {};
\node[minimum width=0.5cm,thick,minimum height=8*0.2cm,fill=red!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (nmt2016) at (smt2016.south east) {}; \node[minimum width=0.7cm,thick,minimum height=8*0.2cm,fill=red!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (nmt2016) at (smt2016.south east) {};
\node[font=\scriptsize,anchor=north] () at ([yshift=-0.2em]smt2016.south east) {2016}; \node[font=\scriptsize,anchor=north] () at ([yshift=-0.2em]smt2016.south east) {2016};
% 2017 % 2017
\node[minimum width=0.5cm,thick,minimum height=3*0.2cm,fill=blue!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (smt2017) at ($(nmt2016.south east)+(0.7,0)$) {}; \node[minimum width=0.7cm,thick,minimum height=3*0.2cm,fill=blue!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (smt2017) at ($(nmt2016.south east)+(0.7,0)$) {};
\node[minimum width=0.5cm,thick,minimum height=13*0.2cm,fill=red!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (nmt2017) at (smt2017.south east) {}; \node[minimum width=0.7cm,thick,minimum height=13*0.2cm,fill=red!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (nmt2017) at (smt2017.south east) {};
\node[font=\scriptsize,anchor=north] () at ([yshift=-0.2em]smt2017.south east) {2017}; \node[font=\scriptsize,anchor=north] () at ([yshift=-0.2em]smt2017.south east) {2017};
% 2018 % 2018
\node[minimum width=0.5cm,thick,minimum height=0cm,draw,fill=blue!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (smt2018) at ($(nmt2017.south east)+(0.7,0)$) {}; \node[minimum width=0.7cm,thick,minimum height=0cm,draw,fill=blue!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (smt2018) at ($(nmt2017.south east)+(0.7,0)$) {};
\node[minimum width=0.5cm,thick,minimum height=14*0.2cm,fill=red!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (nmt2018) at (smt2018.south east) {}; \node[minimum width=0.7cm,thick,minimum height=14*0.2cm,fill=red!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (nmt2018) at (smt2018.south east) {};
\node[font=\scriptsize,anchor=north] () at ([yshift=-0.2em]smt2018.south east) {2018}; \node[font=\scriptsize,anchor=north] () at ([yshift=-0.2em]smt2018.south east) {2018};
% 2019 % 2019
\node[minimum width=0.5cm,thick,minimum height=0cm,draw,fill=blue!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (smt2019) at ($(nmt2018.south east)+(0.7,0)$) {}; \node[minimum width=0.7cm,thick,minimum height=0cm,draw,fill=blue!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (smt2019) at ($(nmt2018.south east)+(0.7,0)$) {};
\node[minimum width=0.5cm,thick,minimum height=21*0.2cm,fill=red!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (nmt2019) at (smt2019.south east) {}; \node[minimum width=0.7cm,thick,minimum height=21*0.2cm,fill=red!30!white,inner sep=0pt,outer sep=0pt,anchor=south west] (nmt2019) at (smt2019.south east) {};
\node[font=\scriptsize,anchor=north] () at ([yshift=-0.2em]smt2019.south east) {2019}; \node[font=\scriptsize,anchor=north] () at ([yshift=-0.2em]smt2019.south east) {2019};
\end{scope} \end{scope}
% legend % legend
\ExtractX{$(nmt2015.west)$} \ExtractX{$(nmt2015.west)$}
\ExtractY{$(WMT.north)$} \ExtractY{$(WMT.north)$}
\node[minimum width=0.5cm,rectangle,fill=blue!30!white,anchor=north west,label={[label distance=1pt,font=\scriptsize]0:统计机器翻译}] () at (\XCoord,\YCoord) {}; \node[minimum width=0.7cm,rectangle,fill=blue!30!white,anchor=north west,label={[label distance=1pt,font=\scriptsize]0:统计机器翻译}] () at (\XCoord,\YCoord) {};
\ExtractX{$(nmt2017.west)$} \ExtractX{$(nmt2017.west)$}
\node[minimum width=0.5cm,rectangle,fill=red!30!white,anchor=north west,label={[label distance=1pt,font=\scriptsize]0:神经机器翻译}] () at (\XCoord,\YCoord) {}; \node[minimum width=0.7cm,rectangle,fill=red!30!white,anchor=north west,label={[label distance=1pt,font=\scriptsize]0:神经机器翻译}] () at (\XCoord,\YCoord) {};
% \node[font=\normalsize,rotate=90] () at ([xshift=-1em]WMT.west) {数量}; % \node[font=\normalsize,rotate=90] () at ([xshift=-1em]WMT.west) {数量};
\node[font=\normalsize] () at (0.4,4.5) {数量}; \node[font=\normalsize] () at (0.4,4.5) {数量};
\node[font=\normalsize] () at (9.5,-0.3) {年份}; \node[font=\normalsize] () at (11.5,-0.3) {年份};
\end{tikzpicture} \end{tikzpicture}
\ No newline at end of file
Chapter10/Figures/figure-presentation-space.tex
...@@ -19,13 +19,13 @@ ...@@ -19,13 +19,13 @@
\end{scope} \end{scope}
\begin{scope}[xshift=1.3in] \begin{scope}[xshift=1.8in]
\node [anchor=south west,draw,thick,red,minimum width=0.9in,minimum height=0.7in] (space1) at (0,0) {}; \node [anchor=south west,draw,thick,red,minimum width=0.9in,minimum height=0.7in] (space1) at (0,0) {};
\node [anchor=south west,fill=blue,minimum width=0.1in,minimum height=0.1in] (unit1) at (0.2,0.8) {}; \node [anchor=south west,fill=blue,minimum width=0.1in,minimum height=0.1in] (unit1) at (0.2,0.8) {};
\node [anchor=south west,fill=ugreen,minimum width=0.1in,minimum height=0.1in] (unit2) at (0.7,0.3) {}; \node [anchor=south west,fill=ugreen,minimum width=0.1in,minimum height=0.1in] (unit2) at (0.7,0.3) {};
\node [anchor=south west,draw,thick,red,minimum width=0.9in,minimum height=0.7in] (space2) at (1.1in,0) {}; \node [anchor=south west,draw,thick,red,minimum width=0.9in,minimum height=0.7in] (space2) at (1.3in,0) {};
\node [anchor=south west,circle,fill=orange,minimum width=0.1in,minimum height=0.1in] (unit3) at (1.5in,1.3) {}; \node [anchor=south west,circle,fill=orange,minimum width=0.1in,minimum height=0.1in] (unit3) at (1.7in,1.3) {};
\draw [->] ([yshift=1pt]unit1.north) .. controls +(north:0.4) and +(west:2) .. ([yshift=0.0em,xshift=-1pt]unit3.west); \draw [->] ([yshift=1pt]unit1.north) .. controls +(north:0.4) and +(west:2) .. ([yshift=0.0em,xshift=-1pt]unit3.west);
\draw [->] ([xshift=1pt]unit2.east) .. controls +(east:1.5) and +(south:1) .. ([xshift=0.0em,yshift=-1pt]unit3.south); \draw [->] ([xshift=1pt]unit2.east) .. controls +(east:1.5) and +(south:1) .. ([xshift=0.0em,yshift=-1pt]unit3.south);
......
Chapter10/Figures/figure-query-model-corresponding-to-attention-mechanism.tex
...@@ -5,26 +5,26 @@ ...@@ -5,26 +5,26 @@
\tikzstyle{rnode} = [draw,minimum width=3.5em,minimum height=1.2em] \tikzstyle{rnode} = [draw,minimum width=3.5em,minimum height=1.2em]
\node [rnode,anchor=south west,fill=red!20!white] (value1) at (0,0) {\scriptsize{${h}(\textrm{你})$}}; \node [rnode,anchor=south west,fill=red!20!white] (value1) at (0,0) {\scriptsize{${h}(\textrm{你})$}};
\node [rnode,anchor=south west,fill=red!20!white] (value2) at ([xshift=1em]value1.south east) {\scriptsize{${h}(\textrm{什么})$}}; \node [rnode,anchor=south west,fill=red!20!white] (value2) at ([xshift=2em]value1.south east) {\scriptsize{${h}(\textrm{什么})$}};
\node [rnode,anchor=south west,fill=red!20!white] (value3) at ([xshift=1em]value2.south east) {\scriptsize{${h}(\textrm{也})$}}; \node [rnode,anchor=south west,fill=red!20!white] (value3) at ([xshift=2em]value2.south east) {\scriptsize{${h}(\textrm{也})$}};
\node [rnode,anchor=south west,fill=red!20!white] (value4) at ([xshift=1em]value3.south east) {\scriptsize{${h}(\textrm{没})$}}; \node [rnode,anchor=south west,fill=red!20!white] (value4) at ([xshift=2em]value3.south east) {\scriptsize{${h}(\textrm{没})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key1) at ([yshift=0.2em]value1.north west) {\scriptsize{${h}(\textrm{你})$}}; \node [rnode,anchor=south west,fill=green!20!white] (key1) at ([yshift=0.2em]value1.north west) {\scriptsize{${h}(\textrm{你})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key2) at ([yshift=0.2em]value2.north west) {\scriptsize{${h}(\textrm{什么})$}}; \node [rnode,anchor=south west,fill=green!20!white] (key2) at ([yshift=0.2em]value2.north west) {\scriptsize{${h}(\textrm{什么})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key3) at ([yshift=0.2em]value3.north west) {\scriptsize{${h}(\textrm{也})$}}; \node [rnode,anchor=south west,fill=green!20!white] (key3) at ([yshift=0.2em]value3.north west) {\scriptsize{${h}(\textrm{也})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key4) at ([yshift=0.2em]value4.north west) {\scriptsize{${h}(\textrm{没})$}}; \node [rnode,anchor=south west,fill=green!20!white] (key4) at ([yshift=0.2em]value4.north west) {\scriptsize{${h}(\textrm{没})$}};
\node [rnode,anchor=east] (query) at ([xshift=-2em]key1.west) {\scriptsize{${s}(\textrm{you})$}}; \node [rnode,anchor=east] (query) at ([xshift=-4em]key1.west) {\scriptsize{${s}(\textrm{you})$}};
\node [anchor=east] (querylabel) at ([xshift=-0.2em]query.west) {\scriptsize{query}}; \node [anchor=east] (querylabel) at ([xshift=-0.2em]query.west) {\scriptsize{query}};
\draw [->] ([yshift=1pt,xshift=6pt]query.north) .. controls +(90:1em) and +(90:1em) .. ([yshift=1pt]key1.north); \draw [->] ([yshift=1pt,xshift=6pt]query.north) .. controls +(90:1em) and +(90:1em) .. ([yshift=1pt]key1.north);
\draw [->] ([yshift=1pt,xshift=3pt]query.north) .. controls +(90:1.5em) and +(90:1.5em) .. ([yshift=1pt]key2.north); \draw [->] ([yshift=1pt,xshift=3pt]query.north) .. controls +(90:1.5em) and +(90:1.5em) .. ([yshift=1pt]key2.north);
\draw [->] ([yshift=1pt]query.north) .. controls +(90:2em) and +(90:2em) .. ([yshift=1pt]key3.north); \draw [->] ([yshift=1pt]query.north) .. controls +(90:2em) and +(90:2em) .. ([yshift=1pt]key3.north);
\draw [->] ([yshift=1pt,xshift=-3pt]query.north) .. controls +(90:2.5em) and +(90:2.5em) .. ([yshift=1pt]key4.north); \draw [->] ([yshift=1pt,xshift=-3pt]query.north) .. controls +(90:2.5em) and +(90:2.5em) .. ([yshift=1pt]key4.north);
\node [anchor=south east] (alpha1) at ([xshift=1em]key1.north east) {\scriptsize{$\alpha_1=.4$}}; \node [anchor=south east] (alpha1) at ([xshift=1.5em]key1.north east) {\scriptsize{$\alpha_1=0.4$}};
\node [anchor=south east] (alpha2) at ([xshift=1em]key2.north east) {\scriptsize{$\alpha_2=.4$}}; \node [anchor=south east] (alpha2) at ([xshift=1.5em]key2.north east) {\scriptsize{$\alpha_2=0.4$}};
\node [anchor=south east] (alpha3) at ([xshift=1em]key3.north east) {\scriptsize{$\alpha_3=.1$}}; \node [anchor=south east] (alpha3) at ([xshift=1.5em]key3.north east) {\scriptsize{$\alpha_3=0.1$}};
\node [anchor=south east] (alpha4) at ([xshift=1em]key4.north east) {\scriptsize{$\alpha_4=.1$}}; \node [anchor=south east] (alpha4) at ([xshift=1.5em]key4.north east) {\scriptsize{$\alpha_4=0.1$}};
\end{scope} \end{scope}
\end{tikzpicture} \end{tikzpicture}
\ No newline at end of file
Chapter10/Figures/figure-query-model-corresponding-to-traditional-query-model-vs-attention-mechanism.tex
...@@ -5,12 +5,12 @@ ...@@ -5,12 +5,12 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{rnode} = [draw,minimum width=3em,minimum height=1.2em] \tikzstyle{rnode} = [draw,minimum width=3.5em,minimum height=1.2em]
\node [rnode,anchor=south west,fill=blue!20!white] (value1) at (0,0) {\scriptsize{value$_1$}}; \node [rnode,anchor=south west,fill=blue!20!white] (value1) at (0,0) {\scriptsize{value$_1$}};
\node [rnode,anchor=south west,fill=blue!20!white] (value2) at ([xshift=1em]value1.south east) {\scriptsize{value$_2$}}; \node [rnode,anchor=south west,fill=blue!20!white] (value2) at ([xshift=2em]value1.south east) {\scriptsize{value$_2$}};
\node [rnode,anchor=south west,fill=red!20!white] (value3) at ([xshift=1em]value2.south east) {\scriptsize{value$_3$}}; \node [rnode,anchor=south west,fill=red!20!white] (value3) at ([xshift=2em]value2.south east) {\scriptsize{value$_3$}};
\node [rnode,anchor=south west,fill=blue!20!white] (value4) at ([xshift=1em]value3.south east) {\scriptsize{value$_4$}}; \node [rnode,anchor=south west,fill=blue!20!white] (value4) at ([xshift=2em]value3.south east) {\scriptsize{value$_4$}};
\node [rnode,anchor=south west,pattern=north east lines] (key1) at ([yshift=0.2em]value1.north west) {}; \node [rnode,anchor=south west,pattern=north east lines] (key1) at ([yshift=0.2em]value1.north west) {};
\node [rnode,anchor=south west,pattern=dots] (key2) at ([yshift=0.2em]value2.north west) {}; \node [rnode,anchor=south west,pattern=dots] (key2) at ([yshift=0.2em]value2.north west) {};
...@@ -21,7 +21,7 @@ ...@@ -21,7 +21,7 @@
\node [fill=white,inner sep=1pt] (key1label) at (key3) {\scriptsize{key$_3$}}; \node [fill=white,inner sep=1pt] (key1label) at (key3) {\scriptsize{key$_3$}};
\node [fill=white,inner sep=1pt] (key1label) at (key4) {\scriptsize{key$_4$}}; \node [fill=white,inner sep=1pt] (key1label) at (key4) {\scriptsize{key$_4$}};
\node [rnode,anchor=east,pattern=horizontal lines] (query) at ([xshift=-3em]key1.west) {}; \node [rnode,anchor=east,pattern=horizontal lines] (query) at ([xshift=-4em]key1.west) {};
\node [anchor=east] (querylabel) at ([xshift=-0.2em]query.west) {\scriptsize{query}}; \node [anchor=east] (querylabel) at ([xshift=-0.2em]query.west) {\scriptsize{query}};
\draw [->] ([yshift=1pt]query.north) .. controls +(90:2em) and +(90:2em) .. ([yshift=1pt]key3.north) node [pos=0.5,below,yshift=0.2em] {\scriptsize{匹配}}; \draw [->] ([yshift=1pt]query.north) .. controls +(90:2em) and +(90:2em) .. ([yshift=1pt]key3.north) node [pos=0.5,below,yshift=0.2em] {\scriptsize{匹配}};
......
Chapter10/Figures/figure-relationship-between-learning-rate-and-number-of-updates.tex
...@@ -4,7 +4,7 @@ ...@@ -4,7 +4,7 @@
\begin{tikzpicture} \begin{tikzpicture}
\footnotesize{ \footnotesize{
\begin{axis}[ \begin{axis}[
width=.60\textwidth, width=.8\textwidth,
height=.40\textwidth, height=.40\textwidth,
legend style={at={(0.60,0.08)}, anchor=south west}, legend style={at={(0.60,0.08)}, anchor=south west},
xlabel={\scriptsize{更新次数}}, xlabel={\scriptsize{更新次数}},
......
Chapter10/Figures/figure-score-of-mter.tex
...@@ -2,7 +2,7 @@ ...@@ -2,7 +2,7 @@
%\definecolor{ublue}{rgb}{0.152,0.250,0.545} %\definecolor{ublue}{rgb}{0.152,0.250,0.545}
\begin{tikzpicture} \begin{tikzpicture}
\begin{axis}[ \begin{axis}[
width=10cm, height=5cm, width=12cm, height=5cm,
symbolic x coords={1-15,16-25,26-35,>35}, symbolic x coords={1-15,16-25,26-35,>35},
xtick=data, xtick=data,
ytick={6,12,...,28}, ytick={6,12,...,28},
......
Chapter10/Figures/figure-structure-of-a-recurrent-network-model.tex
...@@ -6,11 +6,11 @@ ...@@ -6,11 +6,11 @@
\begin{scope}[scale=0.9] \begin{scope}[scale=0.9]
{\Large {\Large
\tikzstyle{rnnnode} = [draw,inner sep=5pt,minimum width=3em,minimum height=0.8em,fill=green!30!white,blur shadow={shadow xshift=1pt,shadow yshift=-1pt}] \tikzstyle{rnnnode} = [draw,inner sep=5pt,minimum width=3.6em,minimum height=0.8em,fill=green!30!white,blur shadow={shadow xshift=1pt,shadow yshift=-1pt}]
\node [anchor=west,rnnnode] (node11) at (0,0) {\scriptsize{RNN Cell}}; \node [anchor=west,rnnnode] (node11) at (0,0) {\scriptsize{RNN Cell}};
\node [anchor=west,rnnnode] (node12) at ([xshift=1em]node11.east) {\scriptsize{RNN Cell}}; \node [anchor=west,rnnnode] (node12) at ([xshift=2em]node11.east) {\scriptsize{RNN Cell}};
\node [anchor=west,rnnnode] (node13) at ([xshift=1em]node12.east) {\scriptsize{RNN Cell}}; \node [anchor=west,rnnnode] (node13) at ([xshift=2em]node12.east) {\scriptsize{RNN Cell}};
\node [anchor=west,rnnnode] (node14) at ([xshift=1em]node13.east) {\scriptsize{RNN Cell}}; \node [anchor=west,rnnnode] (node14) at ([xshift=2em]node13.east) {\scriptsize{RNN Cell}};
\node [anchor=north,rnnnode,fill=blue!30!white] (e1) at ([yshift=-1em]node11.south) {\scriptsize{}}; \node [anchor=north,rnnnode,fill=blue!30!white] (e1) at ([yshift=-1em]node11.south) {\scriptsize{}};
\node [anchor=north,rnnnode,fill=blue!30!white] (e2) at ([yshift=-1em]node12.south) {\scriptsize{}}; \node [anchor=north,rnnnode,fill=blue!30!white] (e2) at ([yshift=-1em]node12.south) {\scriptsize{}};
......
Chapter10/Figures/figure-structure-of-gnmt.tex
...@@ -12,9 +12,9 @@ ...@@ -12,9 +12,9 @@
% Encoder % Encoder
\begin{scope} \begin{scope}
\node[rnnnode,fill=green!20] (encemb1) at (0,0) {}; \node[rnnnode,fill=green!20] (encemb1) at (0,0) {};
\node[rnnnode,fill=green!20,right=\base of encemb1] (encemb2) {}; \node[rnnnode,fill=green!20,right=1.8\base of encemb1] (encemb2) {};
\node[rnnnode,draw=white,fill=white,right=\base of encemb2] (encemb3) {$\cdots$}; \node[rnnnode,draw=white,fill=white,right=1.8\base of encemb2] (encemb3) {$\cdots$};
\node[rnnnode,fill=green!20,right=\base of encemb3] (encemb4) {}; \node[rnnnode,fill=green!20,right=1.8\base of encemb3] (encemb4) {};
\node[rnnnode,above=\base of encemb1] (enc11) {}; \node[rnnnode,above=\base of encemb1] (enc11) {};
\node[rnnnode,above=\base of encemb2] (enc12) {}; \node[rnnnode,above=\base of encemb2] (enc12) {};
...@@ -103,10 +103,10 @@ ...@@ -103,10 +103,10 @@
\node[rnnnode,fill=orange!20,minimum width=3.5cm,anchor=south west] (attention) at ([yshift=\base]enc61.north west) {注意力机制}; \node[rnnnode,fill=orange!20,minimum width=3.5cm,anchor=south west] (attention) at ([yshift=\base]enc61.north west) {注意力机制};
\begin{scope} \begin{scope}
\node[rnnnode,fill=green!20,right=2.5cm of encemb4] (decemb1) {}; \node[rnnnode,fill=green!20,right=3cm of encemb4] (decemb1) {};
\node[rnnnode,fill=green!20,right=\base of decemb1] (decemb2) {}; \node[rnnnode,fill=green!20,right=1.8\base of decemb1] (decemb2) {};
\node[rnnnode,draw=white,fill=white,right=\base of decemb2] (decemb3) {$\cdots$}; \node[rnnnode,draw=white,fill=white,right=1.8\base of decemb2] (decemb3) {$\cdots$};
\node[rnnnode,fill=green!20,right=\base of decemb3] (decemb4) {}; \node[rnnnode,fill=green!20,right=1.8\base of decemb3] (decemb4) {};
\node[rnnnode,above=\base of decemb1] (dec11) {}; \node[rnnnode,above=\base of decemb1] (dec11) {};
\node[rnnnode,above=\base of decemb2] (dec12) {}; \node[rnnnode,above=\base of decemb2] (dec12) {};
...@@ -189,7 +189,7 @@ ...@@ -189,7 +189,7 @@
\end{scope} \end{scope}
% attention connections % attention connections
\draw[-latex',rounded corners=2pt] (dec11) -| ([xshift=-0.4cm]attention.south east); \draw[-latex',rounded corners=2pt] (dec11) -| ([xshift=-0.2cm]attention.south east);
\ExtractX{$([xshift=9pt]attention.east)$} \ExtractX{$([xshift=9pt]attention.east)$}
\ExtractY{$([yshift=2pt]dec11.north)$} \ExtractY{$([yshift=2pt]dec11.north)$}
...@@ -224,11 +224,11 @@ ...@@ -224,11 +224,11 @@
\draw[decorate,decoration={brace,mirror}] ([xshift=5pt]dec14.south east) to node [auto,swap,font=\scriptsize,name=label2] {8层} ([xshift=5pt]dec54.north east); \draw[decorate,decoration={brace,mirror}] ([xshift=5pt]dec14.south east) to node [auto,swap,font=\scriptsize,name=label2] {8层} ([xshift=5pt]dec54.north east);
\begin{pgfonlayer}{background} \begin{pgfonlayer}{background}
\coordinate (tmp) at ([xshift=-4pt]label1.west); \coordinate (tmp) at ([xshift=-4pt]label1.west);
\node[draw,densely dashed,rounded corners=2pt,inner sep=2pt,fit=(label1) (encword1) (attention) (tmp)] (encoder) {}; \node[thick,draw,densely dashed,rounded corners=2pt,inner sep=2pt,fit=(label1) (encword1) (attention) (tmp)] (encoder) {};
\ExtractX{$([xshift=4pt]label2.east)$} \ExtractX{$([xshift=4pt]label2.east)$}
\ExtractY{$([yshift=6pt]decoutword4.north)$} \ExtractY{$([yshift=6pt]decoutword4.north)$}
\coordinate (tmp) at (\XCoord,\YCoord); \coordinate (tmp) at (\XCoord,\YCoord);
\node[draw,densely dashed,rounded corners=2pt,inner sep=2pt,fit=(label2) (decinword1) (decoutword4) (tmp)] (decoder) {}; \node[thick,draw,densely dashed,rounded corners=2pt,inner sep=2pt,fit=(label2) (decinword1) (decoutword4) (tmp)] (decoder) {};
\end{pgfonlayer} \end{pgfonlayer}
\node[wnode,anchor=north west] () at (encoder.north west) {编码器}; \node[wnode,anchor=north west] () at (encoder.north west) {编码器};
\node[wnode,anchor=north east] () at (decoder.north east) {解码器}; \node[wnode,anchor=north east] () at (decoder.north east) {解码器};
......
Chapter11/Figures/figure-average-pooling.tex
...@@ -2,32 +2,32 @@ ...@@ -2,32 +2,32 @@
\begin{tikzpicture}[node distance = 0cm] \begin{tikzpicture}[node distance = 0cm]
\node(num1)[num,fill=red!10]{1}; \node(num1)[num,fill=red!20]{1};
\node(num2)[num,below of = num1,yshift= -0.6cm,fill=red!10]{5}; \node(num2)[num,below of = num1,yshift= -0.6cm,fill=red!20]{5};
\node(num3)[num,right of = num1,xshift= 0.6cm,fill=red!10]{0}; \node(num3)[num,right of = num1,xshift= 0.6cm,fill=red!20]{0};
\node(num4)[num,below of = num3,yshift= -0.6cm,fill=red!10]{6}; \node(num4)[num,below of = num3,yshift= -0.6cm,fill=red!20]{6};
\node(num5)[num,right of = num3,xshift= 0.6cm,fill=green!10]{4}; \node(num5)[num,right of = num3,xshift= 0.6cm,fill=green!20]{4};
\node(num6)[num,below of = num5,yshift= -0.6cm,fill=green!10]{7}; \node(num6)[num,below of = num5,yshift= -0.6cm,fill=green!20]{7};
\node(num7)[num,right of = num5,xshift= 0.6cm,fill=green!10]{5}; \node(num7)[num,right of = num5,xshift= 0.6cm,fill=green!20]{5};
\node(num8)[num,below of = num7,yshift= -0.6cm,fill=green!10]{8}; \node(num8)[num,below of = num7,yshift= -0.6cm,fill=green!20]{8};
\node(num9)[num,below of = num2,yshift= -0.6cm,fill=yellow!10]{3}; \node(num9)[num,below of = num2,yshift= -0.6cm,fill=yellow!20]{3};
\node(num10)[num,below of = num9,yshift= -0.6cm,fill=yellow!10]{1}; \node(num20)[num,below of = num9,yshift= -0.6cm,fill=yellow!20]{1};
\node(num11)[num,right of = num9,xshift= 0.6cm,fill=yellow!10]{2}; \node(num11)[num,right of = num9,xshift= 0.6cm,fill=yellow!20]{2};
\node(num12)[num,below of = num11,yshift= -0.6cm,fill=yellow!10]{2}; \node(num12)[num,below of = num11,yshift= -0.6cm,fill=yellow!20]{2};
\node(num13)[num,right of = num11,xshift= 0.6cm,fill=blue!10]{1}; \node(num13)[num,right of = num11,xshift= 0.6cm,fill=blue!20]{1};
\node(num14)[num,below of = num13,yshift= -0.6cm,fill=blue!10]{3}; \node(num14)[num,below of = num13,yshift= -0.6cm,fill=blue!20]{3};
\node(num10)[num,right of = num13,xshift= 0.6cm,fill=blue!10]{0}; \node(num20)[num,right of = num13,xshift= 0.6cm,fill=blue!20]{0};
\node(num16)[num,below of = num10,yshift= -0.6cm,fill=blue!10]{4}; \node(num16)[num,below of = num20,yshift= -0.6cm,fill=blue!20]{4};
\draw[->,thick]([xshift=0.4cm,yshift=-0.4cm]num8.east)--([xshift=1.5cm,yshift=-0.4cm]num8.east); \draw[->,thick]([xshift=0.4cm,yshift=-0.4cm]num8.east)--([xshift=1.5cm,yshift=-0.4cm]num8.east);
\node(num17)[num,right of = num8,xshift= 2.5cm,fill=red!10]{3}; \node(num17)[num,right of = num8,xshift= 2.5cm,fill=red!20]{3};
\node(num18)[num,right of = num17,xshift= 0.6cm,fill=green!10]{6}; \node(num18)[num,right of = num17,xshift= 0.6cm,fill=green!20]{6};
\node(num19)[num,below of = num17,yshift=-0.6cm,fill=yellow!10]{2}; \node(num19)[num,below of = num17,yshift=-0.6cm,fill=yellow!20]{2};
\node(num20)[num,below of = num18,yshift= -0.6cm,fill=blue!10]{2}; \node(num20)[num,below of = num18,yshift= -0.6cm,fill=blue!20]{2};
\node [right of = num2,xshift= -0.7cm]{}; \node [right of = num2,xshift= -0.7cm]{};
......
Chapter11/Figures/figure-convolution-kernel.tex
\usetikzlibrary{decorations.pathreplacing} \usetikzlibrary{decorations.pathreplacing}
\tikzstyle{num} = [rectangle, minimum width = 0.8cm, minimum height = 0.8cm, text centered,align=center,thick,draw = black,fill=green!5] \tikzstyle{num} = [rectangle, minimum width = 0.8cm, minimum height = 0.8cm, text centered,align=center,thick,draw = black,fill=green!15]
\tikzstyle{att} = [rectangle, minimum width = 0.8cm, minimum height = 0.8cm, text centered,align=center] \tikzstyle{att} = [rectangle, minimum width = 0.8cm, minimum height = 0.8cm, text centered,align=center]
......
Chapter11/Figures/figure-deep-vs-light.tex
...@@ -30,7 +30,7 @@ ...@@ -30,7 +30,7 @@
\node [anchor=north,font=\tiny,scale=1.1] at ([yshift=-0.2em]t2.south) {深度卷积}; \node [anchor=north,font=\tiny,scale=1.1] at ([yshift=-0.2em]t2.south) {深度卷积};
\end{scope} \end{scope}
\begin{scope}[xshift=4cm] \begin{scope}[xshift=5cm]
\foreach \point in {0,1,2,3}{ \foreach \point in {0,1,2,3}{
\node[unit] (l1_\point) at (0, 0+\point*1.5em){}; \node[unit] (l1_\point) at (0, 0+\point*1.5em){};
\node[unit] (l2_\point) at (0, 8em+\point*1.5em){}; \node[unit] (l2_\point) at (0, 8em+\point*1.5em){};
......
Chapter11/Figures/figure-dimension-transformation.tex
\tikzstyle{num} = [minimum width = 0.6cm,minimum height = 0.6cm,draw,fill=green!10] \tikzstyle{num} = [minimum width = 0.6cm,minimum height = 0.6cm,draw,fill=green!20]
\begin{tikzpicture}[node distance = 0] \begin{tikzpicture}[node distance = 0]
\node[num] at (0,0){1}; \node[num] at (0,0){1};
...@@ -43,7 +43,7 @@ ...@@ -43,7 +43,7 @@
\node[num] at (2.4,3){2}; \node[num] at (2.4,3){2};
\node[num] at (3,3){1}; \node[num] at (3,3){1};
\node[minimum width = 1.8cm,minimum height = 1.8cm,draw=purple!40,line width=0.08cm,fill=purple!40,fill opacity=0.4] at (0.6,2.4) {}; \node[minimum width = 1.8cm,minimum height = 1.8cm,draw=purple!40,line width=0.08cm,fill=purple!50,fill opacity=0.4] at (0.6,2.4) {};
%\fill (4,1.5) circle (2pt); %\fill (4,1.5) circle (2pt);
\node [] at (4,1.5) {*}; \node [] at (4,1.5) {*};
......
Chapter11/Figures/figure-full-connection-vs-cnn-a.tex
\begin{tikzpicture}[node distance = 0] \begin{tikzpicture}[node distance = 0]
\foreach \x in {0,1,...,6}{ \foreach \x in {0,1,...,6}{
\draw[color=ublue,fill=blue!15,thick]( \x,0 )circle( 0.3); \draw[color=ublue,fill=blue!25,thick]( \x,0 )circle( 0.3);
} }
\foreach \x in {1,2,...,5}{ \foreach \x in {1,2,...,5}{
\draw[color=ublue,thick]( \x,2)circle( 0.3); \draw[color=ublue,thick]( \x,2)circle( 0.3);
......
Chapter11/Figures/figure-full-connection-vs-cnn-b.tex
\begin{tikzpicture}[node distance = 0] \begin{tikzpicture}[node distance = 0]
\foreach \x in {0,1,...,6}{ \foreach \x in {0,1,...,6}{
\draw[color=ublue,fill=blue!15,thick]( \x,0 )circle( 0.3); \draw[color=ublue,fill=blue!25,thick]( \x,0 )circle( 0.3);
} }
\foreach \x in {1,2,...,5}{ \foreach \x in {1,2,...,5}{
\draw[color=ublue,thick]( \x,2)circle( 0.3); \draw[color=ublue,thick]( \x,2)circle( 0.3);
......
Chapter11/Figures/figure-max-pooling.tex
...@@ -2,32 +2,32 @@ ...@@ -2,32 +2,32 @@
\begin{tikzpicture}[node distance = 0cm] \begin{tikzpicture}[node distance = 0cm]
\node(num1)[num,fill=red!10]{1}; \node(num1)[num,fill=red!20]{1};
\node(num2)[num,below of = num1,yshift= -0.6cm,fill=red!10]{5}; \node(num2)[num,below of = num1,yshift= -0.6cm,fill=red!20]{5};
\node(num3)[num,right of = num1,xshift= 0.6cm,fill=red!10]{0}; \node(num3)[num,right of = num1,xshift= 0.6cm,fill=red!20]{0};
\node(num4)[num,below of = num3,yshift= -0.6cm,fill=red!10]{6}; \node(num4)[num,below of = num3,yshift= -0.6cm,fill=red!20]{6};
\node(num5)[num,right of = num3,xshift= 0.6cm,fill=green!10]{4}; \node(num5)[num,right of = num3,xshift= 0.6cm,fill=green!20]{4};
\node(num6)[num,below of = num5,yshift= -0.6cm,fill=green!10]{7}; \node(num6)[num,below of = num5,yshift= -0.6cm,fill=green!20]{7};
\node(num7)[num,right of = num5,xshift= 0.6cm,fill=green!10]{5}; \node(num7)[num,right of = num5,xshift= 0.6cm,fill=green!20]{5};
\node(num8)[num,below of = num7,yshift= -0.6cm,fill=green!10]{8}; \node(num8)[num,below of = num7,yshift= -0.6cm,fill=green!20]{8};
\node(num9)[num,below of = num2,yshift= -0.6cm,fill=yellow!10]{3}; \node(num9)[num,below of = num2,yshift= -0.6cm,fill=yellow!20]{3};
\node(num10)[num,below of = num9,yshift= -0.6cm,fill=yellow!10]{1}; \node(num10)[num,below of = num9,yshift= -0.6cm,fill=yellow!20]{1};
\node(num11)[num,right of = num9,xshift= 0.6cm,fill=yellow!10]{2}; \node(num11)[num,right of = num9,xshift= 0.6cm,fill=yellow!20]{2};
\node(num12)[num,below of = num11,yshift= -0.6cm,fill=yellow!10]{2}; \node(num12)[num,below of = num11,yshift= -0.6cm,fill=yellow!20]{2};
\node(num13)[num,right of = num11,xshift= 0.6cm,fill=blue!10]{1}; \node(num13)[num,right of = num11,xshift= 0.6cm,fill=blue!20]{1};
\node(num14)[num,below of = num13,yshift= -0.6cm,fill=blue!10]{3}; \node(num14)[num,below of = num13,yshift= -0.6cm,fill=blue!20]{3};
\node(num10)[num,right of = num13,xshift= 0.6cm,fill=blue!10]{0}; \node(num10)[num,right of = num13,xshift= 0.6cm,fill=blue!20]{0};
\node(num16)[num,below of = num10,yshift= -0.6cm,fill=blue!10]{4}; \node(num16)[num,below of = num10,yshift= -0.6cm,fill=blue!20]{4};
\draw[->,thick]([xshift=0.4cm,yshift=-0.4cm]num8.east)--([xshift=1.5cm,yshift=-0.4cm]num8.east); \draw[->,thick]([xshift=0.4cm,yshift=-0.4cm]num8.east)--([xshift=1.5cm,yshift=-0.4cm]num8.east);
\node(num17)[num,right of = num8,xshift= 2.5cm,fill=red!10]{6}; \node(num17)[num,right of = num8,xshift= 2.5cm,fill=red!20]{6};
\node(num18)[num,right of = num17,xshift= 0.6cm,fill=green!10]{8}; \node(num18)[num,right of = num17,xshift= 0.6cm,fill=green!20]{8};
\node(num19)[num,below of = num17,yshift=-0.6cm,fill=yellow!10]{3}; \node(num19)[num,below of = num17,yshift=-0.6cm,fill=yellow!20]{3};
\node(num20)[num,below of = num18,yshift= -0.6cm,fill=blue!10]{4}; \node(num20)[num,below of = num18,yshift= -0.6cm,fill=blue!20]{4};
\node [right of = num20,xshift= 0.7cm]{}; \node [right of = num20,xshift= 0.7cm]{};
......
Chapter11/Figures/figure-padding-and-conv.tex
\tikzstyle{num} = [minimum width = 0.6cm,minimum height = 0.6cm,draw,fill=green!10] \tikzstyle{num} = [minimum width = 0.6cm,minimum height = 0.6cm,draw,fill=green!20]
\tikzstyle{pad} = [minimum width = 0.6cm,minimum height = 0.6cm,draw,fill=blue!10] \tikzstyle{pad} = [minimum width = 0.6cm,minimum height = 0.6cm,draw,fill=blue!20]
\begin{tikzpicture}[node distance = 0] \begin{tikzpicture}[node distance = 0]
\node[pad] at (-0.6,-0.6){0}; \node[pad] at (-0.6,-0.6){0};
...@@ -74,8 +74,8 @@ ...@@ -74,8 +74,8 @@
\node[pad] at (3,3.6){0}; \node[pad] at (3,3.6){0};
\node[pad] at (3.6,3.6){0}; \node[pad] at (3.6,3.6){0};
\node[minimum width = 1.8cm,minimum height = 1.8cm,draw=purple!40,line width=0.08cm,fill=purple!40,fill opacity=0.4] at (0,3) {}; \node[minimum width = 1.8cm,minimum height = 1.8cm,draw=purple!40,line width=0.08cm,fill=purple!50,fill opacity=0.4] at (0,3) {};
\node[minimum width = 1.8cm,minimum height = 1.8cm,draw=orange!40,line width=0.08cm,fill=orange!40,fill opacity=0.4] at (0.6,2.4) {}; \node[minimum width = 1.8cm,minimum height = 1.8cm,draw=orange!40,line width=0.08cm,fill=orange!50,fill opacity=0.4] at (0.6,2.4) {};
%\fill (4.55,1.5) circle (2pt); %\fill (4.55,1.5) circle (2pt);
\node [] at (4.55,1.5) {*}; \node [] at (4.55,1.5) {*};
......
Chapter11/Figures/figure-single-glu.tex
...@@ -6,16 +6,16 @@ ...@@ -6,16 +6,16 @@
\tikzstyle{word} = [inner sep=0pt, font=\scriptsize,minimum height=\bd] \tikzstyle{word} = [inner sep=0pt, font=\scriptsize,minimum height=\bd]
%\draw[fill=blue!8,xshift=0.3cm,yshift=0.5cm,line width=0.2pt] (0cm,0cm) rectangle (0cm+6*\bd,0cm+9*\bd); %\draw[fill=blue!8,xshift=0.3cm,yshift=0.5cm,line width=0.2pt] (0cm,0cm) rectangle (0cm+6*\bd,0cm+9*\bd);
\draw[fill=cyan!10] (0cm,0cm) -- (4*\bd,0cm) -- (4*\bd,8*\bd) -- (0cm, 8*\bd) -- (0cm,0cm); \draw[fill=cyan!20] (0cm,0cm) -- (4*\bd,0cm) -- (4*\bd,8*\bd) -- (0cm, 8*\bd) -- (0cm,0cm);
\draw[cyan,step=\bd,line width=0.8pt] (0cm,0cm) grid (0cm+4*\bd,0cm+8*\bd); \draw[cyan,step=\bd,line width=0.8pt] (0cm,0cm) grid (0cm+4*\bd,0cm+8*\bd);
\draw[fill=red!5] (0cm,1*\bd) -- (4*\bd,1*\bd) -- (4*\bd,7*\bd) -- (0cm, 7*\bd) -- (0cm,1*\bd); \draw[fill=red!10] (0cm,1*\bd) -- (4*\bd,1*\bd) -- (4*\bd,7*\bd) -- (0cm, 7*\bd) -- (0cm,1*\bd);
\draw[red!50,step=\bd,xshift=0cm,yshift=\bd, line width=0.8pt] (0cm, 0cm) grid (0cm+4*\bd,0cm+6*\bd); \draw[red!60,step=\bd,xshift=0cm,yshift=\bd, line width=0.8pt] (0cm, 0cm) grid (0cm+4*\bd,0cm+6*\bd);
\draw[fill=ugreen!5] (7*\bd,\bd) -- (11*\bd,\bd) -- (11*\bd,7*\bd) -- (7*\bd, 7*\bd) -- (7*\bd,\bd); \draw[fill=ugreen!10] (7*\bd,\bd) -- (11*\bd,\bd) -- (11*\bd,7*\bd) -- (7*\bd, 7*\bd) -- (7*\bd,\bd);
\draw[ugreen!60,step=\bd,xshift=7*\bd,yshift=\bd, line width=0.8pt] (0cm, 0cm) grid (0cm+4*\bd,0cm+6*\bd); \draw[ugreen!70,step=\bd,xshift=7*\bd,yshift=\bd, line width=0.8pt] (0cm, 0cm) grid (0cm+4*\bd,0cm+6*\bd);
\draw[fill=ugreen!5] (11.5*\bd,\bd) -- (15.5*\bd,\bd) -- (15.5*\bd,7*\bd) -- (11.5*\bd, 7*\bd) -- (11.5*\bd,\bd); \draw[fill=ugreen!10] (11.5*\bd,\bd) -- (15.5*\bd,\bd) -- (15.5*\bd,7*\bd) -- (11.5*\bd, 7*\bd) -- (11.5*\bd,\bd);
\draw[ugreen!60,step=\bd,xshift=11.5*\bd,yshift=\bd, line width=0.8pt] (0cm, 0cm) grid (0cm+4*\bd,0cm+6*\bd); \draw[ugreen!70,step=\bd,xshift=11.5*\bd,yshift=\bd, line width=0.8pt] (0cm, 0cm) grid (0cm+4*\bd,0cm+6*\bd);
\draw[fill=blue!5] (18.5*\bd,\bd) -- (22.5*\bd,\bd) -- (22.5*\bd,7*\bd) -- (18.5*\bd, 7*\bd) -- (18.5*\bd,\bd); \draw[fill=blue!10] (18.5*\bd,\bd) -- (22.5*\bd,\bd) -- (22.5*\bd,7*\bd) -- (18.5*\bd, 7*\bd) -- (18.5*\bd,\bd);
\draw[blue!50,step=\bd,xshift=18.5*\bd,yshift=\bd, line width=0.8pt] (0cm, 0cm) grid (0cm+4*\bd,0cm+6*\bd); \draw[blue!60,step=\bd,xshift=18.5*\bd,yshift=\bd, line width=0.8pt] (0cm, 0cm) grid (0cm+4*\bd,0cm+6*\bd);
\draw[-,line width=0.6pt] (4*\bd, 0.5*\bd) -- (5.5*\bd, 1.5*\bd) -- (4*\bd, 2.5*\bd); \draw[-,line width=0.6pt] (4*\bd, 0.5*\bd) -- (5.5*\bd, 1.5*\bd) -- (4*\bd, 2.5*\bd);
...@@ -44,7 +44,7 @@ ...@@ -44,7 +44,7 @@
\node[word] (w7) at ([yshift=-\bd]w6) {。}; \node[word] (w7) at ([yshift=-\bd]w6) {。};
\node[word] (w8) at ([yshift=-\bd]w7) {$<$p$>$}; \node[word] (w8) at ([yshift=-\bd]w7) {$<$p$>$};
\node[inner sep=2pt,word,fill=green!5,draw=ugreen,dotted,very thick,minimum width=6em,align=center,minimum height=4em] (sub) at ([xshift=11cm,yshift=-0.1cm]w1.east) { \node[inner sep=2pt,word,fill=green!10,draw=ugreen,dotted,very thick,minimum width=6em,align=center,minimum height=4em] (sub) at ([xshift=11cm,yshift=-0.1cm]w1.east) {
\begin{tabular}{r l} \begin{tabular}{r l}
$<$p$>$：& 填充 \\ $<$p$>$：& 填充 \\
$\sigma$：& sigmoid函数 \\ $\sigma$：& sigmoid函数 \\
......
Chapter11/Figures/figure-standard-convolution-neural-network-module.tex
\tikzstyle{input} = [rectangle, minimum width = 1cm, minimum height = 3cm, text centered] \tikzstyle{input} = [rectangle, minimum width = 1.5cm, minimum height = 3cm, text centered]
\tikzstyle{output} = [rectangle, minimum width = 1cm, minimum height = 3cm, text centered] \tikzstyle{output} = [rectangle, minimum width = 1.5cm, minimum height = 3cm, text centered]
\tikzstyle{convolution} = [rectangle, minimum width = 0.7cm, minimum height = 2cm, text centered, fill = red!10, draw = black, thick] \tikzstyle{convolution} = [rectangle, minimum width = 1cm, minimum height = 2cm, text centered, fill = red!20, draw = black, thick]
\tikzstyle{activation} = [rectangle, minimum width = 0.7cm, minimum height = 2cm, text centered, fill = blue!10, draw = black, thick] \tikzstyle{activation} = [rectangle, minimum width = 1cm, minimum height = 2cm, text centered, fill = blue!20, draw = black, thick]
\tikzstyle{pooling} = [rectangle, thick, minimum width = 0.7cm, minimum height = 2cm, text centered, draw = black, fill = ugreen!10] \tikzstyle{pooling} = [rectangle, thick, minimum width = 1cm, minimum height = 2cm, text centered, draw = black, fill = ugreen!20]
\tikzstyle{arrow} = [thick, ->, >=stealth] \tikzstyle{arrow} = [thick, ->, >=stealth]
\begin{tikzpicture}[node distance = 0cm] \begin{tikzpicture}[node distance = 0cm]
\node(input)[input, align=center]{输\\入}; \node(input)[input, align=center]{输\\入};
\node(convolution)[convolution,right of = input,xshift = 2cm, align=center]{卷\\积\\层}; \node(convolution)[convolution,right of = input,xshift = 2.5cm, align=center]{卷\\积\\层};
\node(activation)[activation,right of = convolution,xshift = 2cm, align=center]{激\\活\\函\\数}; \node(activation)[activation,right of = convolution,xshift = 2.5cm, align=center]{激\\活\\函\\数};
\node(pooling)[pooling,right of = activation,xshift = 2cm, align=center]{池\\化\\层}; \node(pooling)[pooling,right of = activation,xshift = 2.5cm, align=center]{池\\化\\层};
\node(output)[output,right of = pooling,xshift= 2cm, align=center]{输\\出}; \node(output)[output,right of = pooling,xshift= 2.5cm, align=center]{输\\出};
\draw [arrow] (input) -- ([xshift=-0.15cm]convolution.180); \draw [arrow] (input) -- ([xshift=-0.15cm]convolution.180);
\draw [arrow] ([xshift=0.15cm]convolution.0) -- ([xshift=-0.15cm]activation.180); \draw [arrow] ([xshift=0.15cm]convolution.0) -- ([xshift=-0.15cm]activation.180);
......
Chapter11/Figures/figure-standard.tex
...@@ -43,7 +43,7 @@ ...@@ -43,7 +43,7 @@
\node [anchor=north,font=\tiny,scale=1.1] at ([yshift=-0.2em]t1.south) {(a) 标准卷积}; \node [anchor=north,font=\tiny,scale=1.1] at ([yshift=-0.2em]t1.south) {(a) 标准卷积};
\end{scope} \end{scope}
\begin{scope}[xshift=4cm] \begin{scope}[xshift=5cm]
\foreach \point in {0,1,2}{ \foreach \point in {0,1,2}{
\node[unit] (l1_\point) at (0, 0+\point*1.5em){}; \node[unit] (l1_\point) at (0, 0+\point*1.5em){};
\node[unit] (l2_\point) at (0, 6em+\point*1.5em){}; \node[unit] (l2_\point) at (0, 6em+\point*1.5em){};
...@@ -77,7 +77,7 @@ ...@@ -77,7 +77,7 @@
\node [anchor=north,font=\tiny,scale=1.1] at ([yshift=-0.2em]t2.south) {(b) 深度卷积}; \node [anchor=north,font=\tiny,scale=1.1] at ([yshift=-0.2em]t2.south) {(b) 深度卷积};
\end{scope} \end{scope}
\begin{scope}[xshift=8cm] \begin{scope}[xshift=10cm]
\foreach \point in {0,1,2}{ \foreach \point in {0,1,2}{
\node[unit] (l1_\point) at (0, 0+\point*1.5em){}; \node[unit] (l1_\point) at (0, 0+\point*1.5em){};
\node[unit] (l2_\point) at (0, 6em+\point*1.5em){}; \node[unit] (l2_\point) at (0, 6em+\point*1.5em){};
......
Chapter11/Figures/figure-structural-comparison-a.tex
...@@ -4,13 +4,13 @@ ...@@ -4,13 +4,13 @@
\begin{tikzpicture}[node distance = 0cm] \begin{tikzpicture}[node distance = 0cm]
\node(num1)[num]{$\mathbi{e}_1$}; \node(num1)[num]{$\mathbi{e}_1$};
\node(num2)[num,right of = num1,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_2$}}; \node(num2)[num,right of = num1,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_2$}};
\node(num3)[num,right of = num2,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_3$}}; \node(num3)[num,right of = num2,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_3$}};
\node(num4)[num,right of = num3,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_4$}}; \node(num4)[num,right of = num3,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_4$}};
\node(num5)[num,right of = num4,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_5$}}; \node(num5)[num,right of = num4,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_5$}};
\node(num6)[num,right of = num5,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_6$}}; \node(num6)[num,right of = num5,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_6$}};
\node(num7)[num,right of = num6,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_7$}}; \node(num7)[num,right of = num6,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_7$}};
\node(num8)[num,right of = num7,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_8$}}; \node(num8)[num,right of = num7,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_8$}};
\node(num9)[num,right of = num8,xshift = 1.2cm]{$\mathbi{e}_9$}; \node(num9)[num,right of = num8,xshift = 1.2cm]{$\mathbi{e}_9$};
%\node(A)[below of = num2,yshift = -0.6cm]{A}; %\node(A)[below of = num2,yshift = -0.6cm]{A};
%\node(B)[below of = num8,yshift = -0.6cm]{B}; %\node(B)[below of = num8,yshift = -0.6cm]{B};
...@@ -23,8 +23,8 @@ ...@@ -23,8 +23,8 @@
\draw [->, thick, color = blue!80](num6.east)--(num7.west); \draw [->, thick, color = blue!80](num6.east)--(num7.west);
\draw [->, thick, color = blue!80](num7.east)--(num8.west); \draw [->, thick, color = blue!80](num7.east)--(num8.west);
\draw [->,thick,color = black!70] (num1) -- (num2); \draw [->,thick,color = black!85] (num1) -- (num2);
\draw [->,thick,color =black!70] (num8) -- (num9); \draw [->,thick,color =black!85] (num8) -- (num9);
\end{tikzpicture} \end{tikzpicture}
\ No newline at end of file
Chapter11/Figures/figure-structural-comparison-b.tex
...@@ -4,13 +4,13 @@ ...@@ -4,13 +4,13 @@
\begin{tikzpicture}[node distance = 0cm] \begin{tikzpicture}[node distance = 0cm]
\node(num1_0)[num, fill = blue!40]{\textcolor{white}{$\mathbi{0}$}}; \node(num1_0)[num, fill = blue!40]{\textcolor{white}{$\mathbi{0}$}};
\node(num1_1)[num,right of = num1_0,xshift = 1.2cm]{$\mathbi{e}_1$}; \node(num1_1)[num,right of = num1_0,xshift = 1.2cm]{$\mathbi{e}_1$};
\node(num1_2)[num,right of = num1_1,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_2$}}; \node(num1_2)[num,right of = num1_1,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_2$}};
\node(num1_3)[num,right of = num1_2,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_3$}}; \node(num1_3)[num,right of = num1_2,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_3$}};
\node(num1_4)[num,right of = num1_3,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_4$}}; \node(num1_4)[num,right of = num1_3,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_4$}};
\node(num1_5)[num,right of = num1_4,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_5$}}; \node(num1_5)[num,right of = num1_4,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_5$}};
\node(num1_6)[num,right of = num1_5,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_6$}}; \node(num1_6)[num,right of = num1_5,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_6$}};
\node(num1_7)[num,right of = num1_6,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_7$}}; \node(num1_7)[num,right of = num1_6,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_7$}};
\node(num1_8)[num,right of = num1_7,xshift = 1.2cm]{\textcolor{blue!70}{$\mathbi{e}_8$}}; \node(num1_8)[num,right of = num1_7,xshift = 1.2cm]{\textcolor{blue!85}{$\mathbi{e}_8$}};
\node(num1_9)[num,right of = num1_8,xshift = 1.2cm]{$\mathbi{e}_9$}; \node(num1_9)[num,right of = num1_8,xshift = 1.2cm]{$\mathbi{e}_9$};
\node(num1_10)[num,right of = num1_9,xshift = 1.2cm, fill = blue!40]{$\mathbi{0}$}; \node(num1_10)[num,right of = num1_9,xshift = 1.2cm, fill = blue!40]{$\mathbi{0}$};
%\node(A)[below of = num2,yshift = -0.6cm]{A}; %\node(A)[below of = num2,yshift = -0.6cm]{A};
...@@ -19,11 +19,11 @@ ...@@ -19,11 +19,11 @@
\node(num2_0)[num,above of = num1_0,yshift = 1.2cm, fill = blue!40]{\textcolor{white}{$\mathbi{0}$}}; \node(num2_0)[num,above of = num1_0,yshift = 1.2cm, fill = blue!40]{\textcolor{white}{$\mathbi{0}$}};
\node(num2_1)[num,right of = num2_0,xshift = 1.2cm]{\textbf2}; \node(num2_1)[num,right of = num2_0,xshift = 1.2cm]{\textbf2};
\node(num2_2)[num,right of = num2_1,xshift = 1.2cm]{\textbf2}; \node(num2_2)[num,right of = num2_1,xshift = 1.2cm]{\textbf2};
\node(num2_3)[num,right of = num2_2,xshift = 1.2cm]{\textbf{\textcolor{blue!70}2}}; \node(num2_3)[num,right of = num2_2,xshift = 1.2cm]{\textbf{\textcolor{blue!85}2}};
\node(num2_4)[num,right of = num2_3,xshift = 1.2cm]{\textbf{\textcolor{blue!70}2}}; \node(num2_4)[num,right of = num2_3,xshift = 1.2cm]{\textbf{\textcolor{blue!85}2}};
\node(num2_5)[num,right of = num2_4,xshift = 1.2cm]{\textbf{\textcolor{blue!70}2}}; \node(num2_5)[num,right of = num2_4,xshift = 1.2cm]{\textbf{\textcolor{blue!85}2}};
\node(num2_6)[num,right of = num2_5,xshift = 1.2cm]{\textbf{\textcolor{blue!70}2}}; \node(num2_6)[num,right of = num2_5,xshift = 1.2cm]{\textbf{\textcolor{blue!85}2}};
\node(num2_7)[num,right of = num2_6,xshift = 1.2cm]{\textbf{\textcolor{blue!70}2}}; \node(num2_7)[num,right of = num2_6,xshift = 1.2cm]{\textbf{\textcolor{blue!85}2}};
\node(num2_8)[num,right of = num2_7,xshift = 1.2cm]{\textbf2}; \node(num2_8)[num,right of = num2_7,xshift = 1.2cm]{\textbf2};
\node(num2_9)[num,right of = num2_8,xshift = 1.2cm]{\textbf2}; \node(num2_9)[num,right of = num2_8,xshift = 1.2cm]{\textbf2};
\node(num2_10)[num,right of = num2_9,xshift = 1.2cm, fill = blue!40]{$\mathbi{0}$}; \node(num2_10)[num,right of = num2_9,xshift = 1.2cm, fill = blue!40]{$\mathbi{0}$};
...@@ -32,9 +32,9 @@ ...@@ -32,9 +32,9 @@
\node(num3_1)[num,right of = num3_0,xshift = 1.2cm]{\textbf3}; \node(num3_1)[num,right of = num3_0,xshift = 1.2cm]{\textbf3};
\node(num3_2)[num,right of = num3_1,xshift = 1.2cm]{\textbf3}; \node(num3_2)[num,right of = num3_1,xshift = 1.2cm]{\textbf3};
\node(num3_3)[num,right of = num3_2,xshift = 1.2cm]{\textbf3}; \node(num3_3)[num,right of = num3_2,xshift = 1.2cm]{\textbf3};
\node(num3_4)[num,right of = num3_3,xshift = 1.2cm]{\textbf{\textcolor{blue!70}3}}; \node(num3_4)[num,right of = num3_3,xshift = 1.2cm]{\textbf{\textcolor{blue!85}3}};
\node(num3_5)[num,right of = num3_4,xshift = 1.2cm]{\textbf{\textcolor{blue!70}3}}; \node(num3_5)[num,right of = num3_4,xshift = 1.2cm]{\textbf{\textcolor{blue!85}3}};
\node(num3_6)[num,right of = num3_5,xshift = 1.2cm]{\textbf{\textcolor{blue!70}3}}; \node(num3_6)[num,right of = num3_5,xshift = 1.2cm]{\textbf{\textcolor{blue!85}3}};
\node(num3_7)[num,right of = num3_6,xshift = 1.2cm]{\textbf3}; \node(num3_7)[num,right of = num3_6,xshift = 1.2cm]{\textbf3};
\node(num3_8)[num,right of = num3_7,xshift = 1.2cm]{\textbf3}; \node(num3_8)[num,right of = num3_7,xshift = 1.2cm]{\textbf3};
\node(num3_9)[num,right of = num3_8,xshift = 1.2cm]{\textbf3}; \node(num3_9)[num,right of = num3_8,xshift = 1.2cm]{\textbf3};
...@@ -45,7 +45,7 @@ ...@@ -45,7 +45,7 @@
\node(num4_2)[num,right of = num4_1,xshift = 1.2cm]{\textbf4}; \node(num4_2)[num,right of = num4_1,xshift = 1.2cm]{\textbf4};
\node(num4_3)[num,right of = num4_2,xshift = 1.2cm]{\textbf4}; \node(num4_3)[num,right of = num4_2,xshift = 1.2cm]{\textbf4};
\node(num4_4)[num,right of = num4_3,xshift = 1.2cm]{\textbf4}; \node(num4_4)[num,right of = num4_3,xshift = 1.2cm]{\textbf4};
\node(num4_5)[num,right of = num4_4,xshift = 1.2cm]{\textbf{\textcolor{blue!60}4}}; \node(num4_5)[num,right of = num4_4,xshift = 1.2cm]{\textbf{\textcolor{blue!80}4}};
\node(num4_6)[num,right of = num4_5,xshift = 1.2cm]{\textbf4}; \node(num4_6)[num,right of = num4_5,xshift = 1.2cm]{\textbf4};
\node(num4_7)[num,right of = num4_6,xshift = 1.2cm]{\textbf4}; \node(num4_7)[num,right of = num4_6,xshift = 1.2cm]{\textbf4};
\node(num4_8)[num,right of = num4_7,xshift = 1.2cm]{\textbf4}; \node(num4_8)[num,right of = num4_7,xshift = 1.2cm]{\textbf4};
...@@ -58,19 +58,19 @@ ...@@ -58,19 +58,19 @@
\draw [->, thick](num1_1.north)--([xshift=-0.1em,yshift=-0.1em]num2_2.south); \draw [->, thick](num1_1.north)--([xshift=-0.1em,yshift=-0.1em]num2_2.south);
\draw [->, thick](num2_1.north)--([xshift=-0.1em,yshift=-0.1em]num3_2.south); \draw [->, thick](num2_1.north)--([xshift=-0.1em,yshift=-0.1em]num3_2.south);
\draw [->, thick](num3_1.north)--([xshift=-0.1em,yshift=-0.1em]num4_2.south); \draw [->, thick](num3_1.north)--([xshift=-0.1em,yshift=-0.1em]num4_2.south);
\draw [->, thick, color = blue!60](num1_2.north)--([xshift=-0.1em,yshift=-0.1em]num2_3.south); \draw [->, thick, color = blue!80](num1_2.north)--([xshift=-0.1em,yshift=-0.1em]num2_3.south);
\draw [->, thick](num2_2.north)--([xshift=-0.1em,yshift=-0.1em]num3_3.south); \draw [->, thick](num2_2.north)--([xshift=-0.1em,yshift=-0.1em]num3_3.south);
\draw [->, thick](num3_2.north)--([xshift=-0.1em,yshift=-0.1em]num4_3.south); \draw [->, thick](num3_2.north)--([xshift=-0.1em,yshift=-0.1em]num4_3.south);
\draw [->, thick, color = blue!60](num1_3.north)--([xshift=-0.1em,yshift=-0.1em]num2_4.south); \draw [->, thick, color = blue!80](num1_3.north)--([xshift=-0.1em,yshift=-0.1em]num2_4.south);
\draw [->, thick, color = blue!60](num2_3.north)--([xshift=-0.1em,yshift=-0.1em]num3_4.south); \draw [->, thick, color = blue!80](num2_3.north)--([xshift=-0.1em,yshift=-0.1em]num3_4.south);
\draw [->, thick](num3_3.north)--([xshift=-0.1em,yshift=-0.1em]num4_4.south); \draw [->, thick](num3_3.north)--([xshift=-0.1em,yshift=-0.1em]num4_4.south);
\draw [->, thick, color = blue!60](num1_4.north)--([xshift=-0.1em,yshift=-0.1em]num2_5.south); \draw [->, thick, color = blue!80](num1_4.north)--([xshift=-0.1em,yshift=-0.1em]num2_5.south);
\draw [->, thick, color = blue!60](num2_4.north)--([xshift=-0.1em,yshift=-0.1em]num3_5.south); \draw [->, thick, color = blue!80](num2_4.north)--([xshift=-0.1em,yshift=-0.1em]num3_5.south);
\draw [->, thick, color = blue!60](num3_4.north)--([xshift=-0.1em,yshift=-0.1em]num4_5.south); \draw [->, thick, color = blue!80](num3_4.north)--([xshift=-0.1em,yshift=-0.1em]num4_5.south);
\draw [->, thick, color = blue!60](num1_5.north)--([xshift=-0.1em,yshift=-0.1em]num2_6.south); \draw [->, thick, color = blue!80](num1_5.north)--([xshift=-0.1em,yshift=-0.1em]num2_6.south);
\draw [->, thick, color = blue!60](num2_5.north)--([xshift=-0.1em,yshift=-0.1em]num3_6.south); \draw [->, thick, color = blue!80](num2_5.north)--([xshift=-0.1em,yshift=-0.1em]num3_6.south);
\draw [->, thick](num3_5.north)--([xshift=-0.1em,yshift=-0.1em]num4_6.south); \draw [->, thick](num3_5.north)--([xshift=-0.1em,yshift=-0.1em]num4_6.south);
\draw [->, thick, color = blue!60](num1_6.north)--([xshift=-0.1em,yshift=-0.1em]num2_7.south); \draw [->, thick, color = blue!80](num1_6.north)--([xshift=-0.1em,yshift=-0.1em]num2_7.south);
\draw [->, thick](num2_6.north)--([xshift=-0.1em,yshift=-0.1em]num3_7.south); \draw [->, thick](num2_6.north)--([xshift=-0.1em,yshift=-0.1em]num3_7.south);
\draw [->, thick](num3_6.north)--([xshift=-0.1em,yshift=-0.1em]num4_7.south); \draw [->, thick](num3_6.north)--([xshift=-0.1em,yshift=-0.1em]num4_7.south);
\draw [->, thick](num1_7.north)--([xshift=-0.1em,yshift=-0.1em]num2_8.south); \draw [->, thick](num1_7.north)--([xshift=-0.1em,yshift=-0.1em]num2_8.south);
...@@ -86,19 +86,19 @@ ...@@ -86,19 +86,19 @@
\draw [->, thick](num1_3.north)--([xshift=0.1em,yshift=-0.1em]num2_2.south); \draw [->, thick](num1_3.north)--([xshift=0.1em,yshift=-0.1em]num2_2.south);
\draw [->, thick](num2_3.north)--([xshift=0.1em,yshift=-0.1em]num3_2.south); \draw [->, thick](num2_3.north)--([xshift=0.1em,yshift=-0.1em]num3_2.south);
\draw [->, thick](num3_3.north)--([xshift=0.1em,yshift=-0.1em]num4_2.south); \draw [->, thick](num3_3.north)--([xshift=0.1em,yshift=-0.1em]num4_2.south);
\draw [->, thick, color = blue!60](num1_4.north)--([xshift=0.1em,yshift=-0.1em]num2_3.south); \draw [->, thick, color = blue!80](num1_4.north)--([xshift=0.1em,yshift=-0.1em]num2_3.south);
\draw [->, thick](num2_4.north)--([xshift=0.1em,yshift=-0.1em]num3_3.south); \draw [->, thick](num2_4.north)--([xshift=0.1em,yshift=-0.1em]num3_3.south);
\draw [->, thick](num3_4.north)--([xshift=0.1em,yshift=-0.1em]num4_3.south); \draw [->, thick](num3_4.north)--([xshift=0.1em,yshift=-0.1em]num4_3.south);
\draw [->, thick, color = blue!60](num1_5.north)--([xshift=0.1em,yshift=-0.1em]num2_4.south); \draw [->, thick, color = blue!80](num1_5.north)--([xshift=0.1em,yshift=-0.1em]num2_4.south);
\draw [->, thick, color = blue!60](num2_5.north)--([xshift=0.1em,yshift=-0.1em]num3_4.south); \draw [->, thick, color = blue!80](num2_5.north)--([xshift=0.1em,yshift=-0.1em]num3_4.south);
\draw [->, thick](num3_5.north)--([xshift=0.1em,yshift=-0.1em]num4_4.south); \draw [->, thick](num3_5.north)--([xshift=0.1em,yshift=-0.1em]num4_4.south);
\draw [->, thick, color = blue!60](num1_6.north)--([xshift=0.1em,yshift=-0.1em]num2_5.south); \draw [->, thick, color = blue!80](num1_6.north)--([xshift=0.1em,yshift=-0.1em]num2_5.south);
\draw [->, thick, color = blue!60](num2_6.north)--([xshift=0.1em,yshift=-0.1em]num3_5.south); \draw [->, thick, color = blue!80](num2_6.north)--([xshift=0.1em,yshift=-0.1em]num3_5.south);
\draw [->, thick, color = blue!60](num3_6.north)--([xshift=0.1em,yshift=-0.1em]num4_5.south); \draw [->, thick, color = blue!80](num3_6.north)--([xshift=0.1em,yshift=-0.1em]num4_5.south);
\draw [->, thick, color = blue!60](num1_7.north)--([xshift=0.1em,yshift=-0.1em]num2_6.south); \draw [->, thick, color = blue!80](num1_7.north)--([xshift=0.1em,yshift=-0.1em]num2_6.south);
\draw [->, thick, color = blue!60](num2_7.north)--([xshift=0.1em,yshift=-0.1em]num3_6.south); \draw [->, thick, color = blue!80](num2_7.north)--([xshift=0.1em,yshift=-0.1em]num3_6.south);
\draw [->, thick](num3_7.north)--([xshift=0.1em,yshift=-0.1em]num4_6.south); \draw [->, thick](num3_7.north)--([xshift=0.1em,yshift=-0.1em]num4_6.south);
\draw [->, thick, color = blue!60](num1_8.north)--([xshift=0.1em,yshift=-0.1em]num2_7.south); \draw [->, thick, color = blue!80](num1_8.north)--([xshift=0.1em,yshift=-0.1em]num2_7.south);
\draw [->, thick](num2_8.north)--([xshift=0.1em,yshift=-0.1em]num3_7.south); \draw [->, thick](num2_8.north)--([xshift=0.1em,yshift=-0.1em]num3_7.south);
\draw [->, thick](num3_8.north)--([xshift=0.1em,yshift=-0.1em]num4_7.south); \draw [->, thick](num3_8.north)--([xshift=0.1em,yshift=-0.1em]num4_7.south);
\draw [->, thick](num1_9.north)--([xshift=0.1em,yshift=-0.1em]num2_8.south); \draw [->, thick](num1_9.north)--([xshift=0.1em,yshift=-0.1em]num2_8.south);
......
Chapter11/Figures/figure-use-cnn-in-nmt.tex
...@@ -5,41 +5,41 @@ ...@@ -5,41 +5,41 @@
\begin{scope} \begin{scope}
%\tikzstyle{every node}=[scale=0.8] %\tikzstyle{every node}=[scale=0.8]
\tikzstyle{line} = [dash pattern=on 2pt off 1pt,line width=0.5pt] \tikzstyle{line} = [dash pattern=on 2pt off 1pt,line width=0.5pt]
\tikzstyle{cir} = [thin,fill=blue!8,draw,circle,minimum size =0.5em,drop shadow={shadow xshift=0.15em, shadow yshift=-0.1em}] \tikzstyle{cir} = [thin,fill=blue!15,draw,circle,minimum size =0.5em,drop shadow={shadow xshift=0.15em, shadow yshift=-0.1em}]
\tikzstyle{word} = [inner sep=0pt, font=\scriptsize,minimum height=\bcc] \tikzstyle{word} = [inner sep=0pt, font=\scriptsize,minimum height=\bcc]
\draw[fill=red!8,line width=0.2pt] (0cm,0cm+1*\bcc) rectangle (0cm+4*\bcc,0cm+7*\bcc); \draw[fill=red!15,line width=0.2pt] (0cm,0cm+1*\bcc) rectangle (0cm+4*\bcc,0cm+7*\bcc);
\draw[fill=cyan!14,line width=0.2pt] (0cm,0cm) rectangle (0cm+4*\bcc,0cm+1*\bcc); \draw[fill=cyan!20,line width=0.2pt] (0cm,0cm) rectangle (0cm+4*\bcc,0cm+1*\bcc);
\draw[fill=cyan!14,line width=0.2pt] (0cm,0cm+7*\bcc) rectangle (0cm+4*\bcc,0cm+8*\bcc); \draw[fill=cyan!20,line width=0.2pt] (0cm,0cm+7*\bcc) rectangle (0cm+4*\bcc,0cm+8*\bcc);
\draw[step=\bcc] (0cm,0cm) grid (0cm+4*\bcc,0cm+8*\bcc); \draw[step=\bcc] (0cm,0cm) grid (0cm+4*\bcc,0cm+8*\bcc);
%\draw[line width=0.7pt] (0cm,0cm) rectangle (0cm+4*\bcc,0cm+8*\bcc); %\draw[line width=0.7pt] (0cm,0cm) rectangle (0cm+4*\bcc,0cm+8*\bcc);
\draw[red!50,line width=1.8pt] (0cm,0cm+5*\bcc) rectangle (0cm+4*\bcc,0cm+8*\bcc); \draw[red!70,line width=1.8pt] (0cm,0cm+5*\bcc) rectangle (0cm+4*\bcc,0cm+8*\bcc);
\draw[ugreen!50,line width=1.8pt] (0cm,0cm+1*\bcc) rectangle (0cm+4*\bcc,0cm+4*\bcc); \draw[ugreen!70,line width=1.8pt] (0cm,0cm+1*\bcc) rectangle (0cm+4*\bcc,0cm+4*\bcc);
\draw[fill=blue!8,xshift=5.0cm,yshift=1.0cm,line width=0.2pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc); \draw[fill=blue!15,xshift=5.0cm,yshift=1.0cm,line width=0.2pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc);
\draw[step=\bcc,xshift=5.0cm,yshift=1.0cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+6*\bcc); \draw[step=\bcc,xshift=5.0cm,yshift=1.0cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+6*\bcc);
%\draw[line width=0.7pt,xshift=5.0cm,yshift=1.0cm] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc); %\draw[line width=0.7pt,xshift=5.0cm,yshift=1.0cm] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc);
\draw[fill=blue!8,xshift=5.2cm,yshift=0.8cm,line width=0.2pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc); \draw[fill=blue!15,xshift=5.2cm,yshift=0.8cm,line width=0.2pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc);
\draw[step=\bcc,xshift=5.2cm,yshift=0.8cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+6*\bcc); \draw[step=\bcc,xshift=5.2cm,yshift=0.8cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+6*\bcc);
%\draw[line width=0.7pt,xshift=5.2cm,yshift=0.8cm] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc); %\draw[line width=0.7pt,xshift=5.2cm,yshift=0.8cm] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc);
\draw[ugreen!50,line width=2pt,xshift=5.2cm,yshift=0.8cm] (0cm,0cm+1*\bcc) rectangle (0cm+1*\bcc,0cm+2*\bcc); \draw[ugreen!70,line width=2pt,xshift=5.2cm,yshift=0.8cm] (0cm,0cm+1*\bcc) rectangle (0cm+1*\bcc,0cm+2*\bcc);
\draw[fill=blue!8,xshift=5.4cm,yshift=0.6cm,line width=0.2pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc); \draw[fill=blue!15,xshift=5.4cm,yshift=0.6cm,line width=0.2pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc);
\draw[step=\bcc,xshift=5.4cm,yshift=0.6cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+6*\bcc); \draw[step=\bcc,xshift=5.4cm,yshift=0.6cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+6*\bcc);
%\draw[line width=0.7pt,xshift=5.4cm,yshift=0.6cm] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc); %\draw[line width=0.7pt,xshift=5.4cm,yshift=0.6cm] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc);
\draw[fill=blue!8,xshift=5.6cm,yshift=0.4cm,line width=0.2pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc); \draw[fill=blue!15,xshift=5.6cm,yshift=0.4cm,line width=0.2pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc);
\draw[step=\bcc,xshift=5.6cm,yshift=0.4cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+6*\bcc); \draw[step=\bcc,xshift=5.6cm,yshift=0.4cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+6*\bcc);
%\draw[line width=0.7pt,xshift=5.6cm,yshift=0.4cm] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc); %\draw[line width=0.7pt,xshift=5.6cm,yshift=0.4cm] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc);
\draw[red!50,line width=2pt,xshift=5.6cm,yshift=0.4cm] (0cm,0cm+5*\bcc) rectangle (0cm+1*\bcc,0cm+6*\bcc); \draw[red!70,line width=2pt,xshift=5.6cm,yshift=0.4cm] (0cm,0cm+5*\bcc) rectangle (0cm+1*\bcc,0cm+6*\bcc);
\draw[red!50,line width=0.5pt] (0cm+4*\bcc,0cm+8*\bcc) -- ([xshift=5.6cm,yshift=0.4cm]0cm,0cm+6*\bcc); \draw[red!70,line width=0.5pt] (0cm+4*\bcc,0cm+8*\bcc) -- ([xshift=5.6cm,yshift=0.4cm]0cm,0cm+6*\bcc);
\draw[red!50,line width=0.5pt] (0cm+4*\bcc,0cm+5*\bcc) -- ([xshift=5.6cm,yshift=0.4cm]0cm,0cm+5*\bcc); \draw[red!70,line width=0.5pt] (0cm+4*\bcc,0cm+5*\bcc) -- ([xshift=5.6cm,yshift=0.4cm]0cm,0cm+5*\bcc);
\draw[ugreen!50,line width=0.5pt] (0cm+4*\bcc,0cm+4*\bcc) -- ([xshift=5.2cm,yshift=0.8cm]0cm,0cm+2*\bcc); \draw[ugreen!70,line width=0.5pt] (0cm+4*\bcc,0cm+4*\bcc) -- ([xshift=5.2cm,yshift=0.8cm]0cm,0cm+2*\bcc);
\draw[ugreen!50,line width=0.5pt] (0cm+4*\bcc,0cm+1*\bcc) -- ([xshift=5.2cm,yshift=0.8cm]0cm,0cm+1*\bcc); \draw[ugreen!70,line width=0.5pt] (0cm+4*\bcc,0cm+1*\bcc) -- ([xshift=5.2cm,yshift=0.8cm]0cm,0cm+1*\bcc);
\node[word] (w1) at (-0.5cm, 3.0cm) {$<$p$>$}; \node[word] (w1) at (-0.5cm, 3.0cm) {$<$p$>$};
\node[word] (w2) at ([yshift=-\bcc]w1) {今天}; \node[word] (w2) at ([yshift=-\bcc]w1) {今天};
......
Chapter11/Figures/figure-use-cnn-in-sentence-classification.tex
...@@ -5,7 +5,7 @@ ...@@ -5,7 +5,7 @@
\begin{scope} \begin{scope}
%\tikzstyle{every node}=[scale=0.8] %\tikzstyle{every node}=[scale=0.8]
\tikzstyle{line} = [dash pattern=on 2pt off 1pt,line width=0.6pt] \tikzstyle{line} = [dash pattern=on 2pt off 1pt,line width=0.6pt]
\tikzstyle{cir} = [thin,fill=blue!8,draw,circle,minimum size =0.5em,drop shadow={shadow xshift=0.15em, shadow yshift=-0.1em}] \tikzstyle{cir} = [thin,fill=blue!15,draw,circle,minimum size =0.5em,drop shadow={shadow xshift=0.15em, shadow yshift=-0.1em}]
\tikzstyle{word} = [inner sep=0pt, font=\footnotesize,minimum height=\bcc] \tikzstyle{word} = [inner sep=0pt, font=\footnotesize,minimum height=\bcc]
%\draw[fill=blue!8,xshift=0.3cm,yshift=0.5cm,line width=0.6pt] (0cm,0cm) rectangle (0cm+6*\bcc,0cm+9*\bcc); %\draw[fill=blue!8,xshift=0.3cm,yshift=0.5cm,line width=0.6pt] (0cm,0cm) rectangle (0cm+6*\bcc,0cm+9*\bcc);
...@@ -13,30 +13,30 @@ ...@@ -13,30 +13,30 @@
%\draw[red!60,line width=2pt,xshift=0.3cm,yshift=0.5cm] (0cm,0cm+2*\bcc) rectangle (0cm+6*\bcc,0cm+4*\bcc); %\draw[red!60,line width=2pt,xshift=0.3cm,yshift=0.5cm] (0cm,0cm+2*\bcc) rectangle (0cm+6*\bcc,0cm+4*\bcc);
% 输入矩阵 % 输入矩阵
\draw[thick,fill=blue!8,line width=0.6pt] (0cm,0cm) rectangle (0cm+6*\bcc,0cm+9*\bcc); \draw[thick,fill=blue!15,line width=0.6pt] (0cm,0cm) rectangle (0cm+6*\bcc,0cm+9*\bcc);
\draw[step=\bcc,gray] (0cm,0cm) grid (0cm+6*\bcc,0cm+9*\bcc); \draw[step=\bcc,gray] (0cm,0cm) grid (0cm+6*\bcc,0cm+9*\bcc);
\draw[red!60,line width=2pt] (0cm,0cm) rectangle (0cm+6*\bcc,0cm+2*\bcc); \draw[red!80,line width=2pt] (0cm,0cm) rectangle (0cm+6*\bcc,0cm+2*\bcc);
\draw[ugreen!60,line width=2pt] (0cm,0cm+3*\bcc) rectangle (0cm+6*\bcc,0cm+6*\bcc); \draw[ugreen!80,line width=2pt] (0cm,0cm+3*\bcc) rectangle (0cm+6*\bcc,0cm+6*\bcc);
\draw[red!60,line width=2pt] (0cm,0cm+7*\bcc) rectangle (0cm+6*\bcc,0cm+9*\bcc); \draw[red!80,line width=2pt] (0cm,0cm+7*\bcc) rectangle (0cm+6*\bcc,0cm+9*\bcc);
% 特征图 % 特征图
\draw[fill=blue!8,xshift=5.0cm,yshift=1.3cm,line width=0.6pt] (0cm,0cm-1*\bcc) rectangle (0cm+1*\bcc,0cm+6*\bcc); \draw[fill=blue!15,xshift=5.0cm,yshift=1.3cm,line width=0.6pt] (0cm,0cm-1*\bcc) rectangle (0cm+1*\bcc,0cm+6*\bcc);
\draw[step=\bcc,gray,xshift=5.0cm,yshift=1.3cm] (0cm,0cm-1*\bcc) grid (0cm+1*\bcc,0cm+6*\bcc); \draw[step=\bcc,gray,xshift=5.0cm,yshift=1.3cm] (0cm,0cm-1*\bcc) grid (0cm+1*\bcc,0cm+6*\bcc);
\draw[ugreen!60,line width=2pt,xshift=5.0cm,yshift=1.3cm] (0cm,0cm+2*\bcc) rectangle (0cm+1*\bcc,0cm+3*\bcc); \draw[ugreen!80,line width=2pt,xshift=5.0cm,yshift=1.3cm] (0cm,0cm+2*\bcc) rectangle (0cm+1*\bcc,0cm+3*\bcc);
%最大池化 %最大池化
\draw [gray,fill=blue!8,line width=0.6pt](8cm,2.2cm) -- (8.4cm, 2.2cm) -- (8.7cm,1.4cm) -- (8.3cm, 1.4cm) -- (8cm,2.2cm); \draw [gray,fill=blue!15,line width=0.6pt](8cm,2.2cm) -- (8.4cm, 2.2cm) -- (8.7cm,1.4cm) -- (8.3cm, 1.4cm) -- (8cm,2.2cm);
\draw [gray](8.15cm,1.8cm) -- (8.55cm,1.8cm); \draw [gray](8.15cm,1.8cm) -- (8.55cm,1.8cm);
%\draw [gray](8.3cm,1.8cm) -- (8.7cm,1.8cm); %\draw [gray](8.3cm,1.8cm) -- (8.7cm,1.8cm);
%\draw [gray](8.45cm,1.4cm) -- (8.85cm,1.4cm); %\draw [gray](8.45cm,1.4cm) -- (8.85cm,1.4cm);
%全连接层 %全连接层
\draw [gray,fill=blue!8,line width=0.6pt](11cm,2.2cm) -- (11.4cm, 2.2cm) -- (11.7cm,1.8cm) -- (11.3cm, 1.8cm) -- (11cm,2.2cm); \draw [gray,fill=blue!15,line width=0.6pt](11cm,2.2cm) -- (11.4cm, 2.2cm) -- (11.7cm,1.8cm) -- (11.3cm, 1.8cm) -- (11cm,2.2cm);
%\draw [gray](11.15cm,1.8cm) -- (11.55cm,1.8cm); %\draw [gray](11.15cm,1.8cm) -- (11.55cm,1.8cm);
%最大池化 %最大池化
\draw[ugreen!60,line] ([xshift=5.0cm,yshift=1.3cm]0cm+1*\bcc,0cm+6*\bcc) -- (8cm,2.2cm); \draw[ugreen!80,line] ([xshift=5.0cm,yshift=1.3cm]0cm+1*\bcc,0cm+6*\bcc) -- (8cm,2.2cm);
\draw[ugreen!60,line] ([xshift=5.0cm,yshift=1.3cm]0cm+1*\bcc,0cm-1*\bcc) -- (8.15cm,1.8cm); \draw[ugreen!80,line] ([xshift=5.0cm,yshift=1.3cm]0cm+1*\bcc,0cm-1*\bcc) -- (8.15cm,1.8cm);
%特征图 %特征图
%\draw[fill=blue!8,xshift=5.2cm,yshift=1.0cm,line width=0.6pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc); %\draw[fill=blue!8,xshift=5.2cm,yshift=1.0cm,line width=0.6pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+6*\bcc);
...@@ -45,25 +45,25 @@ ...@@ -45,25 +45,25 @@
%\draw[fill=blue!8,xshift=5.4cm,yshift=0.3cm,line width=0.6pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+7*\bcc); %\draw[fill=blue!8,xshift=5.4cm,yshift=0.3cm,line width=0.6pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+7*\bcc);
%\draw[step=\bcc,gray,xshift=5.4cm,yshift=0.3cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+7*\bcc); %\draw[step=\bcc,gray,xshift=5.4cm,yshift=0.3cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+7*\bcc);
\draw[fill=blue!8,xshift=5.6cm,yshift=0cm,line width=0.6pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+8*\bcc); \draw[fill=blue!15,xshift=5.6cm,yshift=0cm,line width=0.6pt] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+8*\bcc);
\draw[step=\bcc,gray,xshift=5.6cm,yshift=0cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+8*\bcc); \draw[step=\bcc,gray,xshift=5.6cm,yshift=0cm] (0cm,0cm) grid (0cm+1*\bcc,0cm+8*\bcc);
\draw[red!60,line width=2pt,xshift=5.6cm,yshift=0cm] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+1*\bcc); \draw[red!80,line width=2pt,xshift=5.6cm,yshift=0cm] (0cm,0cm) rectangle (0cm+1*\bcc,0cm+1*\bcc);
\draw[red!60,line width=2pt,xshift=5.6cm,yshift=0cm] (0cm,0cm+7*\bcc) rectangle (0cm+1*\bcc,0cm+8*\bcc); \draw[red!80,line width=2pt,xshift=5.6cm,yshift=0cm] (0cm,0cm+7*\bcc) rectangle (0cm+1*\bcc,0cm+8*\bcc);
% 全连接线 % 全连接线
\draw[line] (8.4cm, 2.2cm) -- (11.2cm,2.2cm); \draw[line] (8.4cm, 2.2cm) -- (11.2cm,2.2cm);
\draw[line] (8.7cm,1.4cm) -- (11.3cm, 1.8cm); \draw[line] (8.7cm,1.4cm) -- (11.3cm, 1.8cm);
%全连接上面的红虚线 %全连接上面的红虚线
\draw[red!60,line] ([xshift=5.6cm,yshift=0cm]0cm+1*\bcc,0cm+7*\bcc) -- (8.15cm,1.8cm); \draw[red!80,line] ([xshift=5.6cm,yshift=0cm]0cm+1*\bcc,0cm+7*\bcc) -- (8.15cm,1.8cm);
\draw[red!60,line] ([xshift=5.6cm,yshift=0cm]0cm+1*\bcc,0cm) -- (8.3cm, 1.4cm); \draw[red!80,line] ([xshift=5.6cm,yshift=0cm]0cm+1*\bcc,0cm) -- (8.3cm, 1.4cm);
% 特征图红色虚线 % 特征图红色虚线
\draw[red!60,line] (0cm+6*\bcc,0cm+9*\bcc) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+8*\bcc); \draw[red!80,line] (0cm+6*\bcc,0cm+9*\bcc) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+8*\bcc);
\draw[red!60,line] (0cm+6*\bcc,0cm+7*\bcc) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+7*\bcc); \draw[red!80,line] (0cm+6*\bcc,0cm+7*\bcc) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+7*\bcc);
\draw[red!60,line] (0cm+6*\bcc,0cm+2*\bcc) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+1*\bcc); \draw[red!80,line] (0cm+6*\bcc,0cm+2*\bcc) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+1*\bcc);
\draw[red!60,line] (0cm+6*\bcc,0cm) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm); \draw[red!80,line] (0cm+6*\bcc,0cm) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm);
\draw[ugreen!60,line] (0cm+6*\bcc,0cm+6*\bcc) -- ([xshift=5.0cm,yshift=1.3cm]0cm,0cm+3*\bcc); \draw[ugreen!80,line] (0cm+6*\bcc,0cm+6*\bcc) -- ([xshift=5.0cm,yshift=1.3cm]0cm,0cm+3*\bcc);
\draw[ugreen!60,line] (0cm+6*\bcc,0cm+3*\bcc) -- ([xshift=5.0cm,yshift=1.3cm]0cm,0cm+2*\bcc); \draw[ugreen!80,line] (0cm+6*\bcc,0cm+3*\bcc) -- ([xshift=5.0cm,yshift=1.3cm]0cm,0cm+2*\bcc);
%\draw[red!60,line] ([xshift=0.3cm,yshift=0.5cm]0cm+6*\bcc,0cm+4*\bcc) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+3*\bcc); %\draw[red!60,line] ([xshift=0.3cm,yshift=0.5cm]0cm+6*\bcc,0cm+4*\bcc) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+3*\bcc);
%\draw[red!60,line] ([xshift=0.3cm,yshift=0.5cm]0cm+6*\bcc,0cm+2*\bcc) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+2*\bcc); %\draw[red!60,line] ([xshift=0.3cm,yshift=0.5cm]0cm+6*\bcc,0cm+2*\bcc) -- ([xshift=5.6cm,yshift=0cm]0cm,0cm+2*\bcc);
......
Chapter12/Figures/figure-a-combination-of-position-encoding-and-word-encoding.tex
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{rnode} = [draw,minimum width=3.5em,minimum height=1.2em] \tikzstyle{rnode} = [draw,minimum width=4em,minimum height=1.2em]
\node [rnode,anchor=south west,fill=red!20!white] (e1) at (0,0) {\scriptsize{$e(\textrm{沈阳})$}}; \node [rnode,anchor=south west,fill=red!30!white] (e1) at (0,0) {\scriptsize{$e(\textrm{沈阳})$}};
\node [rnode,anchor=south west,fill=red!20!white] (e2) at ([xshift=1em]e1.south east) {\scriptsize{$e(\textrm{到})$}}; \node [rnode,anchor=south west,fill=red!30!white] (e2) at ([xshift=1.5em]e1.south east) {\scriptsize{$e(\textrm{到})$}};
\node [rnode,anchor=south west,fill=red!20!white] (e3) at ([xshift=1em]e2.south east) {\scriptsize{$e(\textrm{广州})$}}; \node [rnode,anchor=south west,fill=red!30!white] (e3) at ([xshift=1.5em]e2.south east) {\scriptsize{$e(\textrm{广州})$}};
\node [rnode,anchor=south west,fill=red!20!white] (e4) at ([xshift=1em]e3.south east) {\scriptsize{$e(\textrm{的})$}}; \node [rnode,anchor=south west,fill=red!30!white] (e4) at ([xshift=1.5em]e3.south east) {\scriptsize{$e(\textrm{的})$}};
\node [rnode,anchor=south west,fill=red!20!white] (e5) at ([xshift=1em]e4.south east) {\scriptsize{$e(\textrm{机票})$}}; \node [rnode,anchor=south west,fill=red!30!white] (e5) at ([xshift=1.5em]e4.south east) {\scriptsize{$e(\textrm{机票})$}};
\node [rnode,anchor=south west,fill=green!20!white] (h1) at ([yshift=1.5em]e1.north west) {\scriptsize{$h(\textrm{沈阳})$}}; \node [rnode,anchor=south west,fill=green!30!white] (h1) at ([yshift=1.5em]e1.north west) {\scriptsize{$h(\textrm{沈阳})$}};
\node [rnode,anchor=south west,fill=green!20!white] (h2) at ([yshift=1.5em]e2.north west) {\scriptsize{$h(\textrm{到})$}}; \node [rnode,anchor=south west,fill=green!30!white] (h2) at ([yshift=1.5em]e2.north west) {\scriptsize{$h(\textrm{到})$}};
\node [rnode,anchor=south west,fill=green!20!white] (h3) at ([yshift=1.5em]e3.north west) {\scriptsize{$h(\textrm{广州})$}}; \node [rnode,anchor=south west,fill=green!30!white] (h3) at ([yshift=1.5em]e3.north west) {\scriptsize{$h(\textrm{广州})$}};
\node [rnode,anchor=south west,fill=green!20!white] (h4) at ([yshift=1.5em]e4.north west) {\scriptsize{$h(\textrm{的})$}}; \node [rnode,anchor=south west,fill=green!30!white] (h4) at ([yshift=1.5em]e4.north west) {\scriptsize{$h(\textrm{的})$}};
\node [rnode,anchor=south west,fill=green!20!white] (h5) at ([yshift=1.5em]e5.north west) {\scriptsize{$h(\textrm{机票})$}}; \node [rnode,anchor=south west,fill=green!30!white] (h5) at ([yshift=1.5em]e5.north west) {\scriptsize{$h(\textrm{机票})$}};
\foreach \x in {1,2,3,4,5}{ \foreach \x in {1,2,3,4,5}{
\node [anchor=north] (plus\x) at ([yshift=-0em]e\x.south) {\scriptsize{$\mathbf{\oplus}$}}; \node [anchor=north] (plus\x) at ([yshift=-0em]e\x.south) {\scriptsize{$\mathbf{\oplus}$}};
} }
\node [rnode,anchor=north,fill=yellow!20!white] (pos1) at ([yshift=-1.1em]e1.south) {\scriptsize{$\textrm{PE}(1)$}}; \node [rnode,anchor=north,fill=yellow!30!white] (pos1) at ([yshift=-1.1em]e1.south) {\scriptsize{$\textrm{PE}(1)$}};
\node [rnode,anchor=north,fill=yellow!20!white] (pos2) at ([yshift=-1.1em]e2.south) {\scriptsize{$\textrm{PE}(2)$}}; \node [rnode,anchor=north,fill=yellow!30!white] (pos2) at ([yshift=-1.1em]e2.south) {\scriptsize{$\textrm{PE}(2)$}};
\node [rnode,anchor=north,fill=yellow!20!white] (pos3) at ([yshift=-1.1em]e3.south) {\scriptsize{$\textrm{PE}(3)$}}; \node [rnode,anchor=north,fill=yellow!30!white] (pos3) at ([yshift=-1.1em]e3.south) {\scriptsize{$\textrm{PE}(3)$}};
\node [rnode,anchor=north,fill=yellow!20!white] (pos4) at ([yshift=-1.1em]e4.south) {\scriptsize{$\textrm{PE}(4)$}}; \node [rnode,anchor=north,fill=yellow!30!white] (pos4) at ([yshift=-1.1em]e4.south) {\scriptsize{$\textrm{PE}(4)$}};
\node [rnode,anchor=north,fill=yellow!20!white] (pos5) at ([yshift=-1.1em]e5.south) {\scriptsize{$\textrm{PE}(5)$}}; \node [rnode,anchor=north,fill=yellow!30!white] (pos5) at ([yshift=-1.1em]e5.south) {\scriptsize{$\textrm{PE}(5)$}};
\foreach \x in {1,2,3,4,5}{ \foreach \x in {1,2,3,4,5}{
......
Chapter12/Figures/figure-calculation-of-context-vector-c.tex
...@@ -2,13 +2,13 @@ ...@@ -2,13 +2,13 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{rnode} = [draw,minimum width=3.5em,minimum height=1.2em] \tikzstyle{rnode} = [draw,minimum width=4em,minimum height=1.2em]
\node [rnode,anchor=south west,fill=green!20!white] (key1) at (0,0) {\scriptsize{$h(\textrm{沈阳})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key1) at (0,0) {\scriptsize{$h(\textrm{沈阳})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key2) at ([xshift=1em]key1.south east) {\scriptsize{$h(\textrm{到})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key2) at ([xshift=1.5em]key1.south east) {\scriptsize{$h(\textrm{到})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key3) at ([xshift=1em]key2.south east) {\scriptsize{$h(\textrm{广州})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key3) at ([xshift=1.5em]key2.south east) {\scriptsize{$h(\textrm{广州})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key4) at ([xshift=2em]key3.south east) {\scriptsize{$h(\textrm{机票})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key4) at ([xshift=3em]key3.south east) {\scriptsize{$h(\textrm{机票})$}};
\node [rnode,anchor=south west] (key5) at ([xshift=1em]key4.south east) {\scriptsize{$h(\textrm{机票})$}}; \node [rnode,anchor=south west] (key5) at ([xshift=1.5em]key4.south east) {\scriptsize{$h(\textrm{机票})$}};
\node [anchor=west] (sep1) at ([xshift=0.3em]key3.east) {\scriptsize{$\textbf{...}$}}; \node [anchor=west] (sep1) at ([xshift=0.3em]key3.east) {\scriptsize{$\textbf{...}$}};
...@@ -17,17 +17,17 @@ ...@@ -17,17 +17,17 @@
\draw [->] ([yshift=1pt,xshift=3pt]key5.north) .. controls +(90:1.8em) and +(90:1.8em) .. ([yshift=1pt]key2.north); \draw [->] ([yshift=1pt,xshift=3pt]key5.north) .. controls +(90:1.8em) and +(90:1.8em) .. ([yshift=1pt]key2.north);
\draw [->] ([yshift=1pt,xshift=6pt]key5.north) .. controls +(90:2.2em) and +(90:2.2em) .. ([yshift=1pt]key1.north); \draw [->] ([yshift=1pt,xshift=6pt]key5.north) .. controls +(90:2.2em) and +(90:2.2em) .. ([yshift=1pt]key1.north);
\node [anchor=south west] (alpha1) at ([xshift=-1em]key1.north west) {\scriptsize{$\alpha_1=.2$}}; \node [anchor=south west] (alpha1) at ([xshift=-1.2em]key1.north west) {\scriptsize{$\alpha_1=0.2$}};
\node [anchor=south west] (alpha2) at ([xshift=-1em]key2.north west) {\scriptsize{$\alpha_2=.3$}}; \node [anchor=south west] (alpha2) at ([xshift=-1.2em]key2.north west) {\scriptsize{$\alpha_2=0.3$}};
\node [anchor=south west] (alpha3) at ([xshift=-1em]key3.north west) {\scriptsize{$\alpha_3=.1$}}; \node [anchor=south west] (alpha3) at ([xshift=-1.2em]key3.north west) {\scriptsize{$\alpha_3=0.1$}};
\node [anchor=south west] (alpha4) at ([xshift=-1em]key4.north west) {\scriptsize{$\alpha_i=.3$}}; \node [anchor=south west] (alpha4) at ([xshift=-1.2em]key4.north west) {\scriptsize{$\alpha_i=0.3$}};
\vspace{0.5em} \vspace{0.5em}
\node [rnode,anchor=south west,fill=green!20!white] (key6) at ([yshift=2em]key1.north west) {\scriptsize{$h(\textrm{广州})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key6) at ([yshift=2em]key1.north west) {\scriptsize{$h(\textrm{广州})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key7) at ([yshift=2em]key2.north west) {\scriptsize{$h(\textrm{到})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key7) at ([yshift=2em]key2.north west) {\scriptsize{$h(\textrm{到})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key8) at ([yshift=2em]key3.north west) {\scriptsize{$h(\textrm{沈阳})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key8) at ([yshift=2em]key3.north west) {\scriptsize{$h(\textrm{沈阳})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key9) at ([yshift=2em]key4.north west) {\scriptsize{$h(\textrm{机票})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key9) at ([yshift=2em]key4.north west) {\scriptsize{$h(\textrm{机票})$}};
\node [rnode,anchor=south west] (key10) at ([yshift=2em]key5.north west) {\scriptsize{$h(\textrm{机票})$}}; \node [rnode,anchor=south west] (key10) at ([yshift=2em]key5.north west) {\scriptsize{$h(\textrm{机票})$}};
\node [anchor=west] (sep1) at ([xshift=0.3em]key8.east) {\scriptsize{$\textbf{...}$}}; \node [anchor=west] (sep1) at ([xshift=0.3em]key8.east) {\scriptsize{$\textbf{...}$}};
...@@ -37,10 +37,10 @@ ...@@ -37,10 +37,10 @@
\draw [->] ([yshift=1pt,xshift=3pt]key10.north) .. controls +(90:1.8em) and +(90:1.8em) .. ([yshift=1pt]key7.north); \draw [->] ([yshift=1pt,xshift=3pt]key10.north) .. controls +(90:1.8em) and +(90:1.8em) .. ([yshift=1pt]key7.north);
\draw [->] ([yshift=1pt,xshift=6pt]key10.north) .. controls +(90:2.2em) and +(90:2.2em) .. ([yshift=1pt]key6.north); \draw [->] ([yshift=1pt,xshift=6pt]key10.north) .. controls +(90:2.2em) and +(90:2.2em) .. ([yshift=1pt]key6.north);
\node [anchor=south west] (alpha5) at ([xshift=-1em]key6.north west) {\scriptsize{$\alpha_1=.1$}}; \node [anchor=south west] (alpha5) at ([xshift=-1.2em]key6.north west) {\scriptsize{$\alpha_1=0.1$}};
\node [anchor=south west] (alpha6) at ([xshift=-1em]key7.north west) {\scriptsize{$\alpha_2=.3$}}; \node [anchor=south west] (alpha6) at ([xshift=-1.2em]key7.north west) {\scriptsize{$\alpha_2=0.3$}};
\node [anchor=south west] (alpha7) at ([xshift=-1em]key8.north west) {\scriptsize{$\alpha_3=.2$}}; \node [anchor=south west] (alpha7) at ([xshift=-1.2em]key8.north west) {\scriptsize{$\alpha_3=0.2$}};
\node [anchor=south west] (alpha8) at ([xshift=-1em]key9.north west) {\scriptsize{$\alpha_i=.3$}}; \node [anchor=south west] (alpha8) at ([xshift=-1.2em]key9.north west) {\scriptsize{$\alpha_i=0.3$}};
\end{scope} \end{scope}
\end{tikzpicture} \end{tikzpicture}
......
Chapter12/Figures/figure-comparison-of-the-number-of-padding-in-batch.tex
...@@ -4,7 +4,7 @@ ...@@ -4,7 +4,7 @@
\begin{scope}[scale=1.5] \begin{scope}[scale=1.5]
{\Large {\Large
\tikzstyle{snode} = [draw,inner sep=1pt,minimum width=3em,minimum height=0.5em,rounded corners=1pt,fill=green!30!white] \tikzstyle{snode} = [draw,inner sep=1pt,minimum width=3em,minimum height=0.5em,rounded corners=1pt,fill=green!40!white]
\tikzstyle{pnode} = [draw,inner sep=1pt,minimum width=1em,minimum height=0.5em,rounded corners=1pt] \tikzstyle{pnode} = [draw,inner sep=1pt,minimum width=1em,minimum height=0.5em,rounded corners=1pt]
\node [anchor=west,snode] (s1) at (0,0) {}; \node [anchor=west,snode] (s1) at (0,0) {};
\node [anchor=north west,snode,minimum width=6.5em] (s2) at ([yshift=-0.3em]s1.south west) {}; \node [anchor=north west,snode,minimum width=6.5em] (s2) at ([yshift=-0.3em]s1.south west) {};
...@@ -15,7 +15,7 @@ ...@@ -15,7 +15,7 @@
\node [anchor=west,pnode,minimum width=3em] (p1) at ([xshift=0.3em]s1.east) {}; \node [anchor=west,pnode,minimum width=3em] (p1) at ([xshift=0.3em]s1.east) {};
\node [anchor=west,pnode,minimum width=4em] (p3) at ([xshift=0.3em]s3.east) {}; \node [anchor=west,pnode,minimum width=4em] (p3) at ([xshift=0.3em]s3.east) {};
\node [anchor=west,snode,minimum width=5em] (s4) at ([xshift=4em]p1.east) {}; \node [anchor=west,snode,minimum width=5em] (s4) at ([xshift=5em]p1.east) {};
\node [anchor=north west,snode,minimum width=5em] (s5) at ([yshift=-0.3em]s4.south west) {}; \node [anchor=north west,snode,minimum width=5em] (s5) at ([yshift=-0.3em]s4.south west) {};
\node [anchor=north west,snode,minimum width=6.5em] (s6) at ([yshift=-0.3em]s5.south west) {}; \node [anchor=north west,snode,minimum width=6.5em] (s6) at ([yshift=-0.3em]s5.south west) {};
......
Chapter12/Figures/figure-decode-of-transformer.tex
...@@ -3,14 +3,14 @@ ...@@ -3,14 +3,14 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{rnnnode} = [minimum height=1.1em,minimum width=2.1em,inner sep=2pt,rounded corners=1pt,draw,fill=red!20]; \tikzstyle{rnnnode} = [minimum height=1.1em,minimum width=2.7em,inner sep=2pt,rounded corners=1pt,draw,fill=red!20];
\node [rnnnode,anchor=west] (h1) at (0,0) {\tiny{$\mathbi{h}_1$}}; \node [rnnnode,anchor=west] (h1) at (0,0) {\tiny{$\mathbi{h}_1$}};
\node [rnnnode,anchor=west] (h2) at ([xshift=1em]h1.east) {\tiny{$\mathbi{h}_2$}}; \node [rnnnode,anchor=west] (h2) at ([xshift=1.5em]h1.east) {\tiny{$\mathbi{h}_2$}};
\node [rnnnode,anchor=west] (h3) at ([xshift=1em]h2.east) {\tiny{$\mathbi{h}_3$}}; \node [rnnnode,anchor=west] (h3) at ([xshift=1.5em]h2.east) {\tiny{$\mathbi{h}_3$}};
\node [rnnnode,anchor=north,fill=green!20] (e1) at ([yshift=-1em]h1.south) {\tiny{$e_x()$}}; \node [rnnnode,anchor=north,fill=green!20] (e1) at ([yshift=-1em]h1.south) {\tiny{$e_x()$}};
\node [rnnnode,anchor=west,fill=green!20] (e2) at ([xshift=1em]e1.east) {\tiny{$e_x()$}}; \node [rnnnode,anchor=west,fill=green!20] (e2) at ([xshift=1.5em]e1.east) {\tiny{$e_x()$}};
\node [rnnnode,anchor=west,fill=green!20] (e3) at ([xshift=1em]e2.east) {\tiny{$e_x()$}}; \node [rnnnode,anchor=west,fill=green!20] (e3) at ([xshift=1.5em]e2.east) {\tiny{$e_x()$}};
\node [anchor=north,inner sep=2pt] (w1) at ([yshift=-0.6em]e1.south) {\tiny{你}}; \node [anchor=north,inner sep=2pt] (w1) at ([yshift=-0.6em]e1.south) {\tiny{你}};
\node [anchor=north,inner sep=2pt] (w2) at ([yshift=-0.6em]e2.south) {\tiny{好}}; \node [anchor=north,inner sep=2pt] (w2) at ([yshift=-0.6em]e2.south) {\tiny{好}};
\node [anchor=north,inner sep=2pt] (w3) at ([yshift=-0.6em]e3.south) {\tiny{$\langle$eos$\rangle$}}; \node [anchor=north,inner sep=2pt] (w3) at ([yshift=-0.6em]e3.south) {\tiny{$\langle$eos$\rangle$}};
...@@ -33,14 +33,14 @@ ...@@ -33,14 +33,14 @@
\node [anchor=south] (encoder) at ([xshift=-0.2em]h1.north west) {\scriptsize{\textbf{编码器}}}; \node [anchor=south] (encoder) at ([xshift=-0.2em]h1.north west) {\scriptsize{\textbf{编码器}}};
{ {
\node [rnnnode,anchor=west,fill=green!20] (t1) at ([xshift=3em]e3.east) {\tiny{$e_y()$}}; \node [rnnnode,anchor=west,fill=green!20] (t1) at ([xshift=3.5em]e3.east) {\tiny{$e_y()$}};
} }
{ {
\node [rnnnode,anchor=west,fill=green!20] (t2) at ([xshift=1.5em]t1.east) {\tiny{$e_y()$}}; \node [rnnnode,anchor=west,fill=green!20] (t2) at ([xshift=1.8em]t1.east) {\tiny{$e_y()$}};
} }
{ {
\node [rnnnode,anchor=west,fill=green!20] (t3) at ([xshift=1.5em]t2.east) {\tiny{$e_y()$}}; \node [rnnnode,anchor=west,fill=green!20] (t3) at ([xshift=1.8em]t2.east) {\tiny{$e_y()$}};
\node [rnnnode,anchor=west,fill=green!20] (t4) at ([xshift=1.5em]t3.east) {\tiny{$e_y()$}}; \node [rnnnode,anchor=west,fill=green!20] (t4) at ([xshift=1.8em]t3.east) {\tiny{$e_y()$}};
%\node [anchor=west,inner sep=2pt] (t5) at ([xshift=0.3em]t4.east) {\tiny{...}}; %\node [anchor=west,inner sep=2pt] (t5) at ([xshift=0.3em]t4.east) {\tiny{...}};
} }
{ {
......
Chapter12/Figures/figure-dependencies-between-words-in-a-recurrent-neural-network.tex
...@@ -3,11 +3,11 @@ ...@@ -3,11 +3,11 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\node [anchor=west] (w0) at (0,0) {$w_1$}; \node [anchor=west] (w0) at (0,0) {$w_1$};
\node [anchor=west] (w1) at ([xshift=0.5em]w0.east) {$w_2$}; \node [anchor=west] (w1) at ([xshift=1em]w0.east) {$w_2$};
\node [anchor=west] (w2) at ([xshift=0.5em]w1.east) {$w_3$}; \node [anchor=west] (w2) at ([xshift=1em]w1.east) {$w_3$};
\node [anchor=west] (w3) at ([xshift=0.5em]w2.east) {$...$}; \node [anchor=west] (w3) at ([xshift=1em]w2.east) {$...$};
\node [anchor=west] (w4) at ([xshift=0.5em]w3.east) {$w_{m-1}$}; \node [anchor=west] (w4) at ([xshift=1em]w3.east) {$w_{m-1}$};
\node [anchor=west,fill=green!20!white] (w5) at ([xshift=0.5em]w4.east) {$w_{m}$}; \node [anchor=west,fill=green!30!white] (w5) at ([xshift=0.5em]w4.east) {$w_{m}$};
\draw [->,thick,red] (w1.north).. controls +(130:0.5) and +(50:0.5) .. (w0.north); \draw [->,thick,red] (w1.north).. controls +(130:0.5) and +(50:0.5) .. (w0.north);
\draw [->,thick,red] (w2.north).. controls +(130:0.5) and +(50:0.5) .. (w1.north); \draw [->,thick,red] (w2.north).. controls +(130:0.5) and +(50:0.5) .. (w1.north);
\draw [->,thick,red] ([yshift=0.2em]w3.north).. controls +(130:0.5) and +(50:0.5) .. (w2.north); \draw [->,thick,red] ([yshift=0.2em]w3.north).. controls +(130:0.5) and +(50:0.5) .. (w2.north);
......
Chapter12/Figures/figure-dependencies-between-words-of-attention.tex
...@@ -3,11 +3,11 @@ ...@@ -3,11 +3,11 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\node [anchor=west] (w0) at (0,-2) {$w_1$}; \node [anchor=west] (w0) at (0,-2) {$w_1$};
\node [anchor=west] (w1) at ([xshift=0.5em]w0.east) {$w_2$}; \node [anchor=west] (w1) at ([xshift=1em]w0.east) {$w_2$};
\node [anchor=west] (w2) at ([xshift=0.5em]w1.east) {$w_3$}; \node [anchor=west] (w2) at ([xshift=1em]w1.east) {$w_3$};
\node [anchor=west] (w3) at ([xshift=0.5em]w2.east) {$...$}; \node [anchor=west] (w3) at ([xshift=1em]w2.east) {$...$};
\node [anchor=west] (w4) at ([xshift=0.5em]w3.east) {$w_{m-1}$}; \node [anchor=west] (w4) at ([xshift=1em]w3.east) {$w_{m-1}$};
\node [anchor=west,fill=green!20!white] (w5) at ([xshift=0.5em]w4.east) {$w_{m}$}; \node [anchor=west,fill=green!30!white] (w5) at ([xshift=0.5em]w4.east) {$w_{m}$};
\draw [->,thick,red] (w5.north).. controls +(100:0.85) and +(50:0.85) .. (w0.north); \draw [->,thick,red] (w5.north).. controls +(100:0.85) and +(50:0.85) .. (w0.north);
\draw [->,thick,red] (w5.north).. controls +(110:0.75) and +(50:0.75) .. (w1.north); \draw [->,thick,red] (w5.north).. controls +(110:0.75) and +(50:0.75) .. (w1.north);
\draw [->,thick,red] (w5.north).. controls +(110:0.75) and +(50:0.75) .. (w2.north); \draw [->,thick,red] (w5.north).. controls +(110:0.75) and +(50:0.75) .. (w2.north);
......
Chapter12/Figures/figure-different-regularization-methods.tex
...@@ -3,12 +3,12 @@ ...@@ -3,12 +3,12 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{lnode} = [minimum height=1.5em,minimum width=3em,inner sep=3pt,rounded corners=1.5pt,draw,fill=orange!20]; \tikzstyle{lnode} = [minimum height=1.5em,minimum width=3em,inner sep=3pt,rounded corners=1.5pt,draw,fill=orange!30];
\tikzstyle{standard} = [rounded corners=3pt] \tikzstyle{standard} = [rounded corners=3pt]
\node [lnode,anchor=west] (l1) at (0,0) {\scriptsize{子层}}; \node [lnode,anchor=west] (l1) at (0,0) {\scriptsize{子层}};
\node [lnode,anchor=west] (l2) at ([xshift=3em]l1.east) {\scriptsize{层标准化}}; \node [lnode,anchor=west] (l2) at ([xshift=3em]l1.east) {\scriptsize{层标准化}};
\node [lnode,anchor=west] (l3) at ([xshift=4em]l2.east) {\scriptsize{层标准化}}; \node [lnode,anchor=west] (l3) at ([xshift=6em]l2.east) {\scriptsize{层标准化}};
\node [lnode,anchor=west] (l4) at ([xshift=1.5em]l3.east) {\scriptsize{子层}}; \node [lnode,anchor=west] (l4) at ([xshift=1.5em]l3.east) {\scriptsize{子层}};
\node [anchor=west] (plus1) at ([xshift=0.9em]l1.east) {\scriptsize{$\mathbf{\oplus}$}}; \node [anchor=west] (plus1) at ([xshift=0.9em]l1.east) {\scriptsize{$\mathbf{\oplus}$}};
......
Chapter12/Figures/figure-example-of-self-attention-mechanism-calculation.tex
...@@ -3,15 +3,15 @@ ...@@ -3,15 +3,15 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{rnode} = [draw,minimum width=2.8em,minimum height=1.2em] \tikzstyle{rnode} = [draw,minimum width=3.5em,minimum height=1.2em]
\node [rnode,anchor=south west,fill=green!20!white] (key11) at (0,0) {\scriptsize{$h(\textrm{他})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key11) at (0,0) {\scriptsize{$h(\textrm{他})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key12) at ([xshift=0.8em]key11.south east) {\scriptsize{$h(\textrm{什么})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key12) at ([xshift=1.5em]key11.south east) {\scriptsize{$h(\textrm{什么})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key13) at ([xshift=0.8em]key12.south east) {\scriptsize{$h(\textrm{也})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key13) at ([xshift=1.5em]key12.south east) {\scriptsize{$h(\textrm{也})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key14) at ([xshift=0.8em]key13.south east) {\scriptsize{$h(\textrm{没})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key14) at ([xshift=1.5em]key13.south east) {\scriptsize{$h(\textrm{没})$}};
\node [rnode,anchor=south west,fill=green!20!white] (key15) at ([xshift=0.8em]key14.south east) {\scriptsize{$h(\textrm{学})$}}; \node [rnode,anchor=south west,fill=green!30!white] (key15) at ([xshift=1.5em]key14.south east) {\scriptsize{$h(\textrm{学})$}};
\node [rnode,anchor=east] (query1) at ([xshift=-1em]key11.west) {\scriptsize{$h(\textrm{他})$}}; \node [rnode,anchor=east] (query1) at ([xshift=-1em]key11.west) {\scriptsize{$h(\textrm{他})$}};
......
Chapter12/Figures/figure-lrate-of-transformer.tex
...@@ -4,7 +4,7 @@ ...@@ -4,7 +4,7 @@
\begin{tikzpicture} \begin{tikzpicture}
\footnotesize{ \footnotesize{
\begin{axis}[ \begin{axis}[
width=.60\textwidth, width=.8\textwidth,
height=.40\textwidth, height=.40\textwidth,
legend style={at={(0.60,0.08)}, anchor=south west}, legend style={at={(0.60,0.08)}, anchor=south west},
xlabel={\footnotesize{更新步数 (10k)}}, xlabel={\footnotesize{更新步数 (10k)}},
......
Chapter12/Figures/figure-multi-head-attention-model.tex
...@@ -2,29 +2,31 @@ ...@@ -2,29 +2,31 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
%
\node [anchor=west,draw=black!30,inner sep=4pt,fill=ugreen!20!white,text=ugreen!20!white] (Linear0) at (0,0) {\footnotesize{Linear}}; \node [anchor=west,draw=black!30,inner sep=4pt,fill=ugreen!30!white,text=ugreen!30!white,minimum width=4em] (Linear0) at (0,0) {\footnotesize{Linear}};
\node [anchor=south west,draw=black!50,fill=ugreen!20!white,draw,inner sep=4pt,text=ugreen!20!white] (Linear01) at ([shift={(-0.2em,-0.2em)}]Linear0.south west) {\footnotesize{Linear}}; \node [anchor=south west,draw=black!50,fill=ugreen!30!white,draw,inner sep=4pt,text=ugreen!30!white,minimum width=4em] (Linear01) at ([shift={(-0.2em,-0.2em)}]Linear0.south west) {\footnotesize{Linear}};
\node [anchor=south west,fill=ugreen!20!white,draw,inner sep=4pt] (Linear02) at ([shift={(-0.2em,-0.2em)}]Linear01.south west) {\footnotesize{Linear}}; \node [anchor=south west,fill=ugreen!30!white,draw,inner sep=4pt,minimum width=4em] (Linear02) at ([shift={(-0.2em,-0.2em)}]Linear01.south west) {\footnotesize{Linear}};
\node [anchor=north] (Q) at ([xshift=0em,yshift=-1em]Linear02.south) {\footnotesize{$\mathbi{Q}$}}; \node [anchor=north] (Q) at ([xshift=0em,yshift=-1em]Linear02.south) {\footnotesize{$\mathbi{Q}$}};
\node [anchor=west,draw=black!30,inner sep=4pt,fill=ugreen!20!white,text=ugreen!20!white] (Linear1) at ([xshift=1.5em]Linear0.east) {\footnotesize{Linear}}; \node [anchor=west,draw=black!30,inner sep=4pt,fill=ugreen!30!white,text=ugreen!30!white,minimum width=4em] (Linear1) at ([xshift=1.5em]Linear0.east) {\footnotesize{Linear}};
\node [anchor=south west,draw=black!50,fill=ugreen!20!white,draw,inner sep=4pt,text=ugreen!20!white] (Linear11) at ([shift={(-0.2em,-0.2em)}]Linear1.south west) {\footnotesize{Linear}}; \node [anchor=south west,draw=black!50,fill=ugreen!30!white,draw,inner sep=4pt,text=ugreen!30!white,minimum width=4em] (Linear11) at ([shift={(-0.2em,-0.2em)}]Linear1.south west) {\footnotesize{Linear}};
\node [anchor=south west,fill=ugreen!20!white,draw,inner sep=4pt] (Linear12) at ([shift={(-0.2em,-0.2em)}]Linear11.south west) {\footnotesize{Linear}}; \node [anchor=south west,fill=ugreen!30!white,draw,inner sep=4pt,minimum width=4em] (Linear12) at ([shift={(-0.2em,-0.2em)}]Linear11.south west) {\footnotesize{Linear}};
\node [anchor=north] (K) at ([xshift=0em,yshift=-1em]Linear12.south) {\footnotesize{$\mathbi{K}$}}; \node [anchor=north] (K) at ([xshift=0em,yshift=-1em]Linear12.south) {\footnotesize{$\mathbi{K}$}};
\node [anchor=west,draw=black!30,inner sep=4pt,fill=ugreen!20!white,text=ugreen!20!white] (Linear2) at ([xshift=1.5em]Linear1.east) {\footnotesize{Linear}}; \node [anchor=west,draw=black!30,inner sep=4pt,fill=ugreen!30!white,text=ugreen!30!white,minimum width=4em] (Linear2) at ([xshift=1.5em]Linear1.east) {\footnotesize{Linear}};
\node [anchor=south west,draw=black!50,fill=ugreen!20!white,draw,inner sep=4pt,text=ugreen!20!white] (Linear21) at ([shift={(-0.2em,-0.2em)}]Linear2.south west) {\footnotesize{Linear}}; \node [anchor=south west,draw=black!50,fill=ugreen!30!white,draw,inner sep=4pt,text=ugreen!30!white,minimum width=4em] (Linear21) at ([shift={(-0.2em,-0.2em)}]Linear2.south west) {\footnotesize{Linear}};
\node [anchor=south west,fill=ugreen!20!white,draw,inner sep=4pt] (Linear22) at ([shift={(-0.2em,-0.2em)}]Linear21.south west) {\footnotesize{Linear}}; \node [anchor=south west,fill=ugreen!30!white,draw,inner sep=4pt,minimum width=4em] (Linear22) at ([shift={(-0.2em,-0.2em)}]Linear21.south west) {\footnotesize{Linear}};
\node [anchor=north] (V) at ([xshift=0em,yshift=-1em]Linear22.south) {\footnotesize{$\mathbi{V}$}}; \node [anchor=north] (V) at ([xshift=0em,yshift=-1em]Linear22.south) {\footnotesize{$\mathbi{V}$}};
\node [anchor=south,draw=black!30,minimum width=12em,minimum height=2em,inner sep=4pt,fill=blue!20!white] (Scale) at ([yshift=1em]Linear1.north) {\footnotesize{}}; % scaled dot-product attention
\node [anchor=south west,draw=black!50,minimum width=12em,minimum height=2em,fill=blue!20!white,draw,inner sep=4pt] (Scale1) at ([shift={(-0.2em,-0.2em)}]Scale.south west) {\footnotesize{}}; \node [anchor=south,draw=black!30,minimum width=16em,minimum height=2em,inner sep=4pt,fill=blue!30!white] (Scale) at ([yshift=1em]Linear1.north) {\footnotesize{}};
\node [anchor=south west,fill=blue!20!white,draw,minimum width=12em,minimum height=2em,inner sep=4pt] (Scale2) at ([shift={(-0.2em,-0.2em)}]Scale1.south west) {\footnotesize{Scaled Dot-Product Attention}}; \node [anchor=south west,draw=black!50,minimum width=16em,minimum height=2em,fill=blue!30!white,draw,inner sep=4pt] (Scale1) at ([shift={(-0.2em,-0.2em)}]Scale.south west) {\footnotesize{}};
\node [anchor=south west,fill=blue!30!white,draw,minimum width=16em,minimum height=2em,inner sep=4pt] (Scale2) at ([shift={(-0.2em,-0.2em)}]Scale1.south west) {\footnotesize{Scaled Dot-Product Attention}};
\node [anchor=south,draw,minimum width=4em,inner sep=4pt,fill=yellow!30] (Concat) at ([yshift=1em]Scale2.north) {\footnotesize{Concat}}; %
\node [anchor=south,draw,minimum width=6em,inner sep=4pt,fill=yellow!30] (Concat) at ([yshift=1em]Scale2.north) {\footnotesize{Concat}};
\node [anchor=south,draw,minimum width=4em,inner sep=4pt,fill=ugreen!20!white] (Linear) at ([yshift=1em]Concat.north) {\footnotesize{Linear}}; \node [anchor=south,draw,minimum width=6em,inner sep=4pt,fill=ugreen!30!white] (Linear) at ([yshift=1em]Concat.north) {\footnotesize{Linear}};
\draw [->] ([yshift=0.1em]Q.north) -- ([yshift=-0.1em]Linear02.south); \draw [->] ([yshift=0.1em]Q.north) -- ([yshift=-0.1em]Linear02.south);
......
Chapter12/Figures/figure-point-product-attention-model.tex
...@@ -6,13 +6,13 @@ ...@@ -6,13 +6,13 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\node [anchor=south west,fill=white,draw,inner sep=4pt,minimum width=4em,fill=blue!20!white] (MatMul) at (0,0) {\tiny{MatMul}}; \node [anchor=south west,fill=white,draw,inner sep=4pt,minimum width=4em,fill=blue!25!white] (MatMul) at (0,0) {\tiny{MatMul}};
\node [anchor=north] (Q1) at ([xshift=-1.4em,yshift=-1em]MatMul.south) {\footnotesize{$\mathbi{Q}$}}; \node [anchor=north] (Q1) at ([xshift=-1.4em,yshift=-1em]MatMul.south) {\footnotesize{$\mathbi{Q}$}};
\node [anchor=north] (K1) at ([xshift=1.4em,yshift=-1em]MatMul.south) {\footnotesize{$\mathbi{K}$}}; \node [anchor=north] (K1) at ([xshift=1.4em,yshift=-1em]MatMul.south) {\footnotesize{$\mathbi{K}$}};
\node [anchor=south,draw,inner sep=4pt,fill=yellow!30,minimum width=2.5em] (Scale3) at ([yshift=1em]MatMul.north) {\tiny{Scale}}; \node [anchor=south,draw,inner sep=4pt,fill=yellow!25,minimum width=2.5em] (Scale3) at ([yshift=1em]MatMul.north) {\tiny{Scale}};
\node [anchor=south,draw,inner sep=4pt,fill=purple!20,minimum width=3.5em] (Mask) at ([yshift=0.8em]Scale3.north) {\tiny{Mask(opt.)}}; \node [anchor=south,draw,inner sep=4pt,fill=purple!25,minimum width=3.5em] (Mask) at ([yshift=0.8em]Scale3.north) {\tiny{Mask(opt.)}};
\node [anchor=south,draw,inner sep=4pt,fill=ugreen!20!white] (SoftMax) at ([yshift=1em]Mask.north) {\tiny{SoftMax}}; \node [anchor=south,draw,inner sep=4pt,fill=ugreen!25!white] (SoftMax) at ([yshift=1em]Mask.north) {\tiny{SoftMax}};
\node [anchor=south,draw,minimum width=4em,inner sep=4pt,fill=blue!20!white] (MatMul1) at ([xshift=1.7em,yshift=1em]SoftMax.north) {\tiny{MatMul}}; \node [anchor=south,draw,minimum width=4em,inner sep=4pt,fill=blue!25!white] (MatMul1) at ([xshift=1.7em,yshift=1em]SoftMax.north) {\tiny{MatMul}};
\node [anchor=north] (V1) at ([xshift=2em]K1.north) {\footnotesize{$\mathbi{V}$}}; \node [anchor=north] (V1) at ([xshift=2em]K1.north) {\footnotesize{$\mathbi{V}$}};
\node [anchor=north] (null) at ([yshift=0.8em]MatMul1.north) {}; \node [anchor=north] (null) at ([yshift=0.8em]MatMul1.north) {};
...@@ -26,13 +26,13 @@ ...@@ -26,13 +26,13 @@
\draw [->] ([yshift=0.1em]MatMul1.north) -- ([yshift=0.8em]MatMul1.north); \draw [->] ([yshift=0.1em]MatMul1.north) -- ([yshift=0.8em]MatMul1.north);
{ {
\node [anchor=east] (line1) at ([xshift=-4em,yshift=1em]MatMul.west) {\scriptsize{自注意力机制的Query}}; \node [anchor=east] (line1) at ([xshift=-5em,yshift=1em]MatMul.west) {\scriptsize{自注意力机制的Query}};
\node [anchor=north west] (line2) at ([yshift=0.3em]line1.south west) {\scriptsize{Key和Value均来自同一句}}; \node [anchor=north west] (line2) at ([yshift=0.3em]line1.south west) {\scriptsize{Key和Value均来自同一句}};
\node [anchor=north west] (line3) at ([yshift=0.3em]line2.south west) {\scriptsize{子，编码-解码注意力机制}}; \node [anchor=north west] (line3) at ([yshift=0.3em]line2.south west) {\scriptsize{子，编码-解码注意力机制}};
\node [anchor=north west] (line4) at ([yshift=0.3em]line3.south west) {\scriptsize{与前面讲的一样}}; \node [anchor=north west] (line4) at ([yshift=0.3em]line3.south west) {\scriptsize{与前面讲的一样}};
} }
{ {
\node [anchor=west] (line11) at ([xshift=3em,yshift=0em]MatMul.east) {\scriptsize{Query和Key的转置进}}; \node [anchor=west] (line11) at ([xshift=5em,yshift=0em]MatMul.east) {\scriptsize{Query和Key的转置进}};
\node [anchor=north west] (line12) at ([yshift=0.3em]line11.south west) {\scriptsize{行点积,得到句子内部}}; \node [anchor=north west] (line12) at ([yshift=0.3em]line11.south west) {\scriptsize{行点积,得到句子内部}};
\node [anchor=north west] (line13) at ([yshift=0.3em]line12.south west) {\scriptsize{各个位置的相关性}}; \node [anchor=north west] (line13) at ([yshift=0.3em]line12.south west) {\scriptsize{各个位置的相关性}};
} }
...@@ -57,28 +57,28 @@ ...@@ -57,28 +57,28 @@
\begin{pgfonlayer}{background} \begin{pgfonlayer}{background}
{ {
\node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=green!10,drop shadow,draw=ugreen,minimum width=10em] [fit = (line1) (line2) (line3) (line4)] (box1) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=green!20,drop shadow,draw=ugreen,minimum width=10em] [fit = (line1) (line2) (line3) (line4)] (box1) {};
\node [rectangle,inner sep=0.1em,rounded corners=1pt,very thick,dotted,draw=ugreen] [fit = (Q1) (K1) (V1)] (box0) {}; \node [rectangle,inner sep=0.1em,rounded corners=1pt,very thick,dotted,draw=ugreen] [fit = (Q1) (K1) (V1)] (box0) {};
\draw [->,dotted,very thick,ugreen] ([yshift=-1.5em,xshift=1.8em]box1.east) -- ([yshift=-1.5em,xshift=0.1em]box1.east); \draw [->,dotted,very thick,ugreen] ([yshift=-1.5em,xshift=2.8em]box1.east) -- ([yshift=-1.5em,xshift=0.1em]box1.east);
} }
{ {
\node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=blue!20!white,drop shadow,draw=blue] [fit = (line11) (line12) (line13)] (box2) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=blue!25!white,drop shadow,draw=blue] [fit = (line11) (line12) (line13)] (box2) {};
\draw [->,dotted,very thick,blue] ([yshift=1em,xshift=-2.8em]box2.west) -- ([yshift=1em,xshift=-0.1em]box2.west); \draw [->,dotted,very thick,blue] ([yshift=1em,xshift=-4.8em]box2.west) -- ([yshift=1em,xshift=-0.1em]box2.west);
} }
{ {
\node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=yellow!20,drop shadow,draw=black] [fit = (line21) (line22) (line23)] (box3) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=yellow!25,drop shadow,draw=black] [fit = (line21) (line22) (line23)] (box3) {};
\draw [->,dotted,very thick,black] ([xshift=0.1em]Scale3.east) .. controls +(east:1) and +(west:1) .. ([yshift=1.0em]box3.west) ; \draw [->,dotted,very thick,black] ([xshift=0.1em]Scale3.east) .. controls +(east:1) and +(west:1) .. ([yshift=1.0em]box3.west) ;
} }
{ {
\node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=red!10,drop shadow,draw=red] [fit = (line31) (line32) (line33) (line34)] (box4) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=red!20,drop shadow,draw=red] [fit = (line31) (line32) (line33) (line34)] (box4) {};
\draw [->,dotted,very thick,red] ([yshift=-1.2em,xshift=2.2em]box4.east) -- ([yshift=-1.2em,xshift=0.1em]box4.east); \draw [->,dotted,very thick,red] ([yshift=-1.2em,xshift=3.2em]box4.east) -- ([yshift=-1.2em,xshift=0.1em]box4.east);
} }
{ {
\node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=blue!20!white,drop shadow,draw=blue] [fit = (line41) (line42)] (box5) {}; \node [rectangle,inner sep=0.2em,rounded corners=1pt,fill=blue!25!white,drop shadow,draw=blue] [fit = (line41) (line42)] (box5) {};
\draw [->,dotted,very thick,blue] ([yshift=-0.3em,xshift=-1em]box5.west) -- ([yshift=-0.3em,xshift=-0.1em]box5.west); \draw [->,dotted,very thick,blue] ([yshift=-0.3em,xshift=-3em]box5.west) -- ([yshift=-0.3em,xshift=-0.1em]box5.west);
} }
\end{pgfonlayer} \end{pgfonlayer}
......
Chapter12/Figures/figure-position-of-difference-and-layer-regularization-in-the-model.tex
...@@ -4,12 +4,12 @@ ...@@ -4,12 +4,12 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{Sanode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=orange!20]; \tikzstyle{Sanode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=orange!30];
\tikzstyle{Resnode} = [minimum height=1.1em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=yellow!20]; \tikzstyle{Resnode} = [minimum height=1.1em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=yellow!30];
\tikzstyle{ffnnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw]; \tikzstyle{ffnnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw];
\tikzstyle{outputnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw]; \tikzstyle{outputnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw];
\tikzstyle{inputnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=red!10]; \tikzstyle{inputnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=red!20];
\tikzstyle{posnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=black!5!white]; \tikzstyle{posnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=black!10!white];
\tikzstyle{standard} = [rounded corners=3pt] \tikzstyle{standard} = [rounded corners=3pt]
\node [Sanode,anchor=west] (sa1) at (0,0) {\tiny{$\textbf{Self-Attention}$}}; \node [Sanode,anchor=west] (sa1) at (0,0) {\tiny{$\textbf{Self-Attention}$}};
......
Chapter12/Figures/figure-position-of-feedforward-neural-network-in-the-model.tex
...@@ -2,12 +2,12 @@ ...@@ -2,12 +2,12 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{Sanode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=orange!20]; \tikzstyle{Sanode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=orange!30];
\tikzstyle{Resnode} = [minimum height=1.1em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=yellow!20]; \tikzstyle{Resnode} = [minimum height=1.1em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=yellow!30];
\tikzstyle{ffnnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=blue!20]; \tikzstyle{ffnnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=blue!30];
\tikzstyle{outputnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw]; \tikzstyle{outputnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw];
\tikzstyle{inputnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=red!10]; \tikzstyle{inputnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=red!20];
\tikzstyle{posnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=black!5!white]; \tikzstyle{posnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=black!10!white];
\tikzstyle{standard} = [rounded corners=3pt] \tikzstyle{standard} = [rounded corners=3pt]
\node [Sanode,anchor=west] (sa1) at (0,0) {\tiny{$\textbf{Self-Attention}$}}; \node [Sanode,anchor=west] (sa1) at (0,0) {\tiny{$\textbf{Self-Attention}$}};
......
Chapter12/Figures/figure-position-of-self-attention-mechanism-in-the-model.tex
...@@ -3,12 +3,12 @@ ...@@ -3,12 +3,12 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{Sanode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=orange!20]; \tikzstyle{Sanode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=orange!30];
\tikzstyle{Resnode} = [minimum height=1.1em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw]; \tikzstyle{Resnode} = [minimum height=1.1em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw];
\tikzstyle{ffnnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw]; \tikzstyle{ffnnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw];
\tikzstyle{outputnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw]; \tikzstyle{outputnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw];
\tikzstyle{inputnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=red!10]; \tikzstyle{inputnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=red!20];
\tikzstyle{posnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=black!5!white]; \tikzstyle{posnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=black!10!white];
\tikzstyle{standard} = [rounded corners=3pt] \tikzstyle{standard} = [rounded corners=3pt]
\node [Sanode,anchor=west] (sa1) at (0,0) {\tiny{$\textbf{Self-Attention}$}}; \node [Sanode,anchor=west] (sa1) at (0,0) {\tiny{$\textbf{Self-Attention}$}};
......
Chapter12/Figures/figure-process-of-5.tex
...@@ -7,9 +7,9 @@ ...@@ -7,9 +7,9 @@
\node(tbq) at ([xshift=0.5em,yshift=0]atten.east){ \node(tbq) at ([xshift=0.5em,yshift=0]atten.east){
\begin{tabular}{|c|} \begin{tabular}{|c|}
\hline \hline
\rowcolor{yellow!20} \\ \hline \rowcolor{yellow!30} \\ \hline
\rowcolor{yellow!20} \\ \hline \rowcolor{yellow!30} \\ \hline
\rowcolor{yellow!20} \\ \hline \rowcolor{yellow!30} \\ \hline
\end{tabular} \end{tabular}
}; };
\node at ([xshift=0em,yshift=0.5em]tbq.north){$\mathbi{Q}$}; \node at ([xshift=0em,yshift=0.5em]tbq.north){$\mathbi{Q}$};
...@@ -20,9 +20,9 @@ ...@@ -20,9 +20,9 @@
\node(tbk) at ([xshift=1em,yshift=0]tbq.east){ \node(tbk) at ([xshift=1em,yshift=0]tbq.east){
\begin{tabular}{|c|} \begin{tabular}{|c|}
\hline \hline
\rowcolor{blue!20} \\ \hline \rowcolor{blue!30} \\ \hline
\rowcolor{blue!20} \\ \hline \rowcolor{blue!30} \\ \hline
\rowcolor{blue!20} \\ \hline \rowcolor{blue!30} \\ \hline
\end{tabular} \end{tabular}
}; };
\node at ([xshift=0em,yshift=0.5em]tbk.north){$\mathbi{K}$}; \node at ([xshift=0em,yshift=0.5em]tbk.north){$\mathbi{K}$};
...@@ -33,9 +33,9 @@ ...@@ -33,9 +33,9 @@
\node(tbv) at ([xshift=1em,yshift=0]tbk.east){ \node(tbv) at ([xshift=1em,yshift=0]tbk.east){
\begin{tabular}{|c|} \begin{tabular}{|c|}
\hline \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\end{tabular} \end{tabular}
}; };
\node at ([xshift=0em,yshift=0.5em]tbv.north){$\mathbi{V}$}; \node at ([xshift=0em,yshift=0.5em]tbv.north){$\mathbi{V}$};
...@@ -51,9 +51,9 @@ ...@@ -51,9 +51,9 @@
\node(tbq2) at ([xshift=0.5em,yshift=2em]sof1.east){ \node(tbq2) at ([xshift=0.5em,yshift=2em]sof1.east){
\begin{tabular}{|c|} \begin{tabular}{|c|}
\hline \hline
\rowcolor{yellow!20} \\ \hline \rowcolor{yellow!30} \\ \hline
\rowcolor{yellow!20} \\ \hline \rowcolor{yellow!30} \\ \hline
\rowcolor{yellow!20} \\ \hline \rowcolor{yellow!30} \\ \hline
\end{tabular} \end{tabular}
}; };
\node at ([xshift=0em,yshift=0.5em]tbq2.north){$\mathbi{Q}$}; \node at ([xshift=0em,yshift=0.5em]tbq2.north){$\mathbi{Q}$};
...@@ -66,7 +66,7 @@ ...@@ -66,7 +66,7 @@
\node(tbk2) at ([xshift=2em,yshift=0em]times.east){ \node(tbk2) at ([xshift=2em,yshift=0em]times.east){
\begin{tabular}{|l|l|l|} \begin{tabular}{|l|l|l|}
\hline \hline
\cellcolor{blue!20} & \cellcolor{blue!20} &\cellcolor{blue!20} \\ \hline \cellcolor{blue!30} & \cellcolor{blue!30} &\cellcolor{blue!30} \\ \hline
\end{tabular} \end{tabular}
}; };
\node at ([xshift=0em,yshift=0.5em]tbk2.north){$\mathbi{K}^{\mathrm{T}}$}; \node at ([xshift=0em,yshift=0.5em]tbk2.north){$\mathbi{K}^{\mathrm{T}}$};
...@@ -78,9 +78,9 @@ ...@@ -78,9 +78,9 @@
\node(mask) at ([xshift=3em,yshift=-2em]tbk2.east){ \node(mask) at ([xshift=3em,yshift=-2em]tbk2.east){
\begin{tabular}{|l|l|l|} \begin{tabular}{|l|l|l|}
\hline \hline
\cellcolor{green!20} &\cellcolor{green!20} &\cellcolor{green!20} \\ \hline \cellcolor{green!30} &\cellcolor{green!30} &\cellcolor{green!30} \\ \hline
\cellcolor{green!20} &\cellcolor{green!20} &\cellcolor{green!20} \\ \hline \cellcolor{green!30} &\cellcolor{green!30} &\cellcolor{green!30} \\ \hline
\cellcolor{green!20} &\cellcolor{green!20} &\cellcolor{green!20} \\ \hline \cellcolor{green!30} &\cellcolor{green!30} &\cellcolor{green!30} \\ \hline
\end{tabular} \end{tabular}
}; };
\node at ([xshift=0em,yshift=0.5em]mask.north){$\mathbi{Mask}$}; \node at ([xshift=0em,yshift=0.5em]mask.north){$\mathbi{Mask}$};
...@@ -93,9 +93,9 @@ ...@@ -93,9 +93,9 @@
\node(tbv2) at ([xshift=1.2em,yshift=0]mask.east){ \node(tbv2) at ([xshift=1.2em,yshift=0]mask.east){
\begin{tabular}{|c|} \begin{tabular}{|c|}
\hline \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\end{tabular} \end{tabular}
}; };
\node at ([xshift=0em,yshift=0.5em]tbv2.north){$\mathbi{V}$}; \node at ([xshift=0em,yshift=0.5em]tbv2.north){$\mathbi{V}$};
...@@ -108,9 +108,9 @@ ...@@ -108,9 +108,9 @@
\node(mid) at ([xshift=1.5em,yshift=0em]sof2.east){ \node(mid) at ([xshift=1.5em,yshift=0em]sof2.east){
\begin{tabular}{|l|l|l|} \begin{tabular}{|l|l|l|}
\hline \hline
\cellcolor{pink!30} &\cellcolor{pink!30} &\cellcolor{pink!30} \\ \hline \cellcolor{pink!40} &\cellcolor{pink!40} &\cellcolor{pink!40} \\ \hline
\cellcolor{pink!30} &\cellcolor{pink!30} &\cellcolor{pink!30} \\ \hline \cellcolor{pink!40} &\cellcolor{pink!40} &\cellcolor{pink!40} \\ \hline
\cellcolor{pink!30} &\cellcolor{pink!30} &\cellcolor{pink!30} \\ \hline \cellcolor{pink!40} &\cellcolor{pink!40} &\cellcolor{pink!40} \\ \hline
\end{tabular} \end{tabular}
}; };
% ) % )
...@@ -126,9 +126,9 @@ ...@@ -126,9 +126,9 @@
\node(tbv3) at ([xshift=0.5em,yshift=0]bra2.east){ \node(tbv3) at ([xshift=0.5em,yshift=0]bra2.east){
\begin{tabular}{|c|} \begin{tabular}{|c|}
\hline \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\end{tabular} \end{tabular}
}; };
\node at ([xshift=0em,yshift=0.5em]tbv3.north){$\mathbi{V}$}; \node at ([xshift=0em,yshift=0.5em]tbv3.north){$\mathbi{V}$};
...@@ -140,9 +140,9 @@ ...@@ -140,9 +140,9 @@
\node(result) at ([xshift=2em,yshift=0]eq3.east){ \node(result) at ([xshift=2em,yshift=0]eq3.east){
\begin{tabular}{|l|l|l|} \begin{tabular}{|l|l|l|}
\hline \hline
\cellcolor{red!20} &\cellcolor{red!20} &\cellcolor{red!20} \\ \hline \cellcolor{red!30} &\cellcolor{red!30} &\cellcolor{red!30} \\ \hline
\cellcolor{red!20}&\cellcolor{red!20} &\cellcolor{red!20} \\ \hline \cellcolor{red!30}&\cellcolor{red!30} &\cellcolor{red!30} \\ \hline
\cellcolor{red!20} &\cellcolor{red!20} &\cellcolor{red!20} \\ \hline \cellcolor{red!30} &\cellcolor{red!30} &\cellcolor{red!30} \\ \hline
\end{tabular} \end{tabular}
}; };
% x % x
...@@ -151,9 +151,9 @@ ...@@ -151,9 +151,9 @@
\node(tbv4) at ([xshift=0.5em,yshift=0]times.east){ \node(tbv4) at ([xshift=0.5em,yshift=0]times.east){
\begin{tabular}{|c|} \begin{tabular}{|c|}
\hline \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\rowcolor{orange!20} \\ \hline \rowcolor{orange!30} \\ \hline
\end{tabular} \end{tabular}
}; };
\node at ([xshift=0em,yshift=0.5em]tbv4.north){$\mathbi{V}$}; \node at ([xshift=0em,yshift=0.5em]tbv4.north){$\mathbi{V}$};
......
Chapter12/Figures/figure-self-att-vs-enco-deco-att.tex
\begin{tikzpicture} \begin{tikzpicture}
\node[rounded corners=1pt,minimum width=11.0em,minimum height=2.0em,fill=pink!30,draw=black](p1) at (0,0) {\small{Self-Attention}}; \node[rounded corners=1pt,minimum width=11.0em,minimum height=2.0em,fill=pink!60,draw=black](p1) at (0,0) {\small{Self-Attention}};
\node[anchor=north](word1) at ([xshift=0.0em,yshift=-2.0em]p1.south) {\small \mathbi{K}}; \node[anchor=north](word1) at ([xshift=0.0em,yshift=-2.0em]p1.south) {\small \mathbi{K}};
\node[anchor=west](word2) at ([xshift=2.2em]word1.east) {\small \mathbi{V}}; \node[anchor=west](word2) at ([xshift=2.2em]word1.east) {\small \mathbi{V}};
...@@ -19,7 +19,7 @@ ...@@ -19,7 +19,7 @@
\node[anchor=north](caption1) at ([xshift=0.0em,yshift=-9.5em]p1.south){\small{(a) Self-Attention的输入}}; \node[anchor=north](caption1) at ([xshift=0.0em,yshift=-9.5em]p1.south){\small{(a) Self-Attention的输入}};
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\node[anchor=west,rounded corners=1pt,minimum width=14.0em,minimum height=2.0em,fill=pink!30,draw=black](p2) at ([xshift=5.0em]p1.east){\small{Encoder-Decoder Attention}}; \node[anchor=west,rounded corners=1pt,minimum width=14.0em,minimum height=2.0em,fill=pink!50,draw=black](p2) at ([xshift=5.0em]p1.east){\small{Encoder-Decoder Attention}};
\node[anchor=north](word1-2) at ([xshift=0.0em,yshift=-2.0em]p2.south) {\small \mathbi{K}}; \node[anchor=north](word1-2) at ([xshift=0.0em,yshift=-2.0em]p2.south) {\small \mathbi{K}};
\node[anchor=west](word2-2) at ([xshift=2.2em]word1-2.east) {\small \mathbi{V}}; \node[anchor=west](word2-2) at ([xshift=2.2em]word1-2.east) {\small \mathbi{V}};
......
Chapter12/Figures/figure-transformer-input-and-position-encoding.tex
...@@ -6,8 +6,8 @@ ...@@ -6,8 +6,8 @@
\tikzstyle{Resnode} = [minimum height=1.1em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw]; \tikzstyle{Resnode} = [minimum height=1.1em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw];
\tikzstyle{ffnnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw]; \tikzstyle{ffnnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw];
\tikzstyle{outputnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw]; \tikzstyle{outputnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw];
\tikzstyle{inputnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=red!10]; \tikzstyle{inputnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=red!20];
\tikzstyle{posnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=black!5!white]; \tikzstyle{posnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=black!10!white];
\tikzstyle{standard} = [rounded corners=3pt] \tikzstyle{standard} = [rounded corners=3pt]
\node [Sanode,anchor=west] (sa1) at (0,0) {\tiny{$\textbf{Self-Attention}$}}; \node [Sanode,anchor=west] (sa1) at (0,0) {\tiny{$\textbf{Self-Attention}$}};
......
Chapter12/Figures/figure-transformer.tex
...@@ -2,12 +2,12 @@ ...@@ -2,12 +2,12 @@
\begin{tikzpicture} \begin{tikzpicture}
\begin{scope} \begin{scope}
\tikzstyle{Sanode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=orange!20]; \tikzstyle{Sanode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=orange!30];
\tikzstyle{Resnode} = [minimum height=1.1em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=yellow!20]; \tikzstyle{Resnode} = [minimum height=1.1em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=yellow!30];
\tikzstyle{ffnnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=blue!10]; \tikzstyle{ffnnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=blue!20];
\tikzstyle{outputnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=blue!30]; \tikzstyle{outputnode} = [minimum height=1.4em,minimum width=7em,inner sep=3pt,rounded corners=1.5pt,draw,fill=blue!40];
\tikzstyle{inputnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=red!10]; \tikzstyle{inputnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=red!20];
\tikzstyle{posnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=black!5!white]; \tikzstyle{posnode} = [minimum height=1.4em,minimum width=3.5em,inner sep=3pt,rounded corners=1.5pt,draw,fill=black!10!white];
\tikzstyle{standard} = [rounded corners=3pt] \tikzstyle{standard} = [rounded corners=3pt]
\node [Sanode,anchor=west] (sa1) at (0,0) {\tiny{$\textbf{Self-Attention}$}}; \node [Sanode,anchor=west] (sa1) at (0,0) {\tiny{$\textbf{Self-Attention}$}};
......
Chapter12/chapter12.tex
...@@ -132,7 +132,7 @@ ...@@ -132,7 +132,7 @@
\multicolumn{1}{l|}{ConvS2S} & 25.16 & 40.46 & 1.5$\times 10^{20}$ \\ \multicolumn{1}{l|}{ConvS2S} & 25.16 & 40.46 & 1.5$\times 10^{20}$ \\
\multicolumn{1}{l|}{MoE} & 26.03 & 40.56 & 1.2$\times 10^{20}$ \\ \multicolumn{1}{l|}{MoE} & 26.03 & 40.56 & 1.2$\times 10^{20}$ \\
\multicolumn{1}{l|}{Transformer (Base Model) } & 27.3 &38.1 & 3.3$\times 10^{18}$ \\ \multicolumn{1}{l|}{Transformer (Base Model) } & 27.3 &38.1 & 3.3$\times 10^{18}$ \\
\multicolumn{1}{l|}{Transformer (Big Model)} & {\small\sffamily\bfseries{28.4}} & {\small\sffamily\bfseries{41.8}} & 2.3$\times 10^{19}$ \\ \multicolumn{1}{l|}{Transformer (Big Model)} & {\small\bfnew{28.4}} & {\small\bfnew{41.8}} & 2.3$\times 10^{19}$ \\
\end{tabular} \end{tabular}
\end{table} \end{table}
%---------------------------------------------- %----------------------------------------------
...@@ -158,19 +158,19 @@ ...@@ -158,19 +158,19 @@
\begin{itemize} \begin{itemize}
\vspace{0.5em} \vspace{0.5em}
\item {\small\sffamily\bfseries{自注意力子层}}\index{自注意力子层}（Self-Attention Sub-layer）\index{Self-Attention Sub-layer}：使用自注意力机制对输入的序列进行新的表示； \item {\small\bfnew{自注意力子层}}\index{自注意力子层}（Self-Attention Sub-layer）\index{Self-Attention Sub-layer}：使用自注意力机制对输入的序列进行新的表示；
\vspace{0.5em} \vspace{0.5em}
\item {\small\sffamily\bfseries{前馈神经网络子层}}\index{前馈神经网络子层}（Feed-Forward Sub-layer）\index{Feed-Forward Sub-layer}：使用全连接的前馈神经网络对输入向量序列进行进一步变换； \item {\small\bfnew{前馈神经网络子层}}\index{前馈神经网络子层}（Feed-Forward Sub-layer）\index{Feed-Forward Sub-layer}：使用全连接的前馈神经网络对输入向量序列进行进一步变换；
\vspace{0.5em} \vspace{0.5em}
\item {\small\sffamily\bfseries{残差连接}}（标记为“Add”）：对于自注意力子层和前馈神经网络子层，都有一个从输入直接到输出的额外连接，也就是一个跨子层的直连。残差连接可以使深层网络的信息传递更为有效； \item {\small\bfnew{残差连接}}（标记为“Add”）：对于自注意力子层和前馈神经网络子层，都有一个从输入直接到输出的额外连接，也就是一个跨子层的直连。残差连接可以使深层网络的信息传递更为有效；
\vspace{0.5em} \vspace{0.5em}
\item {\small\sffamily\bfseries{层标准化}}（Layer Normalization）：自注意力子层和前馈神经网络子层进行最终输出之前，会对输出的向量进行层标准化，规范结果向量取值范围，这样易于后面进一步的处理。 \item {\small\bfnew{层标准化}}（Layer Normalization）：自注意力子层和前馈神经网络子层进行最终输出之前，会对输出的向量进行层标准化，规范结果向量取值范围，这样易于后面进一步的处理。
\vspace{0.5em} \vspace{0.5em}
\end{itemize} \end{itemize}
\parinterval 以上操作就构成了Transformer的一层，各个模块执行的顺序可以简单描述为：Self-Attention $\to$ Residual Connection $\to$ Layer Normalization $\to$ Feed Forward Network $\to$ Residual Connection $\to$ Layer Normalization。编码器可以包含多个这样的层，比如，可以构建一个六层编码器，每层都执行上面的操作。最上层的结果作为整个编码的结果，会被传入解码器。 \parinterval 以上操作就构成了Transformer的一层，各个模块执行的顺序可以简单描述为：Self-Attention $\to$ Residual Connection $\to$ Layer Normalization $\to$ Feed Forward Network $\to$ Residual Connection $\to$ Layer Normalization。编码器可以包含多个这样的层，比如，可以构建一个六层编码器，每层都执行上面的操作。最上层的结果作为整个编码的结果，会被传入解码器。
\parinterval 解码器的结构与编码器十分类似。它也是由若干层组成，每一层包含编码器中的所有结构，即：自注意力子层、前馈神经网络子层、残差连接和层标准化模块。此外，为了捕捉源语言的信息，解码器又引入了一个额外的{\small\sffamily\bfseries{编码-解码注意力子层}}\index{编码-解码注意力子层}（Encoder-Decoder Attention Sub-layer）\index{Encoder-Decoder Attention Sub-layer}。这个新的子层，可以帮助模型使用源语言句子的表示信息生成目标语言不同位置的表示。编码-解码注意力子层仍然基于自注意力机制，因此它和自注意力子层的结构是相同的，只是$\mathrm{query}$、$\mathrm{key}$、$\mathrm{value}$的定义不同。比如，在解码器端，自注意力子层的$\mathrm{query}$、$\mathrm{key}$、$\mathrm{value}$是相同的，它们都等于解码器每个位置的表示。而在编码-解码注意力子层中，$\mathrm{query}$是解码器每个位置的表示，此时$\mathrm{key}$和$\mathrm{value}$是相同的，等于编码器每个位置的表示。图\ref{fig:12-5}给出了这两种不同注意力子层输入的区别。 \parinterval 解码器的结构与编码器十分类似。它也是由若干层组成，每一层包含编码器中的所有结构，即：自注意力子层、前馈神经网络子层、残差连接和层标准化模块。此外，为了捕捉源语言的信息，解码器又引入了一个额外的{\small\bfnew{编码-解码注意力子层}}\index{编码-解码注意力子层}（Encoder-Decoder Attention Sub-layer）\index{Encoder-Decoder Attention Sub-layer}。这个新的子层，可以帮助模型使用源语言句子的表示信息生成目标语言不同位置的表示。编码-解码注意力子层仍然基于自注意力机制，因此它和自注意力子层的结构是相同的，只是$\mathrm{query}$、$\mathrm{key}$、$\mathrm{value}$的定义不同。比如，在解码器端，自注意力子层的$\mathrm{query}$、$\mathrm{key}$、$\mathrm{value}$是相同的，它们都等于解码器每个位置的表示。而在编码-解码注意力子层中，$\mathrm{query}$是解码器每个位置的表示，此时$\mathrm{key}$和$\mathrm{value}$是相同的，等于编码器每个位置的表示。图\ref{fig:12-5}给出了这两种不同注意力子层输入的区别。
%---------------------------------------------- %----------------------------------------------
\begin{figure}[htp] \begin{figure}[htp]
...@@ -319,7 +319,7 @@ ...@@ -319,7 +319,7 @@
\subsection{多头注意力机制} \subsection{多头注意力机制}
\parinterval Transformer中使用的另一项重要技术是{\small\sffamily\bfseries{多头注意力机制}}\index{多头注意力机制}（Multi-head Attention）\index{Multi-head Attention}。“多头”可以理解成将原来的$\mathbi{Q}$、$\mathbi{K}$、$\mathbi{V}$按照隐层维度平均切分成多份。假设切分$h$份，那么最终会得到$\mathbi{Q} = \{ \mathbi{Q}_1,...,\mathbi{Q}_h \}$，$\mathbi{K}=\{ \mathbi{K}_1,...,\mathbi{K}_h \}$，$\mathbi{V}=\{ \mathbi{V}_1,...,\mathbi{V}_h \}$。多头注意力就是用每一个切分得到的$\mathbi{Q}$，$\mathbi{K}$，$\mathbi{V}$独立的进行注意力计算，即第$i$个头的注意力计算结果$\mathbi{head}_i = \textrm{Attention}(\mathbi{Q}_i,\mathbi{K}_i, \mathbi{V}_i)$。 \parinterval Transformer中使用的另一项重要技术是{\small\bfnew{多头注意力机制}}\index{多头注意力机制}（Multi-head Attention）\index{Multi-head Attention}。“多头”可以理解成将原来的$\mathbi{Q}$、$\mathbi{K}$、$\mathbi{V}$按照隐层维度平均切分成多份。假设切分$h$份，那么最终会得到$\mathbi{Q} = \{ \mathbi{Q}_1,...,\mathbi{Q}_h \}$，$\mathbi{K}=\{ \mathbi{K}_1,...,\mathbi{K}_h \}$，$\mathbi{V}=\{ \mathbi{V}_1,...,\mathbi{V}_h \}$。多头注意力就是用每一个切分得到的$\mathbi{Q}$，$\mathbi{K}$，$\mathbi{V}$独立的进行注意力计算，即第$i$个头的注意力计算结果$\mathbi{head}_i = \textrm{Attention}(\mathbi{Q}_i,\mathbi{K}_i, \mathbi{V}_i)$。
\parinterval 下面根据图\ref{fig:12-12}详细介绍多头注意力的计算过程： \parinterval 下面根据图\ref{fig:12-12}详细介绍多头注意力的计算过程：
......
Chapter15/Figures/figure-learning-rate.tex
...@@ -15,7 +15,7 @@ ...@@ -15,7 +15,7 @@
] ]
\addplot[red,line width=1.25pt] coordinates {(0,0) (1.6,2) (1.8,1.888) (2,1.787) (2.5,1.606) (3,1.462) (3.5,1.3549) (4,1.266) (4.5,1.193) (5,1.131)}; \addplot[red,line width=1.25pt] coordinates {(0,0) (1.6,2) (1.8,1.888) (2,1.787) (2.5,1.606) (3,1.462) (3.5,1.3549) (4,1.266) (4.5,1.193) (5,1.131)};
\addlegendentry{\scriptsize 原始学习率} \addlegendentry{\scriptsize 原始的学习率}
%\addplot[red,line width=1.25pt] coordinates {(0,0) (8000,0.002) (10000,0.00179) (12000,0.00163) (12950,0.001572)}; %\addplot[red,line width=1.25pt] coordinates {(0,0) (8000,0.002) (10000,0.00179) (12000,0.00163) (12950,0.001572)};
\addplot[blue,line width=1.25pt] coordinates {(0,0) (0.8,2) (0.9906,1.7983)}; \addplot[blue,line width=1.25pt] coordinates {(0,0) (0.8,2) (0.9906,1.7983)};
%\addplot[red,line width=1.25pt] coordinates {(0,0) (8000,0.002) (9906,0.0017983)}; %\addplot[red,line width=1.25pt] coordinates {(0,0) (8000,0.002) (9906,0.0017983)};
......
Chapter15/Figures/figure-relative-position-weight.tex
\begin{tikzpicture} \begin{tikzpicture}
\tikzstyle{node1} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=green!80] \tikzstyle{node1} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=green!80]
\tikzstyle{node2} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=green!40] \tikzstyle{node2} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=green!50]
\tikzstyle{node3} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=green!20] \tikzstyle{node3} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=green!20]
\tikzstyle{node4} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt] \tikzstyle{node4} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt]
\tikzstyle{node5} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=red!20] \tikzstyle{node5} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=red!20]
\tikzstyle{node6} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=red!40] \tikzstyle{node6} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=red!50]
\tikzstyle{node7} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=red!80] \tikzstyle{node7} = [anchor=center,draw,minimum height=2em,minimum width=2em,inner sep=0pt,fill=red!80]
\begin{scope}[scale=1.0] \begin{scope}[scale=1.0]
......
Chapter15/Figures/figure-weight-visualization-of-convergence-DLCL-network.tex
\begin{tikzpicture}[node distance = 0,scale = 1] \begin{tikzpicture}[node distance = 0,scale = 1]
\tikzstyle{every node}=[scale=1] \tikzstyle{every node}=[scale=1]
\node[draw=white] (input) at (0,0){\includegraphics[width=0.62\textwidth]{./Chapter15/Figures/DLCL-picture.png}};(1.9,-1.4); \node[draw=white] (input) at (0,0){\includegraphics[width=0.62\textwidth]{./Chapter15/Figures/DLCL-picture.png}};
\node[scale = 2] at (4.5,3.6){4}; {\footnotesize
\node[scale = 2] at (4.5,1.8){2}; \node[anchor=center] at (4.1,3){4};
\node[scale = 2] at (4.5,0){0}; \node[anchor=center] at (4.1,1.5){2};
\node[scale = 2] at (4.5,-1.8){-2}; \node[anchor=center] at (4.1,0){0};
\node[scale = 2] at (4.5,-3.6){-4}; \node[anchor=center] at (4.1,-1.5){-2};
\node[scale = 1.5] at (-4.5,3.75){$\rm x_{1}$}; \node[anchor=center] at (4.1,-3){-4};
\node[scale = 1.5] at (-4.5,2.5){$\rm x_{6}$}; \node[anchor=center] at (-4.2,3.6){$ x_{1}$};
\node[scale = 1.5] at (-4.5,1.4){$\rm x_{11}$}; \node[anchor=center] at (-4.2,2.45){$ x_{6}$};
\node[scale = 1.5] at (-4.5,0.1){$\rm x_{16}$}; \node[anchor=center] at (-4.2,1.3){$ x_{11}$};
\node[scale = 1.5] at (-4.5,-1.1){$\rm x_{21}$}; \node[anchor=center] at (-4.2,0.15){$ x_{16}$};
\node[scale = 1.5] at (-4.5,-2.3){$\rm x_{26}$}; \node[anchor=center] at (-4.2,-1){$ x_{21}$};
\node[scale = 1.5] at (-4.5,-3.4){$\rm x_{31}$}; \node[anchor=center] at (-4.2,-2.15){$ x_{26}$};
\node[scale = 1.5] at (-3.8,-4){$\rm y_{0}$}; \node[anchor=center] at (-4.2,-3.3){$ x_{31}$};
\node[scale = 1.5] at (-2.7,-4){$\rm y_{5}$}; \node[anchor=center] at (-3.75,-3.8){$ y_{0}$};
\node[scale = 1.5] at (-1.5,-4){$\rm y_{10}$}; \node[anchor=center] at (-2.6,-3.8){$ y_{5}$};
\node[scale = 1.5] at (-0.3,-4){$\rm y_{15}$}; \node[anchor=center] at (-1.45,-3.8){$ y_{10}$};
\node[scale = 1.5] at (0.9,-4){$\rm y_{20}$}; \node[anchor=center] at (-0.3,-3.8){$ y_{15}$};
\node[scale = 1.5] at (2.1,-4){$\rm y_{25}$}; \node[anchor=center] at (0.85,-3.8){$ y_{20}$};
\node[scale = 1.5] at (3.3,-4){$\rm y_{30}$}; \node[anchor=center] at (2,-3.8){$ y_{25}$};
\node[anchor=center] at (3.15,-3.8){$ y_{30}$};
}
\end{tikzpicture} \end{tikzpicture}
\ No newline at end of file
Chapter16/chapter16.tex
...@@ -244,9 +244,9 @@ ...@@ -244,9 +244,9 @@
\end{figure} \end{figure}
%---------------------------------------------- %----------------------------------------------
\parinterval 在神经机器翻译中，应用多任务学习的主要策略是将翻译任务作为主任务，同时设置一些仅使用单语数据的子任务，通过这些子任务来捕捉单语数据中的语言知识\upcite{DBLP:conf/emnlp/DomhanH17,DBLP:conf/emnlp/ZhangZ16,DBLP:journals/corr/LuongLSVK15}。一种多任务学习的方法是利用源语言单语数据，通过单个编码器对源语言数据进行建模，再分别使用两个解码器来学习源语言排序和翻译任务。源语言排序任务是指利用预排序规则对源语言句子中词的顺序进行调整\upcite{DBLP:conf/emnlp/WangCK07}，可以通过单语数据来构造训练数据，从而使编码器被训练得更加充分\upcite{DBLP:conf/emnlp/ZhangZ16}，如图\ref{fig:16-7}所示，图中$y_{<}$表示当前时刻之前的译文，$x_{<}$表示源语言句子中词的顺序调整后的句子。 \parinterval 在神经机器翻译中，应用多任务学习的主要策略是将翻译任务作为主任务，同时设置一些仅使用单语数据的子任务，通过这些子任务来捕捉单语数据中的语言知识\upcite{DBLP:conf/emnlp/DomhanH17,DBLP:conf/emnlp/ZhangZ16,DBLP:journals/corr/LuongLSVK15}。一种多任务学习的方法是利用源语言单语数据，通过单个编码器对源语言数据进行建模，再分别使用两个解码器来学习源语言排序和翻译任务。源语言排序任务是指利用预排序规则对源语言句子中词的顺序进行调整\upcite{DBLP:conf/emnlp/WangCK07}，可以通过单语数据来构造训练数据，从而使编码器被训练得更加充分\upcite{DBLP:conf/emnlp/ZhangZ16}，如图\ref{fig:16-7}所示，图中$y_{<}$表示当前时刻之前的单词序列，$x_{<}$表示源语言句子中词的顺序调整后的句子。
\parinterval 虽然神经机器翻译模型可以看作一种语言生成模型，但生成过程中却依赖于源语言信息，因此无法直接利用目标语言单语数据进行多任务学习。针对这个问题，可以对原有翻译模型结构进行修改，在解码器底层增加一个语言模型子层，这个子层用于学习语言模型任务，与编码器端是完全独立的，如图\ref{fig:16-8}所示\upcite{DBLP:conf/emnlp/DomhanH17}，图中$y_{<}$表示当前时刻之前的译文，$z_{<}$表示当前时刻之前的单语数据。在训练过程中，分别将双语数据和单语数据送入翻译模型和语言模型进行计算，双语数据训练产生的梯度用于对整个模型进行参数更新，而单语数据产生的梯度只对语言模型子层进行参数更新。 \parinterval 虽然神经机器翻译模型可以看作一种语言生成模型，但生成过程中却依赖于源语言信息，因此无法直接利用目标语言单语数据进行多任务学习。针对这个问题，可以对原有翻译模型结构进行修改，在解码器底层增加一个语言模型子层，这个子层用于学习语言模型任务，与编码器端是完全独立的，如图\ref{fig:16-8}所示\upcite{DBLP:conf/emnlp/DomhanH17}，图中$y_{<}$表示当前时刻之前的单词序列，$z_{<}$表示当前时刻之前的单语数据。在训练过程中，分别将双语数据和单语数据送入翻译模型和语言模型进行计算，双语数据训练产生的梯度用于对整个模型进行参数更新，而单语数据产生的梯度只对语言模型子层进行参数更新。
%---------------------------------------------- %----------------------------------------------
\begin{figure}[htp] \begin{figure}[htp]
......
Chapter17/Figures/figure-an-end-to-end-voice-translation-model-based-on-transformer.tex
\begin{tikzpicture} \begin{tikzpicture}
\tikzstyle{layer}=[draw,rounded corners=2pt,font=\scriptsize,align=center,minimum width=5em] \tikzstyle{layer}=[draw,rounded corners=2pt,font=\scriptsize,align=center,minimum width=7.1em]
\tikzstyle{word}=[font=\scriptsize] \tikzstyle{word}=[font=\scriptsize]
%%%%encoder
\node[layer,fill=red!20] (en_sa) at (0,0){Multi-Head \\ Attention}; \node[layer,fill=red!20] (en_sa) at (0,0){Multi-Head \\ Attention};
\node[layer,anchor=south,fill=green!20] (en_ffn) at ([yshift=1.4em]en_sa.north){Feed Forward \\ Network}; \node[anchor=south,layer,fill=yellow!20](en_add1) at ([yshift=1.0em]en_sa.north) {Add \& LayerNorm};
\node[draw,circle,inner sep=0pt, minimum size=1em,anchor=north] (en_add) at ([yshift=-1.4em]en_sa.south){}; \node[layer,anchor=south,fill=green!20] (en_ffn) at ([yshift=1.0em]en_add1.north){Feed Forward \\ Network};
\draw[] (en_add.90) -- (en_add.-90); \node[anchor=south,layer,fill=yellow!20](en_add2) at ([yshift=1.0em]en_ffn.north) {Add \& LayerNorm};
\draw[] (en_add.0) -- (en_add.180); \node[draw,circle,inner sep=0pt, minimum size=1em,anchor=north,thick] (en_add) at ([yshift=-1.4em]en_sa.south){};
\node[layer,anchor=north,fill=yellow!20] (en_cnn) at ([yshift=-1.4em]en_add.south){CNN}; \draw[thick] (en_add.90) -- (en_add.-90);
\draw[thick] (en_add.0) -- (en_add.180);
\node[layer,anchor=north,fill=yellow!20] (en_cnn) at ([yshift=-1.0em]en_add.south){CNN};
\node[anchor=east,font=\scriptsize,align=center] (en_pos) at ([xshift=-2em]en_add.west){位置编码};
\node[anchor=north,font=\scriptsize,align=center] (en_input) at ([yshift=-1em]en_cnn.south){源语言语音特征\\(FBank/MFCC)};
\draw[->,thick] (en_input.90) -- ([yshift=-0.1em]en_cnn.-90);
\draw[->,thick] ([yshift=0.1em]en_cnn.90) -- ([yshift=-0.1em]en_add.-90);
\draw[->,thick] ([yshift=0.1em]en_add.90) -- ([yshift=-0.1em]en_sa.-90);
\draw[->,thick] ([yshift=0.1em]en_sa.90) -- ([yshift=-0.1em]en_add1.-90);
\draw[->,thick] ([yshift=0.1em]en_add1.90) -- ([yshift=-0.1em]en_ffn.-90);
\draw[->,thick] ([yshift=0.1em]en_ffn.90) --([yshift=-0.1em]en_add2.-90);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]en_sa.south)--([yshift=-0.6em,xshift=-4.0em]en_sa.south)--([xshift=-0.43em]en_add1.west)--(en_add1.west);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]en_ffn.south)--([yshift=-0.6em,xshift=-4.0em]en_ffn.south)--([xshift=-0.43em]en_add2.west)--(en_add2.west);
\node[draw,circle,inner sep=0pt, minimum size=1em,anchor=west] (de_add) at ([xshift=7em]en_add.east){}; %%%%decoder
\draw[] (de_add.90) -- (de_add.-90); \node[draw,circle,inner sep=0pt, minimum size=1em,anchor=west,thick] (de_add) at ([xshift=9em]en_add.east){};
\draw[] (de_add.0) -- (de_add.180); \draw[thick] (de_add.90) -- (de_add.-90);
\draw[thick] (de_add.0) -- (de_add.180);
\node[layer,anchor=south,fill=red!20] (de_sa) at ([yshift=1.4em]de_add.north){Masked \\Multi-Head\\Attention}; \node[layer,anchor=south,fill=red!20] (de_sa) at ([yshift=1.4em]de_add.north){Masked \\Multi-Head\\Attention};
\node[layer,anchor=south,fill=red!20] (de_ca) at ([yshift=1.4em]de_sa.north){Multi-Head \\ Attention}; \node[anchor=south,layer,fill=yellow!20](de_add1) at ([yshift=1.0em]de_sa.north) {Add \& LayerNorm};
\node[layer,anchor=south,fill=green!20] (de_ffn) at ([yshift=1.4em]de_ca.north){Feed Forward \\ Network}; \node[layer,anchor=south,fill=red!20] (de_ca) at ([yshift=1.0em]de_add1.north){Multi-Head \\ Attention};
\node[anchor=south,layer,fill=yellow!20](de_add2) at ([yshift=1.0em]de_ca.north) {Add \& LayerNorm};
\node[layer,anchor=south,fill=green!20] (de_ffn) at ([yshift=1.0em]de_add2.north){Feed Forward \\ Network};
\node[anchor=south,layer,fill=yellow!20](de_add3) at ([yshift=1.0em]de_ffn.north) {Add \& LayerNorm};
\node[layer,anchor=south,fill=blue!20] (sf) at ([yshift=1.2em]de_add3.north){Softmax};
\node[anchor=north,font=\scriptsize,align=center] (de_input) at ([yshift=-1.1em]de_add.south){目标语言文本\\编码表示};
\node[layer,anchor=south,fill=blue!20] (sf) at ([yshift=1.6em]de_ffn.north){Softmax}; \node[anchor=west,font=\scriptsize,align=center] (de_pos) at ([xshift=2em]de_add.east){位置编码};
%\node[layer,anchor=south,fill=orange!20] (output) at ([yshift=1.4em]sf.north){STLoss};
\node[anchor=north,font=\scriptsize,align=center] (en_input) at ([yshift=-1em]en_cnn.south){语音特征\\(FBank/MFCC)}; \draw[->,thick] (de_input.90) -- ([yshift=-0.1em]de_add.-90);
\node[anchor=north,font=\scriptsize,align=center] (de_input) at ([yshift=-1.1em]de_add.south){标注文本\\编码表示}; \draw[->,thick] ([yshift=0.1em]de_add.90) -- ([yshift=-0.1em]de_sa.-90);
\draw[->,thick] ([yshift=0.1em]de_sa.90) -- ([yshift=-0.1em]de_add1.-90);
\draw[->,thick] ([yshift=0.1em]de_add1.90) -- ([yshift=-0.1em]de_ca.-90);
\draw[->,thick] ([yshift=0.1em]de_ca.90) -- ([yshift=-0.1em]de_add2.-90);
\draw[->,thick] ([yshift=0.1em]de_add2.90) -- ([yshift=-0.1em]de_ffn.-90);
\draw[->,thick] ([yshift=0.1em]de_ffn.90) -- ([yshift=-0.1em]de_add3.-90);
\draw[->,thick] ([yshift=0.1em]de_add3.90) -- ([yshift=-0.1em]sf.-90);
\draw[->,thick] ([yshift=0.1em]sf.90) -- ([yshift=1.0em]sf.90);
\draw[->,thick] ([xshift=0.1em]en_pos.0) -- ([xshift=-0.1em]en_add.180);
\draw[->,thick] ([xshift=-0.1em]de_pos.180) -- ([xshift=0.1em]de_add.0);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]de_sa.south)--([yshift=-0.6em,xshift=4.0em]de_sa.south)--([xshift=0.43em]de_add1.east)--(de_add1.east);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]de_ca.south)--([yshift=-0.6em,xshift=4.0em]de_ca.south)--([xshift=0.43em]de_add2.east)--(de_add2.east);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]de_ffn.south)--([yshift=-0.6em,xshift=4.0em]de_ffn.south)--([xshift=0.43em]de_add3.east)--(de_add3.east);
\draw[->,rounded corners=2pt,thick] ([yshift=0.1em]en_add2.90) -- ([yshift=1.5em]en_add2.90) -- ([xshift=5.0em,yshift=1.5em]en_add2.90) -- ([xshift=-1.5em]de_ca.west) -- ([xshift=-0.1em]de_ca.west);
\node[anchor=east,font=\scriptsize,align=center] (en_pos) at ([xshift=-2em]en_add.west){位置编码};
\node[anchor=west,font=\scriptsize,align=center] (de_pos) at ([xshift=2em]de_add.east){位置编码};
\draw[->] (en_input.90) -- ([yshift=-0.1em]en_cnn.-90);
\draw[->] ([yshift=0.1em]en_cnn.90) -- ([yshift=-0.1em]en_add.-90);
\draw[->] ([yshift=0.1em]en_add.90) -- ([yshift=-0.1em]en_sa.-90);
\draw[->] ([yshift=0.1em]en_sa.90) -- ([yshift=-0.1em]en_ffn.-90);
\draw[->] (de_input.90) -- ([yshift=-0.1em]de_add.-90);
\draw[->] ([yshift=0.1em]de_add.90) -- ([yshift=-0.1em]de_sa.-90);
\draw[->] ([yshift=0.1em]de_sa.90) -- ([yshift=-0.1em]de_ca.-90);
\draw[->] ([yshift=0.1em]de_ca.90) -- ([yshift=-0.1em]de_ffn.-90);
\draw[->] ([yshift=0.1em]de_ffn.90) -- ([yshift=-0.1em]sf.-90);
\draw[->] ([yshift=0.1em]sf.90) -- ([yshift=1.5em]sf.90);
\draw[->] ([xshift=0.1em]en_pos.0) -- ([xshift=-0.1em]en_add.180);
\draw[->] ([xshift=-0.1em]de_pos.180) -- ([xshift=0.1em]de_add.0);
\draw[->,rounded corners=2pt] ([yshift=0.1em]en_ffn.90) -- ([yshift=2em]en_ffn.90) -- ([xshift=4em,yshift=2em]en_ffn.90) -- ([xshift=-1.5em]de_ca.west) -- ([xshift=-0.1em]de_ca.west);
\begin{pgfonlayer}{background} \begin{pgfonlayer}{background}
\node[draw=ugreen,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick][fit=(en_sa)(en_ffn)]{}; \node[draw=ugreen,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick,xshift=-0.2em,yshift=-0.2em][fit=(en_add1)(en_add2)(en_sa)(en_ffn)](box1){};
\node[draw=red,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick][fit=(de_sa)(de_ca)(de_ffn)]{}; \node[draw=red,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick,xshift=0.2em,yshift=-0.2em][fit=(de_sa)(de_ca)(de_ffn)(de_add3)](box2){};
\end{pgfonlayer} \end{pgfonlayer}
\node[anchor=east,font=\scriptsize,text=ugreen] at ([xshift=-0.1em]box1.west){$N \times$}; \node[anchor=east,font=\scriptsize,text=ugreen] at ([xshift=-0.1em]box1.west){$N \times$};
\node[anchor=west,font=\scriptsize,text=red] at ([xshift=0.1em]box2.east){$\times N$}; \node[anchor=west,font=\scriptsize,text=red] at ([xshift=0.1em]box2.east){$\times N$};
\node[anchor=east,font=\scriptsize] at ([xshift=-0.1em]en_cnn.west){$2 \times$}; \node[anchor=east,font=\scriptsize] at ([xshift=-0.1em]en_cnn.west){$2 \times$};
\node[anchor=east,font=\scriptsize,align=center,text=ugreen] at ([xshift=-0.1em,yshift=3em]box1.west){ST\\ 编码器}; \node[anchor=east,font=\scriptsize,align=center,text=ugreen] at ([xshift=-0.1em,yshift=3em]box1.west){ST \\ 编码器};
\node[anchor=west,font=\scriptsize,align=center,text=red] at ([xshift=0.1em,yshift=5em]box2.east){ST\\解码器}; \node[anchor=west,font=\scriptsize,align=center,text=red] at ([xshift=0.1em,yshift=5em]box2.east){ST \\ 解码器};
\end{tikzpicture} \end{tikzpicture}
\ No newline at end of file
Chapter17/Figures/figure-application-of-multimodal-machine-translation-to-multitask-learning.tex
...@@ -10,6 +10,9 @@ ...@@ -10,6 +10,9 @@
\node(figure)[draw=white,above of = decoder_right,yshift=6.5em,scale=0.25] {\includegraphics[width=0.62\textwidth]{./Chapter17/Figures/figure-bank-without-attention.jpg}}; \node(figure)[draw=white,above of = decoder_right,yshift=6.5em,scale=0.25] {\includegraphics[width=0.62\textwidth]{./Chapter17/Figures/figure-bank-without-attention.jpg}};
\node [anchor=south,scale=1.2] (node1) at ([xshift=-2.5em,yshift=4.5em]y.north) {\small{$x$：源语言文本数据}};
\node [anchor=north,scale=1.2] (node2) at ([xshift=0.57em]node1.south){\small{$y$：目标语言文本数据}};
\draw[->,thick](x)to(encoder); \draw[->,thick](x)to(encoder);
\draw[->,thick](encoder)to(decoder_left)node[right,xshift=-0.1cm,yshift=-1.25cm,scale=1.2]{\small{翻译}}; \draw[->,thick](encoder)to(decoder_left)node[right,xshift=-0.1cm,yshift=-1.25cm,scale=1.2]{\small{翻译}};
\draw[->,thick](decoder_left)to(y_hat); \draw[->,thick](decoder_left)to(y_hat);
......
Chapter17/Figures/figure-audio-processing.tex
...@@ -36,6 +36,7 @@ ...@@ -36,6 +36,7 @@
\draw[->,very thick](process_2.east)to([xshift=1.8cm]process_2.east); \draw[->,very thick](process_2.east)to([xshift=1.8cm]process_2.east);
%%%%音频 %%%%音频
\node(signal)[right of = process_2,xshift=5.5cm]{}; \node(signal)[right of = process_2,xshift=5.5cm]{};
\node(text_3)[below of = signal,yshift=-1.98cm,scale=1.3]{语音信号};
\draw[-,thick,]([xshift=-1.2cm]signal.center)--([xshift=1.2cm]signal.center); \draw[-,thick,]([xshift=-1.2cm]signal.center)--([xshift=1.2cm]signal.center);
\draw[-,thick]([xshift=-1cm,yshift=-0.8cm]signal.center)--([xshift=-0.9cm,yshift=0.4cm]signal.center)--([xshift=-0.8cm,yshift=-0.3cm]signal.center)--([xshift=-0.7cm,yshift=0.7cm]signal.center)--([xshift=-0.6cm,yshift=-0.1cm]signal.center)--([xshift=-0.5cm,yshift=0.3cm]signal.center)--([xshift=-0.4cm,yshift=-0.5cm]signal.center)--([xshift=-0.3cm,yshift=0.7cm]signal.center)--([xshift=-0.2cm,yshift=-0.2cm]signal.center)--([xshift=-0.1cm,yshift=0.4cm]signal.center)--([xshift=0cm,yshift=-0.9cm]signal.center)--([xshift=0.1cm,yshift=0.5cm]signal.center)--([xshift=0.2cm,yshift=-0.4cm]signal.center)--([xshift=0.3cm,yshift=0.3cm]signal.center)--([xshift=0.4cm,yshift=-0.2cm]signal.center)--([xshift=0.5cm,yshift=0.1cm]signal.center)--([xshift=0.6cm,yshift=-0.8cm]signal.center)--([xshift=0.7cm,yshift=0.4cm]signal.center)--([xshift=0.8cm,yshift=-0.6cm]signal.center)--([xshift=0.9cm,yshift=0.7cm]signal.center)--([xshift=1cm,yshift=-0.2cm]signal.center); \draw[-,thick]([xshift=-1cm,yshift=-0.8cm]signal.center)--([xshift=-0.9cm,yshift=0.4cm]signal.center)--([xshift=-0.8cm,yshift=-0.3cm]signal.center)--([xshift=-0.7cm,yshift=0.7cm]signal.center)--([xshift=-0.6cm,yshift=-0.1cm]signal.center)--([xshift=-0.5cm,yshift=0.3cm]signal.center)--([xshift=-0.4cm,yshift=-0.5cm]signal.center)--([xshift=-0.3cm,yshift=0.7cm]signal.center)--([xshift=-0.2cm,yshift=-0.2cm]signal.center)--([xshift=-0.1cm,yshift=0.4cm]signal.center)--([xshift=0cm,yshift=-0.9cm]signal.center)--([xshift=0.1cm,yshift=0.5cm]signal.center)--([xshift=0.2cm,yshift=-0.4cm]signal.center)--([xshift=0.3cm,yshift=0.3cm]signal.center)--([xshift=0.4cm,yshift=-0.2cm]signal.center)--([xshift=0.5cm,yshift=0.1cm]signal.center)--([xshift=0.6cm,yshift=-0.8cm]signal.center)--([xshift=0.7cm,yshift=0.4cm]signal.center)--([xshift=0.8cm,yshift=-0.6cm]signal.center)--([xshift=0.9cm,yshift=0.7cm]signal.center)--([xshift=1cm,yshift=-0.2cm]signal.center);
\end{tikzpicture} \end{tikzpicture}
\ No newline at end of file
Chapter17/Figures/figure-speech-recognition-model-based-on-transformer.tex
\begin{tikzpicture} \begin{tikzpicture}
\tikzstyle{layer}=[draw,rounded corners=2pt,font=\scriptsize,align=center,minimum width=5em] \tikzstyle{layer}=[draw,rounded corners=2pt,font=\scriptsize,align=center,minimum width=7.1em]
\tikzstyle{word}=[font=\scriptsize] \tikzstyle{word}=[font=\scriptsize]
%%%%encoder
\node[layer,fill=red!20] (en_sa) at (0,0){Multi-Head \\ Attention}; \node[layer,fill=red!20] (en_sa) at (0,0){Multi-Head \\ Attention};
\node[layer,anchor=south,fill=green!20] (en_ffn) at ([yshift=1.4em]en_sa.north){Feed Forward \\ Network}; \node[anchor=south,layer,fill=yellow!20](en_add1) at ([yshift=1.0em]en_sa.north) {Add \& LayerNorm};
\node[draw,circle,inner sep=0pt, minimum size=1em,anchor=north] (en_add) at ([yshift=-1.4em]en_sa.south){}; \node[layer,anchor=south,fill=green!20] (en_ffn) at ([yshift=1.0em]en_add1.north){Feed Forward \\ Network};
\draw[] (en_add.90) -- (en_add.-90); \node[anchor=south,layer,fill=yellow!20](en_add2) at ([yshift=1.0em]en_ffn.north) {Add \& LayerNorm};
\draw[] (en_add.0) -- (en_add.180); \node[draw,circle,inner sep=0pt, minimum size=1em,anchor=north,thick] (en_add) at ([yshift=-1.4em]en_sa.south){};
\node[layer,anchor=north,fill=yellow!20] (en_cnn) at ([yshift=-1.4em]en_add.south){CNN}; \draw[thick] (en_add.90) -- (en_add.-90);
\draw[thick] (en_add.0) -- (en_add.180);
\node[layer,anchor=north,fill=yellow!20] (en_cnn) at ([yshift=-1.0em]en_add.south){CNN};
\node[anchor=east,font=\scriptsize,align=center] (en_pos) at ([xshift=-2em]en_add.west){位置编码};
\node[anchor=north,font=\scriptsize,align=center] (en_input) at ([yshift=-1em]en_cnn.south){语音特征\\(FBank/MFCC)};
\draw[->,thick] (en_input.90) -- ([yshift=-0.1em]en_cnn.-90);
\draw[->,thick] ([yshift=0.1em]en_cnn.90) -- ([yshift=-0.1em]en_add.-90);
\draw[->,thick] ([yshift=0.1em]en_add.90) -- ([yshift=-0.1em]en_sa.-90);
\draw[->,thick] ([yshift=0.1em]en_sa.90) -- ([yshift=-0.1em]en_add1.-90);
\draw[->,thick] ([yshift=0.1em]en_add1.90) -- ([yshift=-0.1em]en_ffn.-90);
\draw[->,thick] ([yshift=0.1em]en_ffn.90) --([yshift=-0.1em]en_add2.-90);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]en_sa.south)--([yshift=-0.6em,xshift=-4.0em]en_sa.south)--([xshift=-0.43em]en_add1.west)--(en_add1.west);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]en_ffn.south)--([yshift=-0.6em,xshift=-4.0em]en_ffn.south)--([xshift=-0.43em]en_add2.west)--(en_add2.west);
\node[draw,circle,inner sep=0pt, minimum size=1em,anchor=west] (de_add) at ([xshift=7em]en_add.east){}; %%%%decoder
\draw[] (de_add.90) -- (de_add.-90); \node[draw,circle,inner sep=0pt, minimum size=1em,anchor=west,thick] (de_add) at ([xshift=9em]en_add.east){};
\draw[] (de_add.0) -- (de_add.180); \draw[thick] (de_add.90) -- (de_add.-90);
\draw[thick] (de_add.0) -- (de_add.180);
\node[layer,anchor=south,fill=red!20] (de_sa) at ([yshift=1.4em]de_add.north){Masked \\Multi-Head\\Attention}; \node[layer,anchor=south,fill=red!20] (de_sa) at ([yshift=1.4em]de_add.north){Masked \\Multi-Head\\Attention};
\node[layer,anchor=south,fill=red!20] (de_ca) at ([yshift=1.4em]de_sa.north){Multi-Head \\ Attention}; \node[anchor=south,layer,fill=yellow!20](de_add1) at ([yshift=1.0em]de_sa.north) {Add \& LayerNorm};
\node[layer,anchor=south,fill=green!20] (de_ffn) at ([yshift=1.4em]de_ca.north){Feed Forward \\ Network}; \node[layer,anchor=south,fill=red!20] (de_ca) at ([yshift=1.0em]de_add1.north){Multi-Head \\ Attention};
\node[anchor=south,layer,fill=yellow!20](de_add2) at ([yshift=1.0em]de_ca.north) {Add \& LayerNorm};
\node[layer,anchor=south,fill=blue!20] (sf) at ([yshift=1.6em]de_ffn.north){Softmax}; \node[layer,anchor=south,fill=green!20] (de_ffn) at ([yshift=1.0em]de_add2.north){Feed Forward \\ Network};
%\node[layer,anchor=south,fill=orange!20] (output) at ([yshift=1.4em]sf.north){Output Probabilities}; \node[anchor=south,layer,fill=yellow!20](de_add3) at ([yshift=1.0em]de_ffn.north) {Add \& LayerNorm};
\node[layer,anchor=south,fill=blue!20] (sf) at ([yshift=1.2em]de_add3.north){Softmax};
\node[anchor=north,font=\scriptsize,align=center] (en_input) at ([yshift=-1em]en_cnn.south){语音特征\\(FBank/MFCC)};
\node[anchor=north,font=\scriptsize,align=center] (de_input) at ([yshift=-1.1em]de_add.south){标注文本\\编码表示}; \node[anchor=north,font=\scriptsize,align=center] (de_input) at ([yshift=-1.1em]de_add.south){标注文本\\编码表示};
\node[anchor=east,font=\scriptsize,align=center] (en_pos) at ([xshift=-2em]en_add.west){位置编码};
\node[anchor=west,font=\scriptsize,align=center] (de_pos) at ([xshift=2em]de_add.east){位置编码}; \node[anchor=west,font=\scriptsize,align=center] (de_pos) at ([xshift=2em]de_add.east){位置编码};
\draw[->] (en_input.90) -- ([yshift=-0.1em]en_cnn.-90); \draw[->,thick] (de_input.90) -- ([yshift=-0.1em]de_add.-90);
\draw[->] ([yshift=0.1em]en_cnn.90) -- ([yshift=-0.1em]en_add.-90); \draw[->,thick] ([yshift=0.1em]de_add.90) -- ([yshift=-0.1em]de_sa.-90);
\draw[->] ([yshift=0.1em]en_add.90) -- ([yshift=-0.1em]en_sa.-90); \draw[->,thick] ([yshift=0.1em]de_sa.90) -- ([yshift=-0.1em]de_add1.-90);
\draw[->] ([yshift=0.1em]en_sa.90) -- ([yshift=-0.1em]en_ffn.-90); \draw[->,thick] ([yshift=0.1em]de_add1.90) -- ([yshift=-0.1em]de_ca.-90);
\draw[->] (de_input.90) -- ([yshift=-0.1em]de_add.-90); \draw[->,thick] ([yshift=0.1em]de_ca.90) -- ([yshift=-0.1em]de_add2.-90);
\draw[->] ([yshift=0.1em]de_add.90) -- ([yshift=-0.1em]de_sa.-90); \draw[->,thick] ([yshift=0.1em]de_add2.90) -- ([yshift=-0.1em]de_ffn.-90);
\draw[->] ([yshift=0.1em]de_sa.90) -- ([yshift=-0.1em]de_ca.-90); \draw[->,thick] ([yshift=0.1em]de_ffn.90) -- ([yshift=-0.1em]de_add3.-90);
\draw[->] ([yshift=0.1em]de_ca.90) -- ([yshift=-0.1em]de_ffn.-90); \draw[->,thick] ([yshift=0.1em]de_add3.90) -- ([yshift=-0.1em]sf.-90);
\draw[->] ([yshift=0.1em]de_ffn.90) -- ([yshift=-0.1em]sf.-90); \draw[->,thick] ([yshift=0.1em]sf.90) -- ([yshift=1.0em]sf.90);
\draw[->] ([yshift=0.1em]sf.90) -- ([yshift=1.5em]sf.90); \draw[->,thick] ([xshift=0.1em]en_pos.0) -- ([xshift=-0.1em]en_add.180);
\draw[->] ([xshift=0.1em]en_pos.0) -- ([xshift=-0.1em]en_add.180); \draw[->,thick] ([xshift=-0.1em]de_pos.180) -- ([xshift=0.1em]de_add.0);
\draw[->] ([xshift=-0.1em]de_pos.180) -- ([xshift=0.1em]de_add.0); \draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]de_sa.south)--([yshift=-0.6em,xshift=4.0em]de_sa.south)--([xshift=0.43em]de_add1.east)--(de_add1.east);
\draw[->,rounded corners=2pt] ([yshift=0.1em]en_ffn.90) -- ([yshift=2em]en_ffn.90) -- ([xshift=4em,yshift=2em]en_ffn.90) -- ([xshift=-1.5em]de_ca.west) -- ([xshift=-0.1em]de_ca.west); \draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]de_ca.south)--([yshift=-0.6em,xshift=4.0em]de_ca.south)--([xshift=0.43em]de_add2.east)--(de_add2.east);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]de_ffn.south)--([yshift=-0.6em,xshift=4.0em]de_ffn.south)--([xshift=0.43em]de_add3.east)--(de_add3.east);
\draw[->,rounded corners=2pt,thick] ([yshift=0.1em]en_add2.90) -- ([yshift=1.5em]en_add2.90) -- ([xshift=5.0em,yshift=1.5em]en_add2.90) -- ([xshift=-1.5em]de_ca.west) -- ([xshift=-0.1em]de_ca.west);
\begin{pgfonlayer}{background} \begin{pgfonlayer}{background}
\node[draw=ugreen,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick][fit=(en_sa)(en_ffn)](box1){}; \node[draw=ugreen,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick,xshift=-0.2em,yshift=-0.2em][fit=(en_add1)(en_add2)(en_sa)(en_ffn)](box1){};
\node[draw=red,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick][fit=(de_sa)(de_ca)(de_ffn)](box2){}; \node[draw=red,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick,xshift=0.2em,yshift=-0.2em][fit=(de_sa)(de_ca)(de_ffn)(de_add3)](box2){};
\end{pgfonlayer} \end{pgfonlayer}
\node[anchor=east,font=\scriptsize,text=ugreen] at ([xshift=-0.1em]box1.west){$N \times$}; \node[anchor=east,font=\scriptsize,text=ugreen] at ([xshift=-0.1em]box1.west){$N \times$};
......
Chapter17/Figures/figure-speech-translation-model-based-on-CTC.tex
\begin{tikzpicture} \begin{tikzpicture}
\tikzstyle{layer}=[draw,rounded corners=2pt,font=\scriptsize,align=center,minimum width=5em] \tikzstyle{layer}=[draw,rounded corners=2pt,font=\scriptsize,align=center,minimum width=7.1em]
\tikzstyle{word}=[font=\scriptsize] \tikzstyle{word}=[font=\scriptsize]
%%%%encoder
\node[layer,fill=red!20] (en_sa) at (0,0){Multi-Head \\ Attention}; \node[layer,fill=red!20] (en_sa) at (0,0){Multi-Head \\ Attention};
\node[layer,anchor=south,fill=green!20] (en_ffn) at ([yshift=1.4em]en_sa.north){Feed Forward \\ Network}; \node[anchor=south,layer,fill=yellow!20](en_add1) at ([yshift=1.0em]en_sa.north) {Add \& LayerNorm};
\node[draw,circle,inner sep=0pt, minimum size=1em,anchor=north] (en_add) at ([yshift=-1.4em]en_sa.south){}; \node[layer,anchor=south,fill=green!20] (en_ffn) at ([yshift=1.0em]en_add1.north){Feed Forward \\ Network};
\draw[] (en_add.90) -- (en_add.-90); \node[anchor=south,layer,fill=yellow!20](en_add2) at ([yshift=1.0em]en_ffn.north) {Add \& LayerNorm};
\draw[] (en_add.0) -- (en_add.180); \node[layer,anchor=south,fill=blue!20] (en_sf) at ([yshift=2.4em]en_add2.north){Softmax};
\node[layer,anchor=north,fill=yellow!20] (en_cnn) at ([yshift=-1.4em]en_add.south){CNN}; \node[layer,anchor=south,fill=orange!20] (en_output) at ([yshift=1.0em]en_sf.north){CTC Output};
\node[draw,circle,inner sep=0pt, minimum size=1em,anchor=north,thick] (en_add) at ([yshift=-1.4em]en_sa.south){};
\draw[thick] (en_add.90) -- (en_add.-90);
\draw[thick] (en_add.0) -- (en_add.180);
\node[layer,anchor=north,fill=yellow!20] (en_cnn) at ([yshift=-1.0em]en_add.south){CNN};
\node[anchor=east,font=\scriptsize,align=center] (en_pos) at ([xshift=-2em]en_add.west){位置编码};
\node[anchor=north,font=\scriptsize,align=center] (en_input) at ([yshift=-1em]en_cnn.south){源语言语音特征\\(FBank/MFCC)};
\draw[->,thick] (en_input.90) -- ([yshift=-0.1em]en_cnn.-90);
\draw[->,thick] ([yshift=0.1em]en_cnn.90) -- ([yshift=-0.1em]en_add.-90);
\draw[->,thick] ([yshift=0.1em]en_add.90) -- ([yshift=-0.1em]en_sa.-90);
\draw[->,thick] ([yshift=0.1em]en_sa.90) -- ([yshift=-0.1em]en_add1.-90);
\draw[->,thick] ([yshift=0.1em]en_add1.90) -- ([yshift=-0.1em]en_ffn.-90);
\draw[->,thick] ([yshift=0.1em]en_ffn.90) --([yshift=-0.1em]en_add2.-90);
\draw[->,thick] ([yshift=0.1em]en_add2.90) -- ([yshift=-0.1em]en_sf.-90);
\draw[->,thick] ([yshift=0.1em]en_sf.90) -- ([yshift=-0.1em]en_output.-90);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]en_sa.south)--([yshift=-0.6em,xshift=-4.0em]en_sa.south)--([xshift=-0.43em]en_add1.west)--(en_add1.west);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]en_ffn.south)--([yshift=-0.6em,xshift=-4.0em]en_ffn.south)--([xshift=-0.43em]en_add2.west)--(en_add2.west);
\node[draw,circle,inner sep=0pt, minimum size=1em,anchor=west] (de_add) at ([xshift=7em]en_add.east){}; %%%%decoder
\draw[] (de_add.90) -- (de_add.-90); \node[draw,circle,inner sep=0pt, minimum size=1em,anchor=west,thick] (de_add) at ([xshift=9em]en_add.east){};
\draw[] (de_add.0) -- (de_add.180); \draw[thick] (de_add.90) -- (de_add.-90);
\draw[thick] (de_add.0) -- (de_add.180);
\node[layer,anchor=south,fill=red!20] (de_sa) at ([yshift=1.4em]de_add.north){Masked \\Multi-Head\\Attention}; \node[layer,anchor=south,fill=red!20] (de_sa) at ([yshift=1.4em]de_add.north){Masked \\Multi-Head\\Attention};
\node[layer,anchor=south,fill=red!20] (de_ca) at ([yshift=1.4em]de_sa.north){Multi-Head \\ Attention}; \node[anchor=south,layer,fill=yellow!20](de_add1) at ([yshift=1.0em]de_sa.north) {Add \& LayerNorm};
\node[layer,anchor=south,fill=green!20] (de_ffn) at ([yshift=1.4em]de_ca.north){Feed Forward \\ Network}; \node[layer,anchor=south,fill=red!20] (de_ca) at ([yshift=1.0em]de_add1.north){Multi-Head \\ Attention};
\node[anchor=south,layer,fill=yellow!20](de_add2) at ([yshift=1.0em]de_ca.north) {Add \& LayerNorm};
\node[layer,anchor=south,fill=green!20] (de_ffn) at ([yshift=1.0em]de_add2.north){Feed Forward \\ Network};
\node[anchor=south,layer,fill=yellow!20](de_add3) at ([yshift=1.0em]de_ffn.north) {Add \& LayerNorm};
\node[layer,anchor=south,fill=blue!20] (sf) at ([yshift=1.2em]de_add3.north){Softmax};
\node[anchor=north,font=\scriptsize,align=center] (de_input) at ([yshift=-1.1em]de_add.south){目标语言文本\\编码表示};
\node[layer,anchor=south,fill=blue!20] (en_sf) at ([yshift=3em]en_ffn.north){Softmax}; \node[anchor=west,font=\scriptsize,align=center] (de_pos) at ([xshift=2em]de_add.east){位置编码};
\node[layer,anchor=south,fill=blue!20] (sf) at ([yshift=2em]de_ffn.north){Softmax};
\node[layer,anchor=south,fill=orange!20] (en_output) at ([yshift=1.4em]en_sf.north){CTC Output};
%\node[layer,anchor=south,fill=orange!20] (output) at ([yshift=1.4em]sf.north){ST Output};
\node[anchor=north,font=\scriptsize,align=center] (en_input) at ([yshift=-1em]en_cnn.south){语音特征\\(FBank/MFCC)}; \draw[->,thick] (de_input.90) -- ([yshift=-0.1em]de_add.-90);
\node[anchor=north,font=\scriptsize,align=center] (de_input) at ([yshift=-1em]de_add.south){标注文本\\编码表示}; \draw[->,thick] ([yshift=0.1em]de_add.90) -- ([yshift=-0.1em]de_sa.-90);
\draw[->,thick] ([yshift=0.1em]de_sa.90) -- ([yshift=-0.1em]de_add1.-90);
\draw[->,thick] ([yshift=0.1em]de_add1.90) -- ([yshift=-0.1em]de_ca.-90);
\draw[->,thick] ([yshift=0.1em]de_ca.90) -- ([yshift=-0.1em]de_add2.-90);
\draw[->,thick] ([yshift=0.1em]de_add2.90) -- ([yshift=-0.1em]de_ffn.-90);
\draw[->,thick] ([yshift=0.1em]de_ffn.90) -- ([yshift=-0.1em]de_add3.-90);
\draw[->,thick] ([yshift=0.1em]de_add3.90) -- ([yshift=-0.1em]sf.-90);
\draw[->,thick] ([yshift=0.1em]sf.90) -- ([yshift=1.0em]sf.90);
\draw[->,thick] ([xshift=0.1em]en_pos.0) -- ([xshift=-0.1em]en_add.180);
\draw[->,thick] ([xshift=-0.1em]de_pos.180) -- ([xshift=0.1em]de_add.0);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]de_sa.south)--([yshift=-0.6em,xshift=4.0em]de_sa.south)--([xshift=0.43em]de_add1.east)--(de_add1.east);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]de_ca.south)--([yshift=-0.6em,xshift=4.0em]de_ca.south)--([xshift=0.43em]de_add2.east)--(de_add2.east);
\draw[->,rounded corners=2pt,thick] ([yshift=-0.6em]de_ffn.south)--([yshift=-0.6em,xshift=4.0em]de_ffn.south)--([xshift=0.43em]de_add3.east)--(de_add3.east);
\draw[->,rounded corners=2pt,thick] ([yshift=0.1em]en_add2.90) -- ([yshift=1.5em]en_add2.90) -- ([xshift=5.0em,yshift=1.5em]en_add2.90) -- ([xshift=-1.5em]de_ca.west) -- ([xshift=-0.1em]de_ca.west);
\node[anchor=east,font=\scriptsize,align=center] (en_pos) at ([xshift=-2em]en_add.west){位置编码};
\node[anchor=west,font=\scriptsize,align=center] (de_pos) at ([xshift=2em]de_add.east){位置编码};
\draw[->] (en_input.90) -- ([yshift=-0.1em]en_cnn.-90);
\draw[->] ([yshift=0.1em]en_cnn.90) -- ([yshift=-0.1em]en_add.-90);
\draw[->] ([yshift=0.1em]en_add.90) -- ([yshift=-0.1em]en_sa.-90);
\draw[->] ([yshift=0.1em]en_sa.90) -- ([yshift=-0.1em]en_ffn.-90);
\draw[->] (de_input.90) -- ([yshift=-0.1em]de_add.-90);
\draw[->] ([yshift=0.1em]de_add.90) -- ([yshift=-0.1em]de_sa.-90);
\draw[->] ([yshift=0.1em]de_sa.90) -- ([yshift=-0.1em]de_ca.-90);
\draw[->] ([yshift=0.1em]de_ca.90) -- ([yshift=-0.1em]de_ffn.-90);
\draw[->] ([yshift=0.1em]en_ffn.90) -- ([yshift=-0.1em]en_sf.-90);
\draw[->] ([yshift=0.1em]en_sf.90) -- ([yshift=-0.1em]en_output.-90);
\draw[->] ([yshift=0.1em]de_ffn.90) -- ([yshift=-0.1em]sf.-90);
\draw[->] ([yshift=0.1em]sf.90) -- ([yshift=1.5em]sf.90);
\draw[->] ([xshift=0.1em]en_pos.0) -- ([xshift=-0.1em]en_add.180);
\draw[->] ([xshift=-0.1em]de_pos.180) -- ([xshift=0.1em]de_add.0);
\draw[->,rounded corners=2pt] ([yshift=2em]en_ffn.90) -- ([xshift=4em,yshift=2em]en_ffn.90) -- ([xshift=-1.5em]de_ca.west) -- ([xshift=-0.1em]de_ca.west);
\begin{pgfonlayer}{background} \begin{pgfonlayer}{background}
\node[draw=ugreen,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick][fit=(en_sa)(en_ffn)]{}; \node[draw=ugreen,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick,xshift=-0.2em,yshift=-0.2em][fit=(en_add1)(en_add2)(en_sa)(en_ffn)](box1){};
\node[draw=red,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick][fit=(de_sa)(de_ca)(de_ffn)]{}; \node[draw=red,rounded corners=2pt,inner xsep=6pt,inner ysep=8pt,dashed,thick,xshift=0.2em,yshift=-0.2em][fit=(de_sa)(de_ca)(de_ffn)(de_add3)](box2){};
\end{pgfonlayer} \end{pgfonlayer}
\node[anchor=east,font=\scriptsize,text=ugreen] at ([xshift=-0.1em]box1.west){$N \times$}; \node[anchor=east,font=\scriptsize,text=ugreen] at ([xshift=-0.1em]box1.west){$N \times$};
\node[anchor=west,font=\scriptsize,text=red] at ([xshift=0.1em]box2.east){$\times N$}; \node[anchor=west,font=\scriptsize,text=red] at ([xshift=0.1em]box2.east){$\times N$};
\node[anchor=east,font=\scriptsize] at ([xshift=-0.1em]en_cnn.west){$2 \times$}; \node[anchor=east,font=\scriptsize] at ([xshift=-0.1em]en_cnn.west){$2 \times$};
\node[anchor=east,font=\scriptsize,align=center,text=ugreen] at ([xshift=-0.1em,yshift=3em]box1.west){ST\\ 编码器}; \node[anchor=east,font=\scriptsize,align=center,text=ugreen] at ([xshift=-0.1em,yshift=3em]box1.west){ST \\ 编码器};
\node[anchor=west,font=\scriptsize,align=center,text=red] at ([xshift=0.1em,yshift=5em]box2.east){ST\\解码器}; \node[anchor=west,font=\scriptsize,align=center,text=red] at ([xshift=0.1em,yshift=5em]box2.east){ST \\ 解码器};
\end{tikzpicture} \end{tikzpicture}
\ No newline at end of file
Chapter2/Figures/figure-word-frequency-distribution.tex
\definecolor{ublue}{rgb}{0.152,0.250,0.545} \definecolor{ublue}{rgb}{0.152,0.250,0.545}
\begin{tikzpicture} \begin{tikzpicture}
\begin{axis}[ \begin{axis}[
width=11cm, width=13cm,
height=5.5cm, height=5.5cm,
xlabel={WikiText-103上的词表}, xlabel={WikiText-103上的词表},
ylabel={词汇出现总次数}, ylabel={词汇出现总次数},
......
Chapter2/chapter2.tex
...@@ -862,9 +862,6 @@ c(\cdot) & \textrm{当计算最高阶模型时} \\ ...@@ -862,9 +862,6 @@ c(\cdot) & \textrm{当计算最高阶模型时} \\
\parinterval 当任务对单词序列长度没有限制时，上述两种方法枚举出的单词序列也是无穷无尽的。因此这两种枚举策略并不具备完备性而且会导致枚举过程无法停止。由于日常生活中通常不会见到特别长的句子，因此可以通过限制单词序列的最大长度来避免这个问题。一旦单词序列的最大长度被确定，以上两种枚举策略就可以在一定时间内枚举出所有可能的单词序列，因而一定可以找到最优的单词序列，即具备最优性。 \parinterval 当任务对单词序列长度没有限制时，上述两种方法枚举出的单词序列也是无穷无尽的。因此这两种枚举策略并不具备完备性而且会导致枚举过程无法停止。由于日常生活中通常不会见到特别长的句子，因此可以通过限制单词序列的最大长度来避免这个问题。一旦单词序列的最大长度被确定，以上两种枚举策略就可以在一定时间内枚举出所有可能的单词序列，因而一定可以找到最优的单词序列，即具备最优性。
\parinterval 此时上述生成策略虽然可以满足完备性和最优性，但其仍然算不上是优秀的生成策略，因为这两种算法在时间复杂度和空间复杂度上的表现很差，如表\ref{tab:2-4}所示。其中$|V|$为词表大小，$m$ 为序列长度。值得注意的是，在之前的遍历过程中，除了在序列开头一定会挑选<sos>之外，其他位置每次可挑选的单词并不只有词表中的单词，还有结束符号<eos>，因此实际上生成过程中每个位置的单词候选数量为$|V|+1$。
\vspace{0.5em}
%------------------------------------------------------ %------------------------------------------------------
\begin{table}[htp]{ \begin{table}[htp]{
\begin{center} \begin{center}
...@@ -881,6 +878,8 @@ c(\cdot) & \textrm{当计算最高阶模型时} \\ ...@@ -881,6 +878,8 @@ c(\cdot) & \textrm{当计算最高阶模型时} \\
}\end{table} }\end{table}
%------------------------------------------------------ %------------------------------------------------------
\parinterval 此时上述生成策略虽然可以满足完备性和最优性，但其仍然算不上是优秀的生成策略，因为这两种算法在时间复杂度和空间复杂度上的表现很差，如表\ref{tab:2-4}所示。其中$|V|$为词表大小，$m$ 为序列长度。值得注意的是，在之前的遍历过程中，除了在序列开头一定会挑选<sos>之外，其他位置每次可挑选的单词并不只有词表中的单词，还有结束符号<eos>，因此实际上生成过程中每个位置的单词候选数量为$|V|+1$。
\parinterval 那么是否有比枚举策略更高效的方法呢？答案是肯定的。一种直观的方法是将搜索的过程表示成树型结构，称为解空间树。它包含了搜索过程中可生成的全部序列。该树的根节点恒为<sos>，代表序列均从<sos> 开始。该树结构中非叶子节点的兄弟节点有$|V|+1$个，由词表和结束符号<eos>构成。从图\ref{fig:2-13}可以看到，对于一个最大长度为4的序列的搜索过程，生成某个单词序列的过程实际上就是访问解空间树中从根节点<sos> 开始一直到叶子节点<eos>结束的某条路径，而这条的路径上节点按顺序组成了一段独特的单词序列。此时对所有可能单词序列的枚举就变成了对解空间树的遍历。并且枚举的过程与语言模型打分的过程也是一致的，每枚举一个词$i$也就是在图\ref{fig:2-13}选择$w_i$一列的一个节点，语言模型就可以为当前的树节点$w_i$给出一个分值，即$\funp{P}(w_i | w_1 w_2 \ldots w_{i-1})$。对于$n$-gram语言模型，这个分值可以表示为$\funp{P}(w_i | w_1 w_2 \ldots w_{i-1})=\funp{P}(w_i | w_{i-n+1} \ldots w_{i-1})$ \parinterval 那么是否有比枚举策略更高效的方法呢？答案是肯定的。一种直观的方法是将搜索的过程表示成树型结构，称为解空间树。它包含了搜索过程中可生成的全部序列。该树的根节点恒为<sos>，代表序列均从<sos> 开始。该树结构中非叶子节点的兄弟节点有$|V|+1$个，由词表和结束符号<eos>构成。从图\ref{fig:2-13}可以看到，对于一个最大长度为4的序列的搜索过程，生成某个单词序列的过程实际上就是访问解空间树中从根节点<sos> 开始一直到叶子节点<eos>结束的某条路径，而这条的路径上节点按顺序组成了一段独特的单词序列。此时对所有可能单词序列的枚举就变成了对解空间树的遍历。并且枚举的过程与语言模型打分的过程也是一致的，每枚举一个词$i$也就是在图\ref{fig:2-13}选择$w_i$一列的一个节点，语言模型就可以为当前的树节点$w_i$给出一个分值，即$\funp{P}(w_i | w_1 w_2 \ldots w_{i-1})$。对于$n$-gram语言模型，这个分值可以表示为$\funp{P}(w_i | w_1 w_2 \ldots w_{i-1})=\funp{P}(w_i | w_{i-n+1} \ldots w_{i-1})$
%---------------------------------------------- %----------------------------------------------
......
Chapter8/Figures/figure-combination-of-translation-with-different-rules.tex
...@@ -5,7 +5,7 @@ ...@@ -5,7 +5,7 @@
\begin{scope}%[scale=0.2] \begin{scope}%[scale=0.2]
\node[anchor=north] (q1) at (0,0) {\scriptsize\sffamily\bfseries{输入字符串：}}; \node[anchor=north] (q1) at (0,0) {\scriptsize\sffamily\bfseries{输入字符串：}};
\node[anchor=west] (q2) at ([xshift=0em,yshift=-2em]q1.west) {\footnotesize{进口$\quad$和$\quad$出口$\quad$大幅度$\quad$下降$\quad$了}}; \node[anchor=west] (q2) at ([xshift=0em,yshift=-1.5em]q1.west) {\footnotesize{进口$\quad$和$\quad$出口$\quad$大幅度$\quad$下降$\quad$了}};
\node[anchor=north,fill=blue!20,minimum height=4em,minimum width=1em] (f1) at ([xshift=2.7em,yshift=-0.7em]q2.south) {}; \node[anchor=north,fill=blue!20,minimum height=4em,minimum width=1em] (f1) at ([xshift=2.7em,yshift=-0.7em]q2.south) {};
......
Chapter8/Figures/figure-hierarchical-phrase-rule-match-generate.tex
...@@ -5,7 +5,7 @@ ...@@ -5,7 +5,7 @@
\begin{scope}%[scale=0.2] \begin{scope}%[scale=0.2]
\node[anchor=north] (q1) at (0,0) {\scriptsize\sffamily\bfseries{输入字符串：}}; \node[anchor=north] (q1) at (0,0) {\scriptsize\sffamily\bfseries{输入字符串：}};
\node[anchor=west] (q2) at ([xshift=0em,yshift=-2em]q1.west) {\footnotesize{进口$\quad$和$\quad$出口$\quad$大幅度$\quad$下降$\quad$了}}; \node[anchor=west] (q2) at ([xshift=0em,yshift=-1.5em]q1.west) {\footnotesize{进口$\quad$和$\quad$出口$\quad$大幅度$\quad$下降$\quad$了}};
\node[anchor=north,fill=blue!20,minimum height=1em,minimum width=1em] (f1) at ([xshift=-4.2em,yshift=-0.8em]q2.south) {}; \node[anchor=north,fill=blue!20,minimum height=1em,minimum width=1em] (f1) at ([xshift=-4.2em,yshift=-0.8em]q2.south) {};
......
Chapter8/Figures/figure-one-best-node-alignment-and-alignment-matrix.tex
...@@ -107,7 +107,7 @@ ...@@ -107,7 +107,7 @@
\vspace{-1em} \vspace{-1em}
\footnotesize{(a)节点对齐矩阵（1-best vs Matrix）} \footnotesize{(a)节点对齐矩阵（1-best vs Matrix）}
\end{center} \end{center}
\vspace{-3em}
\begin{center} \begin{center}
\begin{tabular}[t]{C{0.48\linewidth} C{0.48\linewidth} } \begin{tabular}[t]{C{0.48\linewidth} C{0.48\linewidth} }
......
Markdown 格式
0% 或
您添加了 0 人 到此讨论。请谨慎行事。
请先完成此评论的编辑！
请 注册 或者 后发表评论