【深度学习】序列生成模型（六）：评价方法计算实例：计算ROUGE-N得分【理论到程序】

一、BLEU-N得分（Bilingual Evaluation Understudy）

【深度学习】序列生成模型（五）：评价方法计算实例：计算BLEU-N得分

二、ROUGE-N得分（Recall-Oriented Understudy for Gisting Evaluation）

1. 定义

设

\mathbf{x}

为从模型分布

p_{\theta}

中生成的一个候选序列，

\mathbf{s^{(1)}}, ⋯ , \mathbf{s^{(K)}}

为从真实数据分布中采样得到的一组参考序列，

\mathcal{W}

为从参考序列中提取N元组合的集合，ROUGE-N算法的定义为：

\text{ROUGE-N}(\mathbf{x}) = \frac{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)}))}{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k))}}

其中

c_w(\mathbf{x})

是N元组合

在生成序列

\mathbf{x}

中出现的次数，

c_w(\mathbf{s}^{(k))})

是N元组合

在参考序列

\mathbf{s}^{(k)}

中出现的次数。

2. 计算

N=1

生成序列

\mathbf{x}=\text{the cat sat on the mat}

参考序列
\mathbf{s}^{(1)}=\text{the cat is on the mat}
\mathbf{s}^{(2)}=\text{the bird sat on the bush}

\mathcal{W}=\text{ {the, cat, is, on, mat, bird, sat, bush }}

       w 
      
     
   w</p></div></div></th><th style="text-align:left"><div><div class="table-header"><p>c 
        
       
         w 
        
       
      
        ( 
       
      
        x 
       
      
        ) 
       
      
     
       c_w(\mathbf{x}) 
      
     
   cw(x)</p></div></div></th><th style="text-align:left"><div><div class="table-header"><p>c 
        
       
         w 
        
       
      
        ( 
       
       
       
         s 
        
        
        
          ( 
         
        
          1 
         
        
          ) 
         
        
       
      
        ) 
       
      
     
       c_w(\mathbf{s^{(1)}}) 
      
     
   cw(s(1))</p></div></div></th><th style="text-align:left"><div><div class="table-header"><p>c 
        
       
         w 
        
       
      
        ( 
       
       
       
         s 
        
        
        
          ( 
         
        
          2 
         
        
          ) 
         
        
       
      
        ) 
       
      
     
       c_w(\mathbf{s^{(2)}}) 
      
     
   cw(s(2))</p></div></div></th><th style="text-align:left"><div><div class="table-header"><p>min 
       
      
        ⁡ 
       
      
        ( 
       
       
       
         c 
        
       
         w 
        
       
      
        ( 
       
      
        x 
       
      
        ) 
       
      
        , 
       
       
       
         c 
        
       
         w 
        
       
      
        ( 
       
       
       
         s 
        
        
        
          ( 
         
        
          1 
         
        
          ) 
         
        
       
      
        ) 
       
      
     
       \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(1)}) 
      
     
   min(cw(x),cw(s(1))</p></div></div></th><th style="text-align:left"><div><div class="table-header"><p>min 
       
      
        ⁡ 
       
      
        ( 
       
       
       
         c 
        
       
         w 
        
       
      
        ( 
       
      
        x 
       
      
        ) 
       
      
        , 
       
       
       
         c 
        
       
         w 
        
       
      
        ( 
       
       
       
         s 
        
        
        
          ( 
         
        
          2 
         
        
          ) 
         
        
       
      
        ) 
       
      
     
       \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(2)}) 
      
     
   min(cw(x),cw(s(2))</p></div></div></th></tr></thead><tbody><tr><td style="text-align:left"><div><div class="table-cell"><p>the</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>2</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>2</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>2</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>2</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>2</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>cat</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>is</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>on</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>mat</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>bird</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>sat</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>bush</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr></tbody></table></div><figure class=""><span>w</span></figure><figure class=""><span>c_w(\mathbf{x})</span></figure><figure class=""><span>c_w(\mathbf{s^{(1)}})</span></figure><figure class=""><span>c_w(\mathbf{s^{(2)}})</span></figure><figure class=""><span>\min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(1)})</span></figure><figure class=""><span>\min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(2)})</span></figure><p>the22222cat11010is01000on11111mat11010bird00100sat10101bush00100</p><ul class="ul-level-0"><li>分子</li></ul><figure class=""><span>\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)}))</span></figure><ul class="ul-level-0"><li>分母</li></ul><figure class=""><span>\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k)})</span></figure><ul class="ul-level-0"><li></li></ul><figure class=""><span>\text{ROUGE-N}(\mathbf{x}) = \frac{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)}))}{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k))}}=\frac{5+4}{6+6}=\frac{9}{12}=0.75</span></figure><h5 id="a9sia" name="N=2">N=2</h5><ul class="ul-level-0"><li><strong>生成序列</strong> </li></ul><figure class=""><span>\mathbf{x}=\text{the cat sat on the mat}</span></figure><ul class="ul-level-0"><li><strong>参考序列</strong> <ul class="ul-level-1"><li></li></ul><figure class=""><span>\mathbf{s}^{(1)}=\text{the cat is on the mat}</span></figure><ul class="ul-level-1"><li></li></ul><figure class=""><span>\mathbf{s}^{(2)}=\text{the bird sat on the bush}</span></figure></li><li></li></ul><figure class=""><span>\mathcal{W}=\text{ {the cat, cat is, is on, on the, the mat, the bird, bird sat, sat on, the bush }}</span></figure><div class="table-wrapper"><table><thead><tr><th style="text-align:left"><div><div class="table-header"><p>w 
       
      
     
       w 
      
     
   w</p></div></div></th><th style="text-align:left"><div><div class="table-header"><p>c 
        
       
         w 
        
       
      
        ( 
       
      
        x 
       
      
        ) 
       
      
     
       c_w(\mathbf{x}) 
      
     
   cw(x)</p></div></div></th><th style="text-align:left"><div><div class="table-header"><p>c 
        
       
         w 
        
       
      
        ( 
       
       
       
         s 
        
        
        
          ( 
         
        
          1 
         
        
          ) 
         
        
       
      
        ) 
       
      
     
       c_w(\mathbf{s^{(1)}}) 
      
     
   cw(s(1))</p></div></div></th><th style="text-align:left"><div><div class="table-header"><p>c 
        
       
         w 
        
       
      
        ( 
       
       
       
         s 
        
        
        
          ( 
         
        
          2 
         
        
          ) 
         
        
       
      
        ) 
       
      
     
       c_w(\mathbf{s^{(2)}}) 
      
     
   cw(s(2))</p></div></div></th><th style="text-align:left"><div><div class="table-header"><p>min 
       
      
        ⁡ 
       
      
        ( 
       
       
       
         c 
        
       
         w 
        
       
      
        ( 
       
      
        x 
       
      
        ) 
       
      
        , 
       
       
       
         c 
        
       
         w 
        
       
      
        ( 
       
       
       
         s 
        
        
        
          ( 
         
        
          1 
         
        
          ) 
         
        
       
      
        ) 
       
      
     
       \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(1)}) 
      
     
   min(cw(x),cw(s(1))</p></div></div></th><th style="text-align:left"><div><div class="table-header"><p>min 
       
      
        ⁡ 
       
      
        ( 
       
       
       
         c 
        
       
         w 
        
       
      
        ( 
       
      
        x 
       
      
        ) 
       
      
        , 
       
       
       
         c 
        
       
         w 
        
       
      
        ( 
       
       
       
         s 
        
        
        
          ( 
         
        
          2 
         
        
          ) 
         
        
       
      
        ) 
       
      
     
       \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(2)}) 
      
     
   min(cw(x),cw(s(2))</p></div></div></th></tr></thead><tbody><tr><td style="text-align:left"><div><div class="table-cell"><p>the cat</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>cat is</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>is on</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>on the</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>the mat</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>the bird</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>bird sat</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>sat on</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td></tr><tr><td style="text-align:left"><div><div class="table-cell"><p>the bush</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>1</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td><td style="text-align:left"><div><div class="table-cell"><p>0</p></div></div></td></tr></tbody></table></div><figure class=""><span>w</span></figure><figure class=""><span>c_w(\mathbf{x})</span></figure><figure class=""><span>c_w(\mathbf{s^{(1)}})</span></figure><figure class=""><span>c_w(\mathbf{s^{(2)}})</span></figure><figure class=""><span>\min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(1)})</span></figure><figure class=""><span>\min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(2)})</span></figure><p>the cat11010cat is01000is on01000on the11111the mat11000the bird00100bird sat00100sat on10111the bush00100</p><ul class="ul-level-0"><li>分子</li></ul><figure class=""><span>\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)}))</span></figure><ul class="ul-level-0"><li>分母</li></ul><figure class=""><span>\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k)})</span></figure><ul class="ul-level-0"><li></li></ul><figure class=""><span>\text{ROUGE-N}(\mathbf{x}) = \frac{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} \min(c_w(\mathbf{x}), c_w(\mathbf{s}^{(k)}))}{\sum_{k=1}^{K} \sum_{w \in \mathcal{W}} c_w(\mathbf{s}^{(k))}}=\frac{3+2}{5+5}=\frac{5}{10}=0.5</span></figure><h4 id="ajl7m" name="3.-%E7%A8%8B%E5%BA%8F">3. 程序</h4><div class="rno-markdown-code"><div class="rno-markdown-code-toolbar"><div class="rno-markdown-code-toolbar-info"><div class="rno-markdown-code-toolbar-item is-type"><span class="is-m-hidden">代码语言：</span>javascript</div></div><div class="rno-markdown-code-toolbar-opt"><div class="rno-markdown-code-toolbar-copy"><i class="icon-copy"></i><span class="is-m-hidden">复制</span></div></div></div><div class="developer-code-block"><pre class="prism-token token line-numbers language-javascript"><code class="language-javascript" style="margin-left:0">main_string = &#39;the cat sat on the mat&#39;

string1 = 'the cat is on the mat'
string2 = 'the bird sat on the bush'

words = list(set(string1.split(' ')+string2.split(' '))) # 去除重复元素

total_occurrences, matching_occurrences = 0, 0
for word in words:
matching_occurrences += min(main_string.count(word), string1.count(word)) + min(main_string.count(word), string2.count(word))
total_occurrences += string1.count(word) + string2.count(word)

print(matching_occurrences / total_occurrences)

bigrams = []
split1 = string1.split(' ')
for i in range(len(split1) - 1):
bigrams.append(split1[i] + ' ' + split1[i + 1])

split2 = string2.split(' ')
for i in range(len(split2) - 1):
bigrams.append(split2[i] + ' ' + split2[i + 1])

bigrams = list(set(bigrams)) # 去除重复元素

total_occurrences, matching_occurrences = 0, 0
for bigram in bigrams:
matching_occurrences += min(main_string.count(bigram), string1.count(bigram)) + min(main_string.count(bigram), string2.count(bigram))
total_occurrences += string1.count(bigram) + string2.count(bigram)

print(matching_occurrences / total_occurrences)

输出：

代码语言：javascript

复制

0.75

0.5