Hoes of Techhttps://www.horsal.dev/2022-07-10T00:00:00+09:00Conceptualization of Technology日本国籍笔记2022-07-10T00:00:00+09:002022-07-10T00:00:00+09:00Horsaltag:www.horsal.dev,2022-07-10:/ri-ben-guo-ji-bi-ji.html<h2>Stage0 - 条件</h2>
<p>申请之前最好保证自己条件没问题,否则预约之前还没进门就被拒了。一般的条件都能在<a href="https://www.moj.go.jp/MINJI/minji78.html">法务省的网站上找到</a>,
其中值得一提的是居住条件(国籍法第5条第1項第1号),虽然法律只要求了连续居住5年,但在实际操作上都是要求要有至少3年的工作,大部分法务局都会明确提到这一点。
不过这里的3年工作时间只需要在正式申请的时间点达成即可,考虑到正式申请之前几个月的准备时间,工作2年9个月也并不会被拒绝。</p>
<p>实际上,除了网上能够看到的一般的条件,实际的审查有详细的内部参考资料(帰化事件処理要領),里面规定了更详细的条件以及审查要点。
这份文件通常不会公开,通过律师提出公开请求也最多只能活得一个<a href="https://yamanaka-bengoshi.jp/wp-content/uploads/2020/02/240322-帰化事件処理要領(本文・黒塗り多し).pdf">关键部分均被涂黑的版本</a>,不过记载的内容依然比网上的公开内容详细了很多,建议在申请之前通读一遍。</p>
<h2>Stage1 - 第一次面谈(预审)</h2>
<p>跟当地的法务局打电话,每个法务局都只管自己附近的地区,你住在北海道给东京法务局打电话人家也不管你,具体要联系哪里可以参考<a href="https://www.moj.go.jp/MINJI/kokuseki.html">法务省的网站</a>。在电话里说明自己是第一次去,会有人安排面谈时间,并让你带着自己的在留卡和护照去法务局。由于大部分法务局都挺忙,一般能预约的时间都是一个月后。</p>
<p>预约之后法务局会安排一个帮你准备材料的“相談員”,在正式申请之前都是他负责,需要注意的是这个阶段里负责人可以拒绝接受申请。
第一次到法务局之后会安排一次杂谈时间,听取申请人的家庭状况,工作状况等,之后会确定申请人的性质并给出一份需要的材料列表。</p>
<p>很多法务局都先要求先办理在中国国内的材料,在这些材料没问题之后才要求办理日本方面的材料。
根据家庭构成需要的材料各有不同,但大部分都可以在户籍地的公证处办理,一些公证处还提供翻译服务 …</p><h2>Stage0 - 条件</h2>
<p>申请之前最好保证自己条件没问题,否则预约之前还没进门就被拒了。一般的条件都能在<a href="https://www.moj.go.jp/MINJI/minji78.html">法务省的网站上找到</a>,
其中值得一提的是居住条件(国籍法第5条第1項第1号),虽然法律只要求了连续居住5年,但在实际操作上都是要求要有至少3年的工作,大部分法务局都会明确提到这一点。
不过这里的3年工作时间只需要在正式申请的时间点达成即可,考虑到正式申请之前几个月的准备时间,工作2年9个月也并不会被拒绝。</p>
<p>实际上,除了网上能够看到的一般的条件,实际的审查有详细的内部参考资料(帰化事件処理要領),里面规定了更详细的条件以及审查要点。
这份文件通常不会公开,通过律师提出公开请求也最多只能活得一个<a href="https://yamanaka-bengoshi.jp/wp-content/uploads/2020/02/240322-帰化事件処理要領(本文・黒塗り多し).pdf">关键部分均被涂黑的版本</a>,不过记载的内容依然比网上的公开内容详细了很多,建议在申请之前通读一遍。</p>
<h2>Stage1 - 第一次面谈(预审)</h2>
<p>跟当地的法务局打电话,每个法务局都只管自己附近的地区,你住在北海道给东京法务局打电话人家也不管你,具体要联系哪里可以参考<a href="https://www.moj.go.jp/MINJI/kokuseki.html">法务省的网站</a>。在电话里说明自己是第一次去,会有人安排面谈时间,并让你带着自己的在留卡和护照去法务局。由于大部分法务局都挺忙,一般能预约的时间都是一个月后。</p>
<p>预约之后法务局会安排一个帮你准备材料的“相談員”,在正式申请之前都是他负责,需要注意的是这个阶段里负责人可以拒绝接受申请。
第一次到法务局之后会安排一次杂谈时间,听取申请人的家庭状况,工作状况等,之后会确定申请人的性质并给出一份需要的材料列表。</p>
<p>很多法务局都先要求先办理在中国国内的材料,在这些材料没问题之后才要求办理日本方面的材料。
根据家庭构成需要的材料各有不同,但大部分都可以在户籍地的公证处办理,一些公证处还提供翻译服务。
由于国内办的这些材料会用来填写日本的户籍,人名都要对应成日本的正字,对自己的转写没有把握的话建议用翻译服务。
遇到信息没联网不能办理的情况,可以用现有证件复印+公证的形式来绕过,效果是一样的。
(比如结婚比较早的,由于信息没联网不能直接出示公证书,但可以复印结婚证,然后公证这个复印件是真的)</p>
<p>另外,预审跟下面的几次面谈可以同时进行,如果在第一次面谈的时候就准备好所有资料的话,理论上一次面谈就可以完成Stage1-Stage3的所有步骤进入正式申请。
通常能够节省3个月的时间。</p>
<h2>Stage2 - 第二次面谈</h2>
<p>跟上次一样预约之后,带着第一次材料去法务局。担当者会确认材料,没什么问题就能获得之后需要的材料列表了。</p>
<h3>自动丧失中国国籍的领事证明</h3>
<p>有些法务局正式提交时可以没有这个证明,但不提交不会进入正式审查,还是建议一起办好。
办理不需要预约,去分管住所的领事馆或者大使馆即可办理。</p>
<ul>
<li>事先填写<a href="http://jp.chineseembassy.org/chn/lszc/gzjrz/t1890635.htm">《领事证明申请表》</a>,其中领事证明内容填写“办理自动丧失中国国籍的领事证明”即可,出生地只需要填写到省一级。</li>
<li>在留卡和护照都要事先复印,领事馆或大使馆通常也提供收费的复印机。</li>
<li>在最初的窗口说明来意之后会获得一张发送人是大使馆(顺丰)的邮寄单,填好之后到正式办理服务的窗口。</li>
<li>提交材料之后会同时收下护照,几个工作日之后会跟领事证明同时邮寄到家里。办理时不收费,但邮寄时会以到付的形式收费。</li>
</ul>
<p>办理领事证明之后中国护照也不会失效,在正式申请通过后需要再次到大使馆注销护照。</p>
<h3>各种证明</h3>
<ul>
<li>居住地发行的住民税的最新课税证明,以及没有欠税的纳税证明,可以去役所或者自助服务机上打印。普通员工通常是<strong>上一个年度</strong>的纳税证明以及<strong>当前年度</strong>的课税证明。</li>
<li>年金证明(年金定期便)每年都会邮寄一张明信片,上面有年金记录,如果弄丢了也可以<a href="https://www.nenkin.go.jp/index.html">从网上打印</a>。注意如果没有注册过这个网站,还没有マイナンバーカード的话只能通过申请明信片的方式注册,最短也要一个星期。</li>
<li>如果不记得自己的出入国历史,可以<a href="https://www.moj.go.jp/isa/applications/disclosure/record.html">申请获得出入境的记录</a>。</li>
</ul>
<h3>其他材料</h3>
<ul>
<li>各种申请书的申请人名字需要是正字,而非在留卡上的名字。因此就算在留卡上是简体字,在填写申请书时也应该自己修改成日本的常用以及人名用汉字。因为这个名字是用来制作户籍的,需要符合户籍文字要求,父母的名字也同样(户籍上会注明直系亲属的名字)。</li>
<li>入籍之后的名字必须是日本字,即常用汉字,命名用汉字,平假名,片假名,除此之外都不能用,英文(即罗马字)也不能用,但未来可以在护照上标注名字的汉语拼音。名字可以随便取,不论是日本风格还是纯假名都没有问题,但进入户籍之后就不能再改(想改名字只能打官司),孩子也会继承姓氏。当然如果未来有结婚的想法,可以通过进入对方户籍的方法修改自己的姓氏。另外修改名字会导致目前的印章失效,需要重新制作登录。</li>
<li>本籍所在地可以自由设定,如果愿意甚至可以设置成皇宫或者名胜古迹。通常可以填写自己住址的前半部分(即○丁目◯番◯),具体填写方法由本籍地决定,正式申请之后审查官会帮你修改,不需要太过在意。</li>
<li>关于住民票,如果5年内搬过家,还要办理5年内住过的所有地方的住民除票,就是转出之后的老住民票,上面记载了转出到了哪里。一般可以通过邮寄申请,申请需要的费用在邮局购买同样金额的定額小為替,跟申请单一起寄出。</li>
<li>需要填写一份理由书,写明自己申请入籍的理由,理由书必须手写。如果不知道写什么可以在网上搜索,能够找到中介提供的样本,但内容都很空洞,不建议直接照抄。填写时显然只能用日语,<a href="https://www.bunka.go.jp/kokugo_nihongo/sisaku/joho/joho/kijun/naikaku/pdf/joyokanjihyo_20101130.pdf">但楷体或者明朝体皆可</a>。另外就是建议写之前练一练字尽量写的好看点,否则写的实在不标准搞不好会被怀疑语言水平。</li>
<li>照片的大小是5cm*5cm,比较奇特。自动照相机没法打印的话可以取得电子版然后在附近的便利店打印。有照片打印功能的便利店大都支持这个大小的照片。</li>
</ul>
<h2>Stage3 - 正式申请</h2>
<p>有了上面的所有材料就可以预约正式申请了,正式申请要求预约连续的两个时间段(3个小时),因此预约的难度是最高,一般都排在一个月之后。
当天带着所有的材料去提交申请,在申请之前会确认材料是否齐全。如果材料有问题可以当场重新填写,税务和住民票等材料可以后来尽快邮寄。除此之外还会现场签署两份宣誓书。</p>
<p>在正式申请之后,法务局会分配一个负责审查的人员,并给你一张回执单。如果有问题或者补交材料,都可以直接联系回执单上写的地址。之后审查官会打电话要求进行面谈,在那之前只需要等着就行了。</p>
<ul>
<li>如果期间内发生了特殊情况,比如出国,结婚,搬家等,要尽快联系审查官。</li>
<li>在申请被正式承认之前,能够修改自己入籍之后的名字,联系审查官即可。</li>
<li>申请虽然没有标准处理时间,但内参文件要求对于每个申请制定处理计划并尽快处理。</li>
</ul>
<h2>Stage4 - 面谈审查</h2>
<p>大部分情况下都是3个月之后会接到审查官的电话,审查官可能会要求一些补充材料,准备好就可以去法务局面谈了。
有小道消息说高等人才的审查会快一些,实际上并没有影响,审查花费的时间主要取决于审查官有多忙。
面谈方式跟正式申请之前没有区别,依然是在单间里跟审查官一对一交谈。在证实开始之前审查官会说明之后的流程以及说实话的重要性,然后对之间交上去的材料进行确认。
除了交上去的材料,审查官手里还有从各个部门收集的资料,包括并不限于出入境的详细信息,年金办理时间点等等,因此不管被问到什么都不要试图掩盖。
如果申请材料没有问题,审查官会制作一份户籍案(帰化者の身分証明書案),反过来如果审查官认为申请人不合适,则会建议放弃申请。</p>
<p>面谈之后,审查官会将材料整理并进行支局的内部承认,之后送往法务省进行进一步的调查,调查完毕后会最终送往法务大臣处承认生效,这个过程需要数个月。</p>
<h2>Stage5 - 查询结果,国籍注销和办理户籍</h2>
<p>在正式公布结果之前的一周左右,会接到法务局的电话,确认审查时间内没有重大变故,以及未来一段时间不会出国。
接到电话后一般在一两周之后结果就会公布在<a href="https://kanpou.npb.go.jp">官報的告示部分</a>里,日本国籍也在登报的当天生效,几天后法务局会打来通知电话,并要求办理接下来的手续。</p>
<ul>
<li>官报每个工作日的上午8:30都会更新,如果想第一时间知道结果,可以在接到法务局的确认电话之后关注每天的官报。</li>
<li>官报从<a href="https://kanpou-ad.com/index6.html">申请登报到实际见报至少需要7天</a>,就算审查官当天上报,考虑到制作名单的时间延迟,公布最快也要到下一周。</li>
</ul>
<p>在获得身份证明之前,审查官会要求到当地中国大使馆办理注销护照(国籍)的手续,办理不需要预约,去所在区域大使馆办理即可。</p>
<ul>
<li>带上法务局提供的领事证明复印件(上面有入籍的标识),中国护照,以及二者的复印件各一份,到窗口直接办理。</li>
<li>办理分类为公证,办理时护照上会获得“持照人已申请退籍”的标识,之后需要将盖章之后的护照复印件交给大使馆工作人员。</li>
<li>上述手续结束后,工作人员会在领事证明上加盖“已办理中国护照注销手续”的章,这个证明可以复印或扫描存底用来作为申请中国签证的材料。</li>
</ul>
<p>把上面领事证明交给法务局之后,会获得「帰化者の身分証明書」,上面记载有户籍的基本信息。</p>
<ul>
<li>将证明书的<strong>复印件</strong>以及在留カード<a href="https://www.moj.go.jp/isa/applications/procedures/nyuukokukanri10_00020.html">邮寄给出入国在留管理局</a>或者到附近的出入国在留管理局直接办理。邮寄的在留卡并不会打孔返还,想留作纪念的话最好现场办理。</li>
<li>填写<a href="https://www.city.kasukabe.lg.jp/material/files/group/20/kika.pdf">帰化届</a>,跟证明书的原本一起交给当地役所办理户籍。</li>
<li>办理户籍中如果手持マイナンバーカード,那么卡上会加注「戸籍届出」,有效期也会延长至办理户籍日算起的10年之后。</li>
<li>办理户籍大概一周之后能够获得「戸籍謄本」,可以拿来办理护照,办理护照时可以同时要求用汉语拼音进行名字标注(如果保留了中文名的话)。</li>
<li>如果还有中国户籍的话,在下一次到中国时需要自行到<strong>户籍所在地</strong>的派出所注销户籍。</li>
</ul>
<p>到此为止国际相关的手续就结束了,如果需要出国可以办理护照。</p>
<ul>
<li>填写<a href="https://www.mofa.go.jp/mofaj/toko/passport/download/top.html">护照申请表</a>并打印,除此之外还要准备一张45mm * 35mm的照片。</li>
<li>按照<a href="https://www.mofa.go.jp/mofaj/toko/passport/pass_2.html">要求准备证明文件</a>,如果需要保留入籍之前的姓名英文拼写(比如汉语拼音),还需要携带已经失效的中国护照。</li>
<li>到居住地的护照办理中心提交材料,会获得一张护照的回执,一般7天之后就能获得护照了。</li>
</ul>Ergonomic Keyboard in 20222022-05-25T00:00:00+09:002022-05-25T00:00:00+09:00Horsaltag:www.horsal.dev,2022-05-25:/ergonomic-keyboard-in-2022.html<h2>Introduction</h2>
<p>这几天,伴随了我6年的<a href="https://www.amazon.co.jp/Truly-Ergonomic-Mechanical-Keyboard-Tactile/dp/B01KUQJHH0">Truly Ergonomic Keyboard</a>终于撑不下去了,
除了从购买当初就一直存在的Double Typing问题,最近还开始出现Ghost Touch——并且偏偏是Del键。不得不承认,
在无所事事的时候毫无征兆的跳出几十个删除确认窗口的恐怖效果比电影和游戏要强多了。
在那之后,不管是对键盘进行全面清洗,还是折腾微动,都没能解决Del键的Ghost Touch,这也让我下定决心换一款新的键盘。</p>
<p><img alt="已经停产的Truly Ergonomic" src="https://www.horsal.dev/images/2022-05-25-truly-ergonomic.jpg"></p>
<p>可惜的是最初的Truly Ergonomic Keyboard早就已经停产了,而沉寂了数年之后这家公司才推出了后继的<a href="https://trulyergonomic.com/ergonomic-keyboards/">Truly Ergonomic CLEAVE</a>键盘。</p>
<p><img alt="新版的Truly Ergonomic键盘" src="https://www.horsal.dev/images/2022-05-25-Truly-Ergonomic-CLEAVE.jpg"></p>
<p>新的键盘维持了老键盘的设计,依然是近些年人体工程学所追求的垂直键位排列,以及左右手能成自然状态的倾斜布局,有Function键看起来也很诱人。在快要下单的时候,我发现它似乎不是完全可编程的——说明里只表示Shift,ESC等极少数键位可以修改,官方也并没有提供刷固件的工具。然而作为一个Dvorak使用者,这些年我一直在用自己修改过的layout,回到QWERT显然已经不可能了——因此我决定寻找下一个能用的人体工程学键盘。</p>
<h2>Ergonomic Keyboard in 2022</h2>
<p>人体工程学并不是个什么新的话题,我还记得大概在20年前的大众软件上就出现过微软出品的人体工程学键盘,
在当时,中间凸起向两侧扭曲的键盘是一个不折不扣的新东西。其中 …</p><h2>Introduction</h2>
<p>这几天,伴随了我6年的<a href="https://www.amazon.co.jp/Truly-Ergonomic-Mechanical-Keyboard-Tactile/dp/B01KUQJHH0">Truly Ergonomic Keyboard</a>终于撑不下去了,
除了从购买当初就一直存在的Double Typing问题,最近还开始出现Ghost Touch——并且偏偏是Del键。不得不承认,
在无所事事的时候毫无征兆的跳出几十个删除确认窗口的恐怖效果比电影和游戏要强多了。
在那之后,不管是对键盘进行全面清洗,还是折腾微动,都没能解决Del键的Ghost Touch,这也让我下定决心换一款新的键盘。</p>
<p><img alt="已经停产的Truly Ergonomic" src="https://www.horsal.dev/images/2022-05-25-truly-ergonomic.jpg"></p>
<p>可惜的是最初的Truly Ergonomic Keyboard早就已经停产了,而沉寂了数年之后这家公司才推出了后继的<a href="https://trulyergonomic.com/ergonomic-keyboards/">Truly Ergonomic CLEAVE</a>键盘。</p>
<p><img alt="新版的Truly Ergonomic键盘" src="https://www.horsal.dev/images/2022-05-25-Truly-Ergonomic-CLEAVE.jpg"></p>
<p>新的键盘维持了老键盘的设计,依然是近些年人体工程学所追求的垂直键位排列,以及左右手能成自然状态的倾斜布局,有Function键看起来也很诱人。在快要下单的时候,我发现它似乎不是完全可编程的——说明里只表示Shift,ESC等极少数键位可以修改,官方也并没有提供刷固件的工具。然而作为一个Dvorak使用者,这些年我一直在用自己修改过的layout,回到QWERT显然已经不可能了——因此我决定寻找下一个能用的人体工程学键盘。</p>
<h2>Ergonomic Keyboard in 2022</h2>
<p>人体工程学并不是个什么新的话题,我还记得大概在20年前的大众软件上就出现过微软出品的人体工程学键盘,
在当时,中间凸起向两侧扭曲的键盘是一个不折不扣的新东西。其中<a href="https://www.amazon.co.jp/マイクロソフト-キーボード-Comfort-Keyboard-B2L-00009/dp/B000BQ6NO0">Comfort Curve Keyboard 2000</a>还附带了当时火热的多媒体键,可以说是历史的缩影了。然而当时的用户们对多媒体并不感冒,新鲜感过后,大部分稀奇古怪的多媒体功能很快就成了误触按钮导致卡顿的元凶,之后的事就成了另一个故事。微软在后续产品Comfort Curve Keyboard 3000上也果断的移除了所有的多媒体按键。</p>
<p><img alt="Microsoft Comfort Keyboard" src="https://www.horsal.dev/images/2022-05-25-microsoft.jpg"></p>
<p>与此同时,Kinesis也是人体工程学里的老将,从二十多年前开始,不论是Kinesis Contoured Keyboard,还是最新的Kinesis Advantage 2,招牌式的凹陷设计一直没有改动过,大拇指区的两大四小排列也是现在各种人体工程学键盘的祖先。不但如此,Kinesis连一直以来备受诟病的微型功能键也没有改动过,就算多少有些缺点,在没多少选择的过去,Kinesis依然是很多程序员梦想中的终极键盘。</p>
<p><img alt="Kinesis Advantage 2" src="https://www.horsal.dev/images/2022-05-25-kinesis.jpg"></p>
<p>在著名厂商之外,还有一些并不算是很出名的产品,比如<a href="http://www.typematrix.com/">TypeMatrix</a>就是其中一个。它不仅有着垂直键位,还在最初就支持硬件Dvorak,在<a href="https://wikiwiki.jp/fpag/TypeMatrix%20EZ-Reach%202020">最初版本的图片上</a>可以看到F7键上面标有Dvorak。这家公司现在依然在制作人体工程学键盘,而键盘布局跟最初的设计也是几乎没有改变,唯一的区别就是从双空格键变成了单一的长空格。</p>
<p>除此之外,<a href="https://www.moma.org/collection/works/3423">Apple甚至在93年就制作过一款人体工程学键盘</a>,键盘半分离甚至能够调整两手的角度,放到现在也是不多见的设计,而<a href="https://www.personal-media.co.jp/utronkb/article.html">日本也有人在91年发售了独特布局的键盘</a>,不过在那个年代,这些产品没有多少机会出现在中国的市场上,毕竟90年代末开始兴起的电脑热潮才将个人电脑带入了寻常百姓家。</p>
<p>进入00年代后半,国内对人体工程学的关注也逐渐多了起来,不论是Kensington的轨迹球产品,还是各种宣称能够减轻腱鞘炎的键盘都开始有了市场。除此之外,自制键盘也逐渐兴起,<a href="https://www.drop.com">Massdrop(现在的Drop)</a>也成了自制键盘相关的集换地。从自制键盘的潮流中,<a href="https://www.indiegogo.com/projects/ergodox-ez-an-incredible-mechanical-keyboard#/">Ergodox EZ成功成为了量产制品</a>,现在也成为了人体工程学键盘的代表。</p>
<p><img alt="Ergodox EZ" src="https://www.horsal.dev/images/2022-05-25-ergodoxez.png"></p>
<p>从设计上能够看到,Ergodox EZ继承了Kinesis的拇指按键排列,同时也继承了最近流行的分离式设计。除了经典的Ergodox之外,ZSA公司还推出了集成度更高的Moonlander,以及一个<a href="https://www.zsa.io/planck/">极其精简的Planck EZ</a>。</p>
<p>如果退一步,并不要求特殊的按键排列,只要分离式键盘即可的话则有更多的选择,比如<a href="https://dygma.com/products/dygma-raise">Dygma Raise</a>, <a href="https://ultimatehackingkeyboard.com/">Ultimate Hacking Keyboard</a>,还有<a href="https://www.mistelkeyboard.com/products/807a19ebcb8b2924891d471974d65438">Mistel的一些型号</a>。</p>
<h2>Ergodox EZ First Impression</h2>
<p>在比较了能够简单找到的产品之后,我最终决定切换到Ergodox EZ,一个重要的原因是<a href="https://ergodox-ez.com/pages/our-firmware">调整它的Firmware很容易</a>,官方提供的工具能够很轻松的设计一个自己的layout出来。</p>
<p><img alt="Ergodox EZ layout" src="https://www.horsal.dev/images/2022-05-25-layout.png"></p>
<p>在设计Layout的过程中,最大的问题就是右上角的按键数量不够——Ergodox EZ追求对称设计讲右上角的6个按键砍到了2个(实际上Home Row上的<code>'</code>键也被顺带砍掉了),这就引出了如何放置括号键的问题。虽然拇指区域有充足的预备,作为亚洲人,我每次试图按4个小按键都十分痛苦,不是需要全力伸展拇指,就是不得不整体离开Home Row,所以拇指区的8歌按键几乎成为了摆设。经过一段时间的实验之后,我决定把括号放在右手的正下方,这样拇指完全放松时恰好<code>[</code>键上,不容易出错并且负担较小。实际上,作为替代方案,还可以把这两个键分别放在现在的左手<code>Tab</code>以及<code>Backspace</code>上,不过跟现在的方案相比,左手并不能自然的按到这两个键。</p>
<p>另外,值得一提的是最初我曾经试图合并方向键和<code>CMD</code>, <code>Control</code>的,Ergodox EZ允许给长按设定单独的功能,这样可以省下4个珍贵的空间。但在使用中发现大部分情况下这么做并不是很实用,因为长按需要一段时间识别,而通常使用<code>CMD+L</code>等快捷键时不够注意就会导致没有反应。同时,方向键也经常需要长按,不论是网页的Scroll,还是在命令行里移动cursor。在一段时间仍然无法适应长按之后,我最终选择退回到普通的一对一layout上。</p>
<p>在固件之外,ergodox ez还有很多有趣的设计,比如它用来连接左右键盘的是3.5mm的接口,也就是曾经耳机最常用的规格,不过是稍微少见一点的4-pole的TRRS接口。而其他的产品,比如Mistel的分离式键盘,则都是用了Micro-usb之类的更加普通的规格。在Ergodox EZ到手之前,我的确用过一段时间Mistel,它的Micro-USB供电给人留下了很不好的印象——通过USB Hub切换电脑的时候会有一定概率导致左侧供电不足而无法识别,需要重新拔插才可以;Ergodox EZ则非常安定,不论是怎么拔插和切换都没有出现过类似的问题。另外,连接电脑端的USB Type-C也显得挺时髦的。</p>
<p>最后,在购买时可以选择是否需要Tilt Kit(用来垫高键盘的支架)和Wing Rest(腕垫),没买Tilt Kit让我有些后悔,键盘本身比较厚,平放在桌面上用起来会让手腕始终有个夹角,因此我选择把Mac Air垫在前面,还意外的合适。但下次再买,还是有Tilt Kit更好。</p>
<h2>Summary</h2>
<p>在几天的适应之后,我已经基本从失去Truly Ergonomic Keyboard的悲伤之中回过神来,可以正常的使用Ergodox EZ了。在购买的时候,回头看自己10年之前保存的那些人体工程学键盘,才发现实际上这10年并没有什么太大的变化。毕竟一切从几十年前就开始了,人体工程学键盘的设计说不定已经走到了尽头,不过这也是好消息,毕竟意味着我不需要多买几个备用键盘防止停产了。</p>Random Notes on Modern Monetary Theory2021-10-25T00:00:00+09:002021-10-25T00:00:00+09:00Horsaltag:www.horsal.dev,2021-10-25:/random-notes-on-modern-monetary-theory.html<p>黄金和白银都曾扮演过通用货币的角色,它们比较稀有,便于携带还不易损毁,用来作为衡量价值的中介并不是个很让人意外的事情。在很长一段时间里货币需要跟金银等价才能让人信服——不论是让货币本身含有金银,还是将纸币与金银挂钩。只要有一个足够强大的组织保证能这张纸兑换为真正的金银,那么这个货币的使用者和接收者自然不会产生质疑。</p>
<p>然而这个前提现在已经不复存在,美国在几十年前已经不再接受美元兑换黄金(<a href="https://en.wikipedia.org/wiki/Nixon_shock">Nixon shock</a>),绝大多数国家也都不再实行金本位。很显然脱离了黄金很带来很多好处,比如政府可以尽情的发行货币而不用担心黄金储备不足导致的信用破产,社会发展也不会受制于黄金开采速度(毕竟一个软件公司无法扩张是因为黄金开采不够快看起来并不合理)。然而反过来看,既然现代的货币已经跟稀缺资源没有任何关联,那么人们为什么不认为这些货币是废纸,而依然使用它们?</p>
<h2>Tax-Driven Monetary View</h2>
<p>Modern Monetary Theory(MMT)对这个问题的一个假设(或者推测)是货币的需要来自于政府强制性的征税。</p>
<p>如果政府强制性的征税,并且缴税只能使用政府发行的货币,那么人们就不得不试图获取政府的货币。当几乎所有人都持有这个货币的时候,货币就具有流通性了。在货币发行当初,为了让所有人都接受这个货币,政府需要设置人头税等普适的税种;但货币流通之后就算只对部分人征税一样可以达到同样的效果。假设政府废除了所有面向个人的税,从个人角度来看获取货币已经没有必要,公司也可以发行自己的“货币”,比如某种电子支付的点数,实际上日本已经有了用民营企业点数代替货币支付薪酬的讨论(<a href="https://www.nikkei.com/article/DGXZQOFK085NR0Y1A200C2000000/">デジタル給与払い</a>)。</p>
<blockquote>
<p>資金移動業者が発行するプリペイド(前払い …</p></blockquote><p>黄金和白银都曾扮演过通用货币的角色,它们比较稀有,便于携带还不易损毁,用来作为衡量价值的中介并不是个很让人意外的事情。在很长一段时间里货币需要跟金银等价才能让人信服——不论是让货币本身含有金银,还是将纸币与金银挂钩。只要有一个足够强大的组织保证能这张纸兑换为真正的金银,那么这个货币的使用者和接收者自然不会产生质疑。</p>
<p>然而这个前提现在已经不复存在,美国在几十年前已经不再接受美元兑换黄金(<a href="https://en.wikipedia.org/wiki/Nixon_shock">Nixon shock</a>),绝大多数国家也都不再实行金本位。很显然脱离了黄金很带来很多好处,比如政府可以尽情的发行货币而不用担心黄金储备不足导致的信用破产,社会发展也不会受制于黄金开采速度(毕竟一个软件公司无法扩张是因为黄金开采不够快看起来并不合理)。然而反过来看,既然现代的货币已经跟稀缺资源没有任何关联,那么人们为什么不认为这些货币是废纸,而依然使用它们?</p>
<h2>Tax-Driven Monetary View</h2>
<p>Modern Monetary Theory(MMT)对这个问题的一个假设(或者推测)是货币的需要来自于政府强制性的征税。</p>
<p>如果政府强制性的征税,并且缴税只能使用政府发行的货币,那么人们就不得不试图获取政府的货币。当几乎所有人都持有这个货币的时候,货币就具有流通性了。在货币发行当初,为了让所有人都接受这个货币,政府需要设置人头税等普适的税种;但货币流通之后就算只对部分人征税一样可以达到同样的效果。假设政府废除了所有面向个人的税,从个人角度来看获取货币已经没有必要,公司也可以发行自己的“货币”,比如某种电子支付的点数,实际上日本已经有了用民营企业点数代替货币支付薪酬的讨论(<a href="https://www.nikkei.com/article/DGXZQOFK085NR0Y1A200C2000000/">デジタル給与払い</a>)。</p>
<blockquote>
<p>資金移動業者が発行するプリペイド(前払い)式の給与振り込み用カード「ペイロールカード」の導入が想定されている。企業は銀行などの金融機関を経由せずに直接ペイロールカードの口座に振り込むことができる。</p>
</blockquote>
<p>不过这并不代表政府货币变成了废纸,因为存在法人税,公司依然为了缴税而需要获得这个货币,因此一个商品的上游产业链都对货币存在需求,显然这些货币反映在商品上——如果所有人都只用点数买东西,那么公司就无法获得足够的货币了。对货币的需求总是会传导到个人身上,让整个社会都承认政府货币。</p>
<p>从MMT的观点来看,现代的货币与金本位货币的逻辑是相反的——税收是用来创造货币需求的工具,而不是用来回收货币的。在政府能够自由发行货币的前提下,回收自己的货币作为财源对政府没有意义。虽然政府并不需要把税当作财源,收税除了创造货币需求确实能为政府带来利益。税收本身就是对居民施加的义务,需要居民付出一定代价为政府提供利益,过去这个代价可能是一定量的粮食,可能是为政府工作的人力,在货币出现之后这些大都折算成了货币而已。只要税收存在,政府就总是能够获得居民的一部分“生产力”作为利益。</p>
<p>MMT的理论中也提到,货币是政府的一种承诺(负债),即政府承诺人们可以通过货币来完成<strong>未来</strong>纳税的义务。一个人获得了10USD,那么就意味着他可以在未来向政府换取10USD量的纳税义务完结证明。这样看来,政府发行的货币量也就代表了政府承诺完结的纳税义务量,如果政府承诺的数量足够巨大(比如未来100年)并且每个人都有了足够的货币,那么人们就会开始认为这个货币已经不是必须的,如果货币进一步增多,那么人们就会认为它失去了价值,毕竟没有多少人愿意为了100年之后的纳税义务而保存货币,这就导致了通货膨胀:人们会试图“低价甩卖”这个纳税用的货币来换取更有意义的东西。所有人都试图这么做的时候货币(纳税义务)自然就一文不值了。此时政府就必须通过加税来维持货币的必要性。</p>
<h2>Stock-Flow Consistent Model</h2>
<p>MMT里另外一个思想则是把所有部门的负债表放到一起,它们整体是稳定的——即所有部门的结余(赤字或盈余)的总和是0。这个想法在物理上是很自然的,如果存在一个孤立系统,它跟外部不存在任何交换,那么自然内部是守恒的。MMT将部门大致分为三类:政府部门,国内私人部门,海外部门。那么政府部门如果连年盈利,这就代表了国内私人部门+海外部门需要亏损。反过来说,当前备受关注的政府赤字实际上代表了私营部门的盈利,这对经济甚至是有利的。</p>
<p>对于这个观点存在着很多批判,比如MMT的模型过于简单,没有考虑汇率的影响,所以不能单纯的应用孤立系统模型;比如央行的负债表具有特殊意义,不应直接与普通部门的负债表相比较,等等。MMT的支持者则认为MMT成功的预见到了几次经济危机,包括90年代末克林顿政府下的互联网泡沫消退。根据MMT的思想,当时政府连年盈余反应了私人部门的赤字,当时对问题的轻视最终导致了经济危机。</p>
<p>基于这个思想,可以认为政府部门的赤字并不是问题,某种程度的赤字对经济还是有利的。这也是经常见到的MMT=无限财政赤字这一说法的由来。从货币来源上讲,如果财政赤字是以本国货币计价的话,政府显然总是能够通过发行货币填补,赤字并不会导致政府的破产。过去经历过恶性通胀的国家也都不是单纯的发行货币导致的,一战之后的德国由于存在大量的本国货币之外的债务(黄金),印发马克并不能偿还黄金,因此政府需要用手里的马克跟居民争抢有限的黄金资源(严格来说,是政府通过争抢其他资源进行出口获取黄金),这才导致了国内物价的恶性通胀。而津巴布韦的情况也很类似,国内的粮食极度短缺,进口粮食让政府有了外债,最终导致了政府和居民争抢稀缺的资源,从而拉高物价。</p>
<p>从这些例子可以看出,发行货币确实是通货膨胀的必要条件,但却并不是充分条件,因此简单的把“超发货币”等同于“通货膨胀”并不是合理。只要能够阻断别的条件,就完全能够在发行货币的同时回避恶性通货膨胀。MMT给出的其中一个办法就是调整税收,有些观点认为目前的政府结构根本无法做到灵活的调整税收,因此这个方案根本无法实现,但这个批判的背后有着一些不同的思想——即政府是至高无上的,经济应该全面的为政府服务。不然的话,只要合理的改革政府结构,灵活的调整税收并不是无法实现的,虽然几乎不会有人为了一个效果不确定的政策去改革政府。</p>
<h2>Functional Finance</h2>
<p>通过上面的观点,可以得出一个结论:既然政府保证财政平衡没有意义,那么财政政策应该集中在保证货币稳定上。在这个思想下,债券并不能解释为政府的融资,政府既然能够发行无息债券(货币),自然不需要支付利息去融资。政府通过发行债券,设定不同的利率来调整私人部门持有的国债和货币比例,从而影响私人部门的投资规模。只要能够实现这一点,债券甚至不是必要的。除此之外,保证就业也是一个重要的手段,政府提供一个最低工资基准,并且无限制的提供劳动岗位的话,商品价格就能建立在<a href="https://zh.wikipedia.org/wiki/一般均衡理论">瓦尔拉斯一般均衡</a>之上,这个最低工资标准即成为政府调整的工具之一。保证就业的思想即使MMT里所提到的Job Guarantee Program或者Employer of Last Resort Program制度。</p>
<p>值得一提的是,要实现功能性财政需要央行和政府各部门融为一体,不但央行需要无条件配合政府的调控政策,政府内部也不应该存在隔阂,否则调控就无法像MMT所设想的一样迅速(上一章的税收就是一例)。但目前大多数国家的央行都具有一定独立性,行政灵活性也无法达到MMT要求的水准。</p>
<h2>Summary</h2>
<p>虽然MMT里带有Theory,但它看起来并不像一个理论而更像一个观察或者思考。MMT有税收驱动货币的前提假设,在此之上逐步推论了一些与传统思想不同的结论。但作为一个Theory,MMT还是缺乏足够全面的思考,对Corner Case分析,以及实验或者事例证实。尤其是在一般的国家制度之下,MMT的要求并不容易实现,因此即使出现了MMT的失败也不会结束论争——反对者们会声称这是异端的失败,而支持者们则认为只执行MMT的一部分显然不会得到效果。</p>Things Behind ATH-DSR9bt(2)2021-09-10T00:00:00+09:002021-09-10T00:00:00+09:00Horsaltag:www.horsal.dev,2021-09-10:/things-behind-ath-dsr9bt2.html<h2>Beyond PCM</h2>
<p><a href="https://www.horsal.dev/things-behind-ath-dsr9bt1.html">前回のブログ</a>では今のフルデジタル製品の大元、Sonyが提案したパルス符号変調(PCM)ベースのデジタルヘッドホンを紹介した。ただし、最後に問題が残っていた:この方式では、ボイスコイルの巻き数を<span class="math">\(2^{N-1}\)</span>までしないと実現できない。これだと超大型の製品しかできないし、かなりの精度で作らないと音の歪みが発生する。とすると、もっといい方法はないだろうか?</p>
<p>当然改善し続ける選択肢もあるし、例えばPhillipsはソニーの手法を改善している<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>が、そもそも音の出し方自体を変えるとどうなるだろうか?</p>
<h3>From PCM to PWM</h3>
<p>前回で紹介したPCM方式では、一定時間毎に、音声信号の強さを量子化して記録する方式だ。逆に考えば、信号の強さを固定にして時間を変化すれば、信号も記録できるではないか?ということで、<a href="https://ja.wikipedia.org/wiki/%E3%83%91%E3%83%AB%E3%82%B9%E5%B9%85%E5%A4%89%E8%AA%BF">パルス幅変調(PWM)</a>もしくは<a href="https://ja.wikipedia.org/wiki/%E3%83%91%E3%83%AB%E3%82%B9%E5%AF%86%E5%BA%A6%E5%A4%89%E8%AA%BF">パルス密度変調(PDM)</a>はこのような方式だ。</p>
<p><img alt="PDM" src="https://www.horsal.dev/images/2021-08-25-Pulse_density_modulation.png"></p>
<p>緑の曲線は音声で、青線は量子化したPDM信号となっている。PDMの信号は0、1の2つの状態しかない(もしくは、0、1、ー1の3つでもよい)、元の信号が強ければ、PDMの信号は1にいる時間が長い …</p><h2>Beyond PCM</h2>
<p><a href="https://www.horsal.dev/things-behind-ath-dsr9bt1.html">前回のブログ</a>では今のフルデジタル製品の大元、Sonyが提案したパルス符号変調(PCM)ベースのデジタルヘッドホンを紹介した。ただし、最後に問題が残っていた:この方式では、ボイスコイルの巻き数を<span class="math">\(2^{N-1}\)</span>までしないと実現できない。これだと超大型の製品しかできないし、かなりの精度で作らないと音の歪みが発生する。とすると、もっといい方法はないだろうか?</p>
<p>当然改善し続ける選択肢もあるし、例えばPhillipsはソニーの手法を改善している<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>が、そもそも音の出し方自体を変えるとどうなるだろうか?</p>
<h3>From PCM to PWM</h3>
<p>前回で紹介したPCM方式では、一定時間毎に、音声信号の強さを量子化して記録する方式だ。逆に考えば、信号の強さを固定にして時間を変化すれば、信号も記録できるではないか?ということで、<a href="https://ja.wikipedia.org/wiki/%E3%83%91%E3%83%AB%E3%82%B9%E5%B9%85%E5%A4%89%E8%AA%BF">パルス幅変調(PWM)</a>もしくは<a href="https://ja.wikipedia.org/wiki/%E3%83%91%E3%83%AB%E3%82%B9%E5%AF%86%E5%BA%A6%E5%A4%89%E8%AA%BF">パルス密度変調(PDM)</a>はこのような方式だ。</p>
<p><img alt="PDM" src="https://www.horsal.dev/images/2021-08-25-Pulse_density_modulation.png"></p>
<p>緑の曲線は音声で、青線は量子化したPDM信号となっている。PDMの信号は0、1の2つの状態しかない(もしくは、0、1、ー1の3つでもよい)、元の信号が強ければ、PDMの信号は1にいる時間が長い。つまり信号の強さを時間の長さで表現している。厳密に言うと、音声の強さはPDM信号の積分になっていて、音声<span class="math">\(P = \int_{t \in PDM=1}1dt + \int{t \in PDM=-1}-1dt\)</span>のような感じで計算されるのだ。これ以上の紹介はWikiの<a href="https://ja.wikipedia.org/wiki/%CE%94%CE%A3%E5%A4%89%E8%AA%BF">ΔΣ変調</a>を読めば分かるのはず。</p>
<p>ということで、この方式であれば、量子化した信号の強さは0か1の二種類しかないため、強さのバリエーションが多すぎることはまず発生しないので、PCMにあった問題を解決できるかもしれない。実は、今のフルデジタルヘッドホンもこの考えをベースに作っている。とりあえず特許を検索してみると...</p>
<blockquote>
<p>特開昭59-181897 音響装置(ソニー株式会社)</p>
</blockquote>
<p>こっちもだいぶ昔ソニーが関連している特許を出していた。当然ソニーだけでなく、海外含めていらんな所もこの方式を研究していた。複数のスピーカーを利用する方式もあれば<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup><sup id="fnref:3"><a class="footnote-ref" href="#fn:3">3</a></sup>、1つのスピーカーにPWMの0と1それぞれに対応するボイスコイルを入れる方式<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">4</a></sup>も考えられている。ここでは、Sharp Corpが提案した方式<sup id="fnref2:4"><a class="footnote-ref" href="#fn:4">4</a></sup>を簡単に説明する。</p>
<p><img alt="Sharp" src="https://www.horsal.dev/images/2021-09-10-sharp.png"></p>
<p>ボイスコイルは二種類が存在し、プラス(+)方向に動くものと、マイナス(ー)方向に動くものだ。プラス方向のボイスコイルに電流流せばその方向に動き続けるし、切断すると自然に戻ってくる。マイナスは逆方向に同じ動きをする。そうすると、振動板の動きは+、ー、0の3種類の動きができるようになる。</p>
<p>上のPWMの話を思い出すと、PWM信号も同じ構成なので、デジタル信号に合わせて、セレクターで2つボイスコイルのOn/offを切り替えることで、そのまま音声信号に変換できる。この処理は信号をアナログに変換することなく、デジタルそのまま振動板に持って行けたといえるでしょう。</p>
<p>この考え方はDSR9BTまでずっと変わっていないが、上の述べた方法はいくつの問題があった。
* PWM信号の周波数がとんでもない高い。よく使っているPCM音楽なら周波数は44.1KHzだが、<a href="https://ja.wikipedia.org/wiki/Super_Audio_CD">SACD</a>のようなΔΣ変調の形式だと<strong>2922.4KHz</strong>も当たり前は。この周波数に合わせてセレクターやスイッチがOn/Offするが、さすがにこんな高い周波数だとエラーが起きやすい。更に厄介なのは、エラーが一回でも発生すると、あとの処理に影響し続けるので、何回もエラー起きると音がめちゃくちゃになる。
* スイッチが高速にOn/Offすると、熱が溜まったり、効率が下がったりするので、コンパクトな製品はほぼ作れない。</p>
<p>この後、PWM方式の周波数を如何に下げることに対して研究が広がってきた。</p>
<h2>From Single Bit to Multiple Bits</h2>
<p>当然、単純に周波数を削るだけなら、記録できる情報量が減るのでよくない。ならば、量子化ビット数を2、3などに少し増やせばどうだろうか?</p>
<p>ビット数が8や16になると前回で紹介したようにボイスコイルが多すぎて対応できないが、2、3ビットならば全然問題なく対応できるし、周波数も当然その分減らすことができる。ということで、90年代末から多ビットΔΣ変調の方法が考えられてきた。この中の1つは<strong>DNote</strong>と呼ばれている変換方式である<sup id="fnref:5"><a class="footnote-ref" href="#fn:5">5</a></sup><sup id="fnref:6"><a class="footnote-ref" href="#fn:6">6</a></sup><sup id="fnref:7"><a class="footnote-ref" href="#fn:7">7</a></sup>。言うまでもないが、DSR9BTはこの技術を基づいて作った製品だ。</p>
<p>DNoteに対して提案当時からDSR9BT発売まで複数回インタビューがあり、開発元のTrigence Semiconductor社が技術の詳細を紹介してたが、あまりにも一般ユーザーに難しく当時はあまり理解されず、話題にもならなかった。</p>
<p>ただし、今回上に書いているようなことを把握できていれば、その紹介を分かるようになるはずだ。</p>
<p>まず1つは<a href="https://av.watch.impress.co.jp/docs/series/dal/642074.html">藤本健のDigital Audio Laboratory, 第587回, AV Watch</a>。記事に添付されている図はここに載せないが、DNoteは<span class="math">\(-4 \dots 4\)</span>の9通り、4BITを採用している。DSR9BTも同じく<a href="https://ascii.jp/elem/000/001/250/1250078/">「4芯ボイスコイル」</a>になっているため、技術上は同じものだと分かる。1ビットの前と同じ、1ボイスコイルは+1かー1しか出せないが、今回4つも持っているので、4つに同時同じ方向の電流流せば、それで+4になれる。</p>
<h2>Summary</h2>
<p>DSR9BTは正真正銘のフルデジタルで間違いない。ただし、多くのユーザーがまだ3.5MM端子のアナログが最高と考えている中、時代を先取りしすぎた製品はあまり注目されないだろう。また、オーディオテクニカの宣伝力も追いついていなく、仮にソニーのハイレゾ戦略と同じくらいの力を入れたら、1つのカテゴリーを確立できたかもしれない。いずれにせよ、DSR9BTが生産終了している今、オーディオテクニカの再チャレンジを末しかない。しかし、DNoteを使う限り、特許はオーディオテクニカ側に一切ないため<sup id="fnref:8"><a class="footnote-ref" href="#fn:8">8</a></sup>、新製品の企画も開発の調整も難しいだろう。</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>Method of and test arrangement for testing the transmission path within an apparatus of a modular construction for interruptions, US4612421A, Gunther Spath and Werner Zeder, US Philips Corp <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>Method and apparatus to create a sound field, GB2373956A, Tony Hooley, Paul Troughton, Angus Goudie and Mark Easton, 1 Ltd <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>Selection apparatus, Akira Yasuda, Toshiba, US5872532A <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>Speaker driving circuit, Ryutaro Takahashi, Toru Hayase, Sharp Corp., US5592559A <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 4 in the text">↩</a><a class="footnote-backref" href="#fnref2:4" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:5">
<p>特許第4883428号, デジタルアナログ変換装置,株式会社 Trigence Semiconductor <a class="footnote-backref" href="#fnref:5" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:6">
<p>特開2010-28783(P2010-28783A) デジタルスピーカー駆動装置,デジタルスピーカー装置,アクチュエータ,平面ディスプレイ装置及び携帯電子機 <a class="footnote-backref" href="#fnref:6" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:7">
<p>厳密にDNoteはただの方式ではなく、多ビットΔΣ変調を利用した変換装置とその周りをパッケージ化したものだが、ここでは特に区別しない。 <a class="footnote-backref" href="#fnref:7" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:8">
<p>ヘッドホンのデザインの特許はオーディオテクニカ側が持っている。例えば特許第6296669号 デジタル駆動型ヘッドホンや特開2018-46445など。 <a class="footnote-backref" href="#fnref:8" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Formal Concept Analysis with Rust (3) - Parallization2021-08-20T00:00:00+09:002021-08-20T00:00:00+09:00Horsaltag:www.horsal.dev,2021-08-20:/formal-concept-analysis-with-rust-3-parallization.html<p><a href="https://www.horsal.dev/formal-concept-analysis-with-rust-2-basic-algorithm.html">上一篇文章</a>实现了一个简单的Formal Concept的Enumerator。不过实际测试结果表明单纯列举的速度并不足以对应足够大的数据。至少对3000个Object就花掉接近8分钟的算法很难让人接受。这次将利用<a href="https://tokio.rs/">tokio</a>将上一篇里的算法并行化,来看一看改进之后的速度。</p>
<h2>Iterator-style Enumeration</h2>
<p>在动手加入多线程之前,先来把上一次写的代码整理一下。上一篇里用了普通的<code>for obj in cur_obj..r.row.len() { ... }</code>的循环,虽然没有什么问题,但可以写的更加"rust风"一点,比如用上Iterator:</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="n">cur_obj</span><span class="o">..</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">filter</span><span class="p">(</span><span class="o">|</span><span class="n">obj</span><span class="o">|</span><span class="w"> </span><span class="o">!</span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="o">*</span><span class="n">obj</span><span class="p">).</span><span class="n">unwrap_or</span><span class="p">(</span><span class="kc">false</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">filter_map</span><span class="p">(</span><span class="o">|</span><span class="n">obj</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">extent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">extent</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="n">obj …</span></code></pre></div><p><a href="https://www.horsal.dev/formal-concept-analysis-with-rust-2-basic-algorithm.html">上一篇文章</a>实现了一个简单的Formal Concept的Enumerator。不过实际测试结果表明单纯列举的速度并不足以对应足够大的数据。至少对3000个Object就花掉接近8分钟的算法很难让人接受。这次将利用<a href="https://tokio.rs/">tokio</a>将上一篇里的算法并行化,来看一看改进之后的速度。</p>
<h2>Iterator-style Enumeration</h2>
<p>在动手加入多线程之前,先来把上一次写的代码整理一下。上一篇里用了普通的<code>for obj in cur_obj..r.row.len() { ... }</code>的循环,虽然没有什么问题,但可以写的更加"rust风"一点,比如用上Iterator:</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="n">cur_obj</span><span class="o">..</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">filter</span><span class="p">(</span><span class="o">|</span><span class="n">obj</span><span class="o">|</span><span class="w"> </span><span class="o">!</span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="o">*</span><span class="n">obj</span><span class="p">).</span><span class="n">unwrap_or</span><span class="p">(</span><span class="kc">false</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">filter_map</span><span class="p">(</span><span class="o">|</span><span class="n">obj</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">extent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">extent</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="kc">true</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">concept</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">closure</span><span class="p">(</span><span class="o">&</span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">extent</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">mask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BitVec</span>::<span class="n">from_fn</span><span class="p">(</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">(),</span><span class="w"> </span><span class="o">|</span><span class="n">i</span><span class="o">|</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">cur_left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_left</span><span class="p">.</span><span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">mask</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">new_left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">new_left</span><span class="p">.</span><span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">mask</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">cur_left</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">new_left</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Some</span><span class="p">((</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">concept</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">None</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">for_each</span><span class="p">(</span><span class="o">|</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">concept</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="n">enumerate_fc</span><span class="p">(</span><span class="n">concept</span><span class="p">,</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">clone</span><span class="p">(),</span><span class="w"> </span><span class="n">should_print</span><span class="p">));</span><span class="w"></span>
</code></pre></div>
<p>这里把之前的<code>for .. in</code>改成了一个Range的Iterator,首先通过一个<code>filter</code>去掉那些已经被加进来的concept,然后通过<code>filter_map</code>进行计算,最后用一个<code>for_each</code>对所有的结果进行递归。这里为了简单没有对<code>filter_map</code>进行更细的切分,如果你愿意实际上可以把它们写成更小的模块:</p>
<div class="highlight"><pre><span></span><code><span class="p">(</span><span class="n">cur_obj</span><span class="o">..</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">filter</span><span class="p">(</span><span class="o">|</span><span class="n">obj</span><span class="o">|</span><span class="w"> </span><span class="o">!</span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="o">*</span><span class="n">obj</span><span class="p">).</span><span class="n">unwrap_or</span><span class="p">(</span><span class="kc">false</span><span class="p">))</span><span class="w"> </span><span class="c1">// filter out existed branches</span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="n">obj</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">extent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">extent</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="kc">true</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">closure</span><span class="p">(</span><span class="o">&</span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">extent</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="c1">// calculate closure</span>
<span class="w"> </span><span class="p">.</span><span class="n">filter</span><span class="p">(</span><span class="o">|</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">concept</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">mask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BitVec</span>::<span class="n">from_fn</span><span class="p">(</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">(),</span><span class="w"> </span><span class="o">|</span><span class="n">i</span><span class="o">|</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">cur_left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_left</span><span class="p">.</span><span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">mask</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">new_left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">new_left</span><span class="p">.</span><span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">mask</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_left</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">new_left</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="c1">// filter out processed branches</span>
<span class="w"> </span><span class="o">....</span><span class="w"></span>
</code></pre></div>
<p>你或许会担心Iterator对性能的影响,但根据<a href="https://doc.rust-lang.org/book/ch13-04-performance.html">官方的说法,Iterator对性能的影响是微乎其微的</a>,而实际的测试也证明了这一点:</p>
<p><img alt="Iterator_vs_Standard_for" src="https://www.horsal.dev/images/2021-08-20-iterator.png"></p>
<p>蓝色的线是普通的<code>for-loop</code>,而橙色的线是上面的Iterator,可以虽然没有像Rust的文档里面声称的Iterator快于普通的循环,但二者总体上并没有性能差。相比于性能,Iterator能够带来更好的代码模块化(可读性),这一点更加重要。</p>
<h2>Straightforward Parallization</h2>
<p>下面把这个算法写成多线程的版本,不用参考论文也能看到,其实只要把<code>for_each</code>里面的递归改成新的线程就能够实现一个简单的多线程版本了,因此先来动手去实现它。</p>
<p>首先需要一个多线程的环境,就像最初所说的,这里用了<code>tokio 1.x</code>,添加完Crate后先将<code>main</code>函数改成<code>async</code>:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#[tokio::main(flavor=</span><span class="s">"multi_thread"</span><span class="cp">, worker_threads=8)]</span><span class="w"></span>
<span class="k">async</span><span class="w"> </span><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="p">(),</span><span class="w"> </span><span class="n">Error</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p><code>tokio</code>跟Go都Go Routine类似并没有直接使用系统的线程,而是自己有一个线程池和一套Task的定义,<code>worker_threads</code>指定了<code>tokio</code>使用系统线程的数量。另外值得注意的是,跟Go类似,<code>tokio</code>的Task(也就是之后会用到的各种async spawn)是非常轻量的,在它们之间切换并没有很大的开销,因此完全可以向Erlang一样同时Spawn几千甚至几万个线程而不必担心它们之间会产生巨大的Overhead。这也是为什么这里能够简单的把每次递归都变成线程而不用担心性能的原因。</p>
<p>下一步便是把之前的<code>for_each</code>循环写成task的Spawn,它看起来就像是这样:</p>
<div class="highlight"><pre><span></span><code><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="c1">// calculate and filter (obj, concept)</span>
<span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">concept</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">tokio</span>::<span class="n">task</span>::<span class="n">spawn</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="k">move</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">enumerate_fc</span><span class="p">(</span><span class="n">concept</span><span class="p">,</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="n">should_print</span><span class="p">).</span><span class="k">await</span><span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="p">})</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">t</span><span class="p">.</span><span class="k">await</span><span class="p">.</span><span class="n">unwrap</span><span class="p">().</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>但这里仍然有几点需要注意:</p>
<ul>
<li>由于async task的Lifetime要求要足够长(或者说是<code>'static</code>),因此传递一个引用进去是无法编译的,需要把所有的内容都改成传值:</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">enumerate_fc</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_concept</span>: <span class="nc">FormalConcept</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_obj</span>: <span class="kt">usize</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">r</span>: <span class="nc">Arc</span><span class="o"><</span><span class="n">Relation</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">should_print</span>: <span class="kt">bool</span><span class="p">,</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
</code></pre></div>
<ul>
<li>
<p>这里的<code>Relation</code>是读取的数据,在整个处理过程中是不变的,显然而在每次循环中都要把它传给下一个递归函数,因此不得不进行<code>Clone</code>,但纯粹的<code>Clone</code>显然很浪费,因此这里使用了<code>Arc</code>。</p>
</li>
<li>
<p>Rust的设计上并不支持直接对<code>async fn</code>的递归调用,因此通常需要一些<a href="https://users.rust-lang.org/t/tokio-recursion-error/41750/23">别的办法</a>,而至于为什么,<a href="https://rust-lang.github.io/async-book/07_workarounds/04_recursion.html">Rust的官方Doc里有一些解释</a>:</p>
</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">enumerate_fc</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_concept</span>: <span class="nc">FormalConcept</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_obj</span>: <span class="kt">usize</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">r</span>: <span class="nc">Arc</span><span class="o"><</span><span class="n">Relation</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">should_print</span>: <span class="kt">bool</span><span class="p">,</span><span class="w"></span>
<span class="p">)</span><span class="w"> </span>-> <span class="nc">BoxFuture</span><span class="o"><'</span><span class="nb">static</span><span class="p">,</span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="p">(),</span><span class="w"> </span><span class="p">()</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="k">move</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">tokio</span>::<span class="n">spawn</span><span class="p">(</span><span class="k">async</span><span class="w"> </span><span class="k">move</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="n">enumerate_fc</span><span class="p">(</span><span class="o">..</span><span class="p">.).</span><span class="k">await</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}.</span><span class="n">boxed</span><span class="p">()</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>所以,最终这个函数看起来像是这样的:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">enumerate_fc</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_concept</span>: <span class="nc">FormalConcept</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_obj</span>: <span class="kt">usize</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">r</span>: <span class="nc">Arc</span><span class="o"><</span><span class="n">Relation</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">should_print</span>: <span class="kt">bool</span><span class="p">,</span><span class="w"></span>
<span class="p">)</span><span class="w"> </span>-> <span class="nc">BoxFuture</span><span class="o"><'</span><span class="nb">static</span><span class="p">,</span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="p">(),</span><span class="w"> </span><span class="p">()</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="k">move</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">should_print</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">all</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">cur_obj</span><span class="o">..</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">filter</span><span class="p">(</span><span class="o">|</span><span class="n">obj</span><span class="o">|</span><span class="w"> </span><span class="o">!</span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="o">*</span><span class="n">obj</span><span class="p">).</span><span class="n">unwrap_or</span><span class="p">(</span><span class="kc">false</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">filter_map</span><span class="p">(</span><span class="o">|</span><span class="n">obj</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">extent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">extent</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="kc">true</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">concept</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">closure</span><span class="p">(</span><span class="o">&</span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">extent</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// mask: 11111111111111111 00000000000</span>
<span class="w"> </span><span class="c1">// ----------------- -----------</span>
<span class="w"> </span><span class="c1">// 0 ... cur_obj-1 cur_obj ...</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">mask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BitVec</span>::<span class="n">from_fn</span><span class="p">(</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">(),</span><span class="w"> </span><span class="o">|</span><span class="n">i</span><span class="o">|</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">cur_left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_left</span><span class="p">.</span><span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">mask</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">new_left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">new_left</span><span class="p">.</span><span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">mask</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">cur_left</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">new_left</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Some</span><span class="p">((</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">concept</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">None</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">concept</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">tokio</span>::<span class="n">task</span>::<span class="n">spawn</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="k">move</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">enumerate_fc</span><span class="p">(</span><span class="n">concept</span><span class="p">,</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="n">should_print</span><span class="p">).</span><span class="k">await</span><span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}).</span><span class="n">collect</span>::<span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="n">_</span><span class="o">>></span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="nb">Ok</span><span class="p">(())</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">boxed</span><span class="p">()</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这里还有一个问题:如何等待所有的Task结束?</p>
<p>Tokio在Spawn的时候会返回一个<code>JoinHandle</code>,对于每个Handle都可以进行Wait,因此如果你在网上进行搜索,<a href="https://users.rust-lang.org/t/await-multiple-tasks-spawned-by-tokio/57741">可以找到</a>类似这样的做法:</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="w"> </span><span class="n">h</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">HandleVectors</span><span class="p">.</span><span class="n">iter</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">h</span><span class="p">.</span><span class="k">await</span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>写进这里就像是这样的:</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="p">(</span><span class="n">cur_obj</span><span class="o">..</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="o">....</span><span class="w"> </span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">concept</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">tokio</span>::<span class="n">task</span>::<span class="n">spawn</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="k">move</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">enumerate_fc</span><span class="p">(</span><span class="n">concept</span><span class="p">,</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="n">should_print</span><span class="p">).</span><span class="k">await</span><span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"> </span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">i</span><span class="p">.</span><span class="k">await</span><span class="p">.</span><span class="n">unwrap</span><span class="p">().</span><span class="n">unwrap</span><span class="p">()</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>编译之后让来看一看它的速度怎么样:</p>
<p><img alt="for_wait_performance" src="https://www.horsal.dev/images/2021-08-20-parallel-for-await.png"></p>
<p>浅黄色的线是多线程版,可以看到它的速度丝毫没有变快。而至于CPU的使用率:</p>
<p><img alt="cpu_for_wait" src="https://www.horsal.dev/images/2021-08-20-cpu-for-await.png"></p>
<p>很明显这些CPU并没有被(完全)使用,原因是Iterator是Lazy的,也就是如果直接用在for里的话,里面的代码只会在<code>next</code>调用时才会实际计算。而由于每一次<code>for</code>后都会等当前任务完成,所以这个代码实际上就是个单线程。因此应该加上一个<code>collect</code>保证里面的内容能够先被执行:</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="p">(</span><span class="n">cur_obj</span><span class="o">..</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="o">....</span><span class="w"> </span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">concept</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">tokio</span>::<span class="n">task</span>::<span class="n">spawn</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="k">move</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">enumerate_fc</span><span class="p">(</span><span class="n">concept</span><span class="p">,</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="n">should_print</span><span class="p">).</span><span class="k">await</span><span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"> </span>
<span class="w"> </span><span class="p">.</span><span class="n">collect</span>::<span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="n">_</span><span class="o">>></span><span class="p">()</span><span class="w"> </span><span class="c1">// 注意这里</span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">i</span><span class="p">.</span><span class="k">await</span><span class="p">.</span><span class="n">unwrap</span><span class="p">().</span><span class="n">unwrap</span><span class="p">()</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>除此之外,<code>future</code>提供的<code>join_all</code>也可以同时处理所有的JoinHandle:</p>
<div class="highlight"><pre><span></span><code><span class="n">join_all</span><span class="p">((</span><span class="n">cur_obj</span><span class="o">..</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">concept</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">tokio</span>::<span class="n">task</span>::<span class="n">spawn</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">async</span><span class="w"> </span><span class="k">move</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">enumerate_fc</span><span class="p">(</span><span class="n">concept</span><span class="p">,</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="n">should_print</span><span class="p">).</span><span class="k">await</span><span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"> </span><span class="c1">//join_all可以用在Iterator上,不需要collect</span>
<span class="w"> </span><span class="p">).</span><span class="k">await</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p><code>join_all</code>是个<code>Future</code>的Wrapper,每一轮它都会检查所有的task有没有完成,如果全部完成就会返回。</p>
<p>总之,不论是哪一种方法都得到一个真正并列的版本了,看一下CPU,每一个都是100%的使用率:</p>
<p><img alt="CPU_join_all" src="https://www.horsal.dev/images/2021-08-20-cpu-parallel.png"></p>
<p>而执行速度也有了大幅的提升,对于<code>3000 objects</code>的数据变快了5倍:</p>
<p><img alt="Performance_join_all" src="https://www.horsal.dev/images/2021-08-20-parallel.png"></p>
<p>有趣的是,<code>for - await</code>的做法比<code>join_all</code>要稍微快一点点:</p>
<p><img alt="join_all-vs-for-await" src="https://www.horsal.dev/images/2021-08-20-join_all-vs-for_await.png"></p>
<p>因为<code>join_all</code>每次都会轮询所有的task,相比于只看第一个task多少处理上会有些overhead。
到这里,第一个多线程的版本就完成了。</p>
<h2>Impact of Number of Tasks</h2>
<p>不过,从一开始就有的一个问题依然没有解决:这个程序对每一个分支都生成了一个Task,就算Tokio的Task非常轻量,面对几十万甚至上百万的Task,这个Overhead真的能够忽略不计吗?</p>
<p>总之,先来看一下执行过程中Task调度到底执行了多少次。在linux下可以用<code>perf</code>很轻松的得到结果,而MacOS下则要用<code>dtrace</code>,Windows下面通常用<code>Windows Perform Analyzer</code>。这里用了<code>dtrace</code>,得到的FlameGraph是这样的:</p>
<p><img alt="Flamegraph" src="https://www.horsal.dev/images/2021-08-20-flamegraph.png"></p>
<p><code>call_mut::h28c458add25703bf</code>占据了几乎所有的执行时间,需要注意的是,这个函数其实就是<code>enumerate_fc</code>函数,因为为了规避递归出现的错误,这个函数返回了一个future然后在外部执行了它,因此<code>enumerate_fc</code>的名字不会出现在里面,取而代之的是一个async的block。而这个FlameGraph似乎没有什么问题。</p>
<p>不过继续尝试着减少Task的数量看看到底能不能提高性能也不是一件坏事。毕竟在这个程序中有几十万上百万个Task,改进说不定有些效果。
对于Formal Concept的Enumeration,可以通过限制并行的层数来减少Task——只有递归层数小于某一层时才能生成新的Task,否则就执行普通的单线程的列举。如果选择只对最上面一层进行Spawn,那么总的Task数跟Object的数量<code>noo</code>是一致的。如果对上面两层,那么Task的数量是<span class="math">\(\le noo \times (noo-1)\)</span>。根据这个想法,Spawn的<code>map</code>改一下:</p>
<div class="highlight"><pre><span></span><code><span class="p">.</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="n">concept</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">depth</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">max_depth</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">tokio</span>::<span class="n">task</span>::<span class="n">spawn</span><span class="p">(</span><span class="k">async</span><span class="w"> </span><span class="k">move</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">enumerate_fc_seq</span><span class="p">(</span><span class="n">concept</span><span class="p">,</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="n">should_print</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">tokio</span>::<span class="n">task</span>::<span class="n">spawn</span><span class="p">(</span><span class="k">async</span><span class="w"> </span><span class="k">move</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">enumerate_fc</span><span class="p">(</span><span class="n">concept</span><span class="p">,</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="n">should_print</span><span class="p">,</span><span class="w"> </span><span class="n">max_depth</span><span class="p">,</span><span class="w"> </span><span class="n">depth</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="k">await</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">})</span><span class="w"></span>
</code></pre></div>
<p>这里<code>enumerate_fc_seq</code>是一个不包含任何async代码的单线程版本,当depth大于设置的值时就停止生成新的task。
另外需要注意的是为了让类型匹配,对于单线程版本的分支也是通过<code>tokio::spawn</code>放到新线程里执行的——这导致了不管怎么调整参数,第一层总是并列的。为了让参数跟实际执行的结果看起来更匹配(即depth是0时应该是个单线程,而depth=1时只有第一层被并行),对于<span class="math">\(depth = 0\)</span>可以在<code>main</code>函数里单独执行一个单线程的版本,对于<span class="math">\(depth = N (N>0)\)</span>才用<span class="math">\(depth-1\)</span>作为参数来执行上面这个函数。下面的测试中实际上也是这么做的。</p>
<p>对于这个版本,可以得出<code>MaxDepth</code>和执行时间的关系:</p>
<p><img alt="Depth_time_relation" src="https://www.horsal.dev/images/2021-08-20-depth.png"></p>
<p>可以看到,<span class="math">\(depth=0\)</span>(及纯粹的单线程)以及<span class="math">\(depth=1\)</span>(只生成第一层的Task)的表现不太好,但在那之后无论如何增加depth都对结果没有太大影响。因此可以认为Tokio的Task不论如何使用,都不会对性能有着明显的伤害,反过来过分的限制Task反而会让Tokio的调度不够灵敏从而浪费CPU。</p>
<h2>Conclusion</h2>
<p>这篇文章实现了一个多线程版的Formal Concept Enumerator,在实现的过程中,Tokio的Task表现得比想象的要轻量的多——完全可以承受几十万甚至上百万的Task运行。不用考虑Task切换Overhead的好处是显而易见的,不再需要考虑限制线程数,只要把所有能够并行化的地方都扔给Tokio处理就足够了。</p>
<p>到这里,对于Formal Concept的实现就告一段落了,但实际上仍然有很多东西值得去做,比如使用MapReduce来生成Formal Concept<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>, 寻找更好的剪枝方法,甚至是用机器学习来生成Concept Lattice。
其实就算对于目前的内容来说,上一篇文章引用的论文<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>用了另外一种方法去限制Task的数量,验证它的效果也是一个值得去做的事情,不过这个工作就算是留给给读者的一个动手做吧。</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>Chunduri, Ragahvendra Kumar, Aswani Kumar Cherukuri, and Mike Tamir. "Concept generation in formal concept analysis using MapReduce framework." 2017 International Conference on Big Data Analytics and Computational Intelligence (ICBDAC). IEEE, 2017. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>Krajca, Petr, Jan Outrata, and Vilem Vychodil. "Parallel recursive algorithm for FCA." CLA. Vol. 2008. Citeseer, 2008. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Things Behind Apple's CSAM Detection2021-08-19T00:00:00+09:002021-08-19T00:00:00+09:00Horsaltag:www.horsal.dev,2021-08-19:/things-behind-apples-csam-detection.html<h2>TL;DR</h2>
<ul>
<li>NeuralHash is a MobileNetv3 whose last layer is replaced by Fully-connected.</li>
<li>NeuralHash convert the outputs of MobileNetv3 to hashes by Random Projection.</li>
<li>Transfer Protocol used "a two-layer encryption": Images are encrypted with a shared random key <span class="math">\(k_1\)</span>. <span class="math">\(k_1\)</span> is splitted into <span class="math">\(t\)</span> parts <span class="math">\(p_1, \dots, p_t\)</span> by Sharmir's …</li></ul><h2>TL;DR</h2>
<ul>
<li>NeuralHash is a MobileNetv3 whose last layer is replaced by Fully-connected.</li>
<li>NeuralHash convert the outputs of MobileNetv3 to hashes by Random Projection.</li>
<li>Transfer Protocol used "a two-layer encryption": Images are encrypted with a shared random key <span class="math">\(k_1\)</span>. <span class="math">\(k_1\)</span> is splitted into <span class="math">\(t\)</span> parts <span class="math">\(p_1, \dots, p_t\)</span> by Sharmir's Secret Sharing. Each <span class="math">\(p_i\)</span> is attatched to a matched image and further encrypted by using the hash of corresponding image as the key.</li>
<li>This protocol looks good, but NeuralHash may cause false positive.</li>
</ul>
<h2>Introduction</h2>
<p>苹果出人意料的宣布了一个保护儿童,以及限制涉及儿童色情信息传播的举措——<a href="https://www.apple.com/child-safety/">Child Safety</a>。苹果将会对用户的信息、图片、Siri以及搜索服务进行审查,找出其中可能的CSAM(Child Sexual Abuse Material)信息并与执法机关分享。显然苹果预料到了用户的不满,他们不但给出了<a href="https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf">一份面向普通用户的解说文档</a>,还<a href="https://www.apple.com/child-safety/pdf/Apple_PSI_System_Security_Protocol_and_Analysis.pdf">详细解释了这个上传信息的Protocol是如何工作的</a>,甚至还有一些<a href="https://www.apple.com/child-safety/pdf/Alternative_Security_Proof_of_Apple_PSI_System_Mihir_Bellare.pdf">苹果之外人员对这个Protocol的验证</a>。但这些显然不能平息用户的疑虑,对于这种行为是否真的安全,是否存在信息泄露,以及是否能够按照苹果设想的内容去工作,大多数用户都持怀疑的态度。在这篇文章里,我们将通过苹果给出的文档,来看一看这个功能背后是如何实现的。</p>
<h2>General Architecture</h2>
<p>虽然苹果在他们的<a href="https://www.apple.com/child-safety/pdf/CSAM_Detection_Technical_Summary.pdf">CSAM Detection Technical Summary</a>的P.4里给出了一个Overview,但它是有些晦涩难懂——里面充满了苹果自己的用词,以及反复冗长的说教式说明。因此我决定给出一个自己的版本:</p>
<p><img alt="Architecture" src="https://www.horsal.dev/images/2021-08-19-architecture.png"></p>
<p>整个流程可以分为三个部分:</p>
<ol>
<li>
<p><a href="https://www.missingkids.org/HOME">NCMEC</a>以及其他的儿童保护组织会向苹果提供图像数据库,苹果会用他们的<em>NeuralHash</em>根据<strong>图像特征</strong>计算Hash,最终得到一个大的Hash Table。</p>
<p>注意,这里的Hash并不是普通意义上的通过Binary计算得出的Hash,而是来自于图像特征——因此即使有人修改了图片里的几个Bit,或者加上少量的马赛克也不会影响计算结果。</p>
</li>
<li>
<p>上述的Hash Table会储存在用户的设备上。之后,用户的设备会扫描储存的照片,同时使用跟服务器同样的<em>NeuralHash</em>来计算Hash。这个Hash跟之前的Hash Table进行比对,相同的图片会进行加密之后上传到苹果的服务器。</p>
</li>
<li>
<p>苹果对上传的图片进行处理,这个处理分为两个部分:</p>
<ol>
<li>计算有多少图片跟数据库配对。</li>
<li>如果配对的图片大于一定数量,则对配对的图片进行解密,同时交给工作人员处理。如果没有达到这个数量,则不进行解密。</li>
</ol>
<p>一个普遍的担心是,就算苹果不解密数据,如果有人因为算法的问题错误配对了,那么苹果会不会把他告知给相关的团体?为了避免这个问题,苹果对配对数量加入了随机性——苹果自己也不知道具体有多少张配对了。因此只要数量不大于规定的值,苹果很难获得准确的信息。</p>
</li>
</ol>
<p>下面,让我们分别看一看这三个部分。</p>
<h2>NeuralHash</h2>
<p>虽然在开头我们提到,苹果为了平息用户的疑虑公开了很多技术细节,但这些内容都是针对<strong>如何向服务器分享数据</strong>的Protocol,对于识别图片的"NeuralHash",苹果却讳莫如深,只有在Technical Summary里提到了寥寥数笔:</p>
<blockquote>
<p>NeuralHash is a perceptual hashing function that maps images to numbers. Perceptual hashing bases this number on features of the image instead of the precise values of pixels in the image. The system computes these hashes by using an embedding network to produce image descriptors and then converting those descriptors to integers using a Hyperplane LSH (Locality Sensitivity Hashing) process. This process ensures that different images produce different hashes.</p>
</blockquote>
<p>这些内容并没有提供任何有用的信息——比如如何避免神经网络的误判,如何保证Hash对于Dataset能够具有最大的区分度。</p>
<p>不过,有人发现这个模型<a href="https://www.reddit.com/r/MachineLearning/comments/p6hsoh/p_appleneuralhash2onnx_reverseengineered_apple/">早在iOS14.3就已经存在了</a>,不但如此还能够<a href="https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX">实际的执行这个模型</a>,通过目前的模型,可以看到目前的NeuralHash是这样的:</p>
<p><img alt="NeuralHash" src="https://www.horsal.dev/images/2021-08-19-neuralhash.png"></p>
<ol>
<li>首先图片会变换成360x360的RGB格式</li>
<li>变换过的图片通过MobileNetV3<sup id="fnref:5"><a class="footnote-ref" href="#fn:5">1</a></sup>学习特征量,由于这里不是分类问题,所以最后一层并不是Softmax,而是一个Fully-connected。最终会变换成一个128维的输出。</li>
<li>
<p>通过一个预先给定的Seed Matrix <span class="math">\(S\)</span>,进行线性Projection <span class="math">\(v = SX\)</span>,变换成96维的向量之后进行二值化。将所有的bit整合起来就成了最终结果。所以最终一个图片会被变换成<span class="math">\(\frac{96}{8}=12\)</span> bytes长的Hash。</p>
<p>这个Seed Matrix是如何计算的并没有任何说明,但考虑到这个简单的形式,看起来很像是SimHash(Random Projection)<sup id="fnref:6"><a class="footnote-ref" href="#fn:6">2</a></sup>,也就是<span class="math">\(S\)</span>是一个随机生成的矩阵,我们把元数据映射到随机的轴上得到Hash。为了验证这个想法,我们把Seed的每个Weight统计一下:</p>
<p><img alt="SeedWeights" src="https://www.horsal.dev/images/2021-08-19-weight.png"></p>
<p>横轴是Weight的大小,纵轴是每个区间里包含的Weight的数量,可以看到Weight是很漂亮的高斯分布,因此可以认为这个算法来自于Fast LSH<sup id="fnref:7"><a class="footnote-ref" href="#fn:7">3</a></sup>。</p>
</li>
</ol>
<p>总结起来,这个Hash可以写成下面的形式:</p>
<p><span class="math">\(Hash(x) = [S\times MobileNetv3(x)], where S \sim N(0,\sigma) \in R^{96 \times 128}\)</span></p>
<p>实际上,尝试CNN + Locality Sensitive Hashing的论文并不是新鲜的东西,类似的论文比如ICCV’17的这一篇论文<sup id="fnref:8"><a class="footnote-ref" href="#fn:8">6</a></sup>,它的结构是这样的:</p>
<p><img alt="HashNet,ICCV17" src="https://www.horsal.dev/images/2021-08-19-iccv17.png"></p>
<p><a href="https://www.reddit.com/r/MachineLearning/comments/p6hsoh/p_appleneuralhash2onnx_reverseengineered_apple/">Reddit上对于NeuralHash模型的初步测试</a>也显示出这个模型对于缩放,压缩有一定的鲁棒性,但并不能对应旋转和剪切,这跟CNN的特征是一致的。</p>
<p>但考虑到目前发现的模型只是一个古老的版本,Apple完全可以引入Attention<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">4</a></sup>来应对图片剪切,以及引入Capsule Network<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">5</a></sup>来应对旋转,纵使手机芯片并不能完整的这些算法,我也不认为最终出现在成品里的算法会像现在这么脆弱。如果你打算把图片都旋转45°的话那还是放弃为好。</p>
<p>总体来说,这个NeuralHash依然是一个普通的Neural Network,LSH算法也没有做特别的优化,它们都并不能保证Hashing一定是对的,不过对于这种只需要对比已知内容的工作,通过过学习来抑制False Positive似乎也不是一个坏主意。并且,苹果还留出了犯错的空间——要有多枚图片同时配对之后才会交给人工处理。</p>
<h2>Communication Protocol</h2>
<p>在计算完所有图片之后,下一步就是交给用户设备去比对了,而苹果在这个比对过程中尽可能的保证不获取范围之外的信息。对于这个Protocol苹果还给出了<a href="https://www.apple.com/child-safety/pdf/Apple_PSI_System_Security_Protocol_and_Analysis.pdf">一份36页的说明文档(或者说论文)</a>,可以看出相比于NeuralHash,他们对这个Protocol更有信心。当然了,这个Protocol也确实没有问题。让我们忽略各种细节,粗枝大叶的看上去这个Protocol是这样的:</p>
<p><img alt="Protocol" src="https://www.horsal.dev/images/2021-08-19-protocol.png"></p>
<ol>
<li>
<p>首先,在本地配对的图片会用一个随机生成的Key1来加密,与此同时,设备还会生成一些"假的数据",这些数据在加密后看起来就像是跟真的配对了一样。这样最终传到服务器上之后能够保证苹果无法准确的知道用户手上有多少有问题的图片。</p>
</li>
<li>
<p>之后,这个用来加密的Key1会被分割成<span class="math">\(t\)</span>个Partial Key,通过这<span class="math">\(t\)</span>个Partial Key服务器可以反推出Key1。每一副<strong>真正配对的图片</strong>都会带有一个Partial Key,因此,如果真正配对的图片少于<span class="math">\(t\)</span>个,服务器就无法得到全部的Partial Key也就没法得到Key1。这样就保证了图片数据是安全的。</p>
<p>实际上,在看到苹果官网对这一部分的描述时,我就立马联想到了<code>Shamir's Secret Sharing</code><sup id="fnref:3"><a class="footnote-ref" href="#fn:3">7</a></sup>,论文里也确实用了这个方法。Shamir Secret Share的思想非常简单:</p>
<p>我们知道加密用的Key通常是一个巨大的数字,假设是<span class="math">\(k\)</span>, 如果想把它分割成<span class="math">\(t\)</span>个部分,就随机选取<span class="math">\(t\)</span>个数字<span class="math">\(c_1,\dots,c_t\)</span>,这样我们可以构建一个多项式:</p>
<p><span class="math">\(F(x) = k + c_1 x + c_2 x^2 + \dots + c_t x^t\)</span></p>
<p>我们知道,一个<span class="math">\(t\)</span>阶多项式需要<span class="math">\(t\)</span>组解联立才能求出系数,因此,我们取下面一组数字对作为新的Key:</p>
<p><span class="math">\((1, F(1)), (2, F(2)), \dots, (t, F(t)), \dots\)</span></p>
<p>只有服务器知道至少<span class="math">\(t\)</span>组解,才能求出系数,这就做到了安全的把Key分割成多个部分。实际上,这种方法在Google的Federated Learning加密里面也有应用<sup id="fnref:4"><a class="footnote-ref" href="#fn:4">8</a></sup>。</p>
</li>
<li>
<p>之后,对于每个加密过的图像,都有一个附带的Partial Key。根据这个图像所配对的NeuralHash,设备会计算一个Key2,对这个Partial Key进行一次加密。而这个Key2并不会传给服务器——服务器通过自己手里的NeuralHash来计算这个Key2从而进行解密。这样就保证了服务器无法解密Hash Table之外的内容。值得一提的是,苹果采用了一种很有趣的办法来确认数据是否存在于数据库里,如果你对具体算法感兴趣,可以阅读<a href="https://www.apple.com/child-safety/pdf/Apple_PSI_System_Security_Protocol_and_Analysis.pdf">他们的论文</a>。</p>
</li>
</ol>
<p>这就是苹果所说的2次加密的整体流程,对于有问题图片的用户,随机加入的假数据一定程度上模糊了图片的数量,而为了保证图片不被随意看到,苹果采用了Shamir Secret Sharing保证解密时的最小数据数量,而为了保证"清白的"用户,Shamir Secret Sharing得到的Key也只能对于配对的数据才能解密,考虑到网络问题,就算解密了工作人员也只能得到低分辨率的图片。这个Protocol至少看上去是没什么问题的,或者说——NeuralHash看起来更可疑一些。</p>
<h2>Random Thoughts</h2>
<ul>
<li>显然从NeuralHash的结果里无法还原图片,但这并不的代表无法还原"低画质"的图片,用户的手里有NeuralHash的神经网络,也有LSH的矩阵,那么把NeuralHash作为一个discriminator,自己实现一个generator的话,自然可以近似的还原图像。好在苹果已经考虑到了这一点,传送到用户手里的数据库经过了一些变换,用户不知道变换的细节,因此这些Hash无法拿来作为GAN的判定器。</li>
<li>
<p>NeuralHash的False Positive和False Negative都是个让人担心的问题。神经网络不确定性很多来自于分类层,针对分类的攻击并只需要让几个Feature变得足够强大,就能够压倒别的Feature从而让分类器出现误判。苹果通过Hash比对一定程度上抑制了这个问题,Hash来自于所有Feature的线性组合,而这些Feature没有Weight,都被量化成了0和1,这就意味着一两个突出的Feature并不会影响最终结果,除非有人能够构建出所有的Feature都一致的图像。</p>
<p>反过来说,过于严格的判定显然会导致False Negative。因此比对Hash可以放宽到一定的<a href="https://en.wikipedia.org/wiki/Hamming_distance">汉明距离</a>,即有<span class="math">\(k\)</span>以上的Feature一致则判定为符合。而汉明距离设定为多少,需要对iCloud上的图片进行实际统计才能得出结果吧。</p>
</li>
</ul>
<h2>Conclusion</h2>
<p>到这里,我们对苹果的这个CSAM Detection已经有了大致的理解,实际上,技术并不能保证运用中的问题——比如,NeuralHash会不会对亚洲人存在歧视?苹果的客户端没有按照要求把Key分割成<span class="math">\(t\)</span>个呢?或者未来受到压力苹果把iCloud的内容直接交给工作人员审核呢?又或者,这个技术在不知不觉中变成了检测政治内容的工具呢?苹果保证了被监测的人员不会收到任何反馈——那么用户又怎么保证这个数据库只会用来识别CSAM呢?</p>
<p>不过考虑到这篇博客的目的并非批判或者维护苹果的决策,这些问题还是交给秋季发布会之后用户的反应去回答吧。</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:5">
<p>Howard, Andrew, et al. "Searching for mobilenetv3." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019. <a class="footnote-backref" href="#fnref:5" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:6">
<p>Sadowski, Caitlin, and Greg Levin. "Simhash: Hash-based similarity detection." Technical report, Google (2007). <a class="footnote-backref" href="#fnref:6" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:7">
<p>Dasgupta, Anirban, Ravi Kumar, and Tamás Sarlós. "Fast locality-sensitive hashing." Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. 2011. <a class="footnote-backref" href="#fnref:7" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:1">
<p>Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems. 2017. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>Sabour, Sara, Nicholas Frosst, and Geoffrey E. Hinton. "Dynamic routing between capsules." Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:8">
<p>Cao, Zhangjie, et al. "Hashnet: Deep learning to hash by continuation." Proceedings of the IEEE international conference on computer vision. 2017. <a class="footnote-backref" href="#fnref:8" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
<li id="fn:3">
<p>Shamir A. How to share a secret[J]. Communications of the ACM, 1979, 22(11): 612-613. <a class="footnote-backref" href="#fnref:3" title="Jump back to footnote 7 in the text">↩</a></p>
</li>
<li id="fn:4">
<p>Bonawitz, Keith, et al. "Practical secure aggregation for privacy-preserving machine learning." proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 2017. <a class="footnote-backref" href="#fnref:4" title="Jump back to footnote 8 in the text">↩</a></p>
</li>
</ol>
</div>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Formal Concept Analysis with Rust (2) - Basic Algorithm2021-08-16T00:00:00+09:002021-08-16T00:00:00+09:00Horsaltag:www.horsal.dev,2021-08-16:/formal-concept-analysis-with-rust-2-basic-algorithm.html<p>在<a href="https://www.horsal.dev/formal-concept-analysis-with-rust-1-introduction.html">上一篇文章里</a>, 我们介绍了Formal Concept的基本概念,概括起来就是对于一个<em>文章-单词</em>的集合<span class="math">\(D=\{d_1, d_2, \dots, d_n\}, W=\{w_1, w_2, \dots, w_m\}\)</span>,有:</p>
<ul>
<li>对于任何一个文章的集合<span class="math">\(D_n = \{d_{i_1}, \dots, d_{i_2}\}\)</span>, 我们总可以用Closure操作把它变成一个合理的,极大的Formal Concept。即:对于<span class="math">\(D_n\)</span>我们先寻找这些文章的共通单词的集合<span class="math">\(D_n^{\downarrow}\)</span>, 再通过这个单词的集合去寻找包含这些单词的文章集合<span class="math">\({D_n^{\downarrow\uparrow}}\)</span>,最终得到的文章-单词组合<span class="math">\(\{D_n^{\downarrow\uparrow}, D_n^{\downarrow}\}\)</span>就是一个极大的Formal Concept。这个寻找共通单词,共通文章的操作就是Closure。在本文里,我们会把寻找共同单词的操作写成 …</li></ul><p>在<a href="https://www.horsal.dev/formal-concept-analysis-with-rust-1-introduction.html">上一篇文章里</a>, 我们介绍了Formal Concept的基本概念,概括起来就是对于一个<em>文章-单词</em>的集合<span class="math">\(D=\{d_1, d_2, \dots, d_n\}, W=\{w_1, w_2, \dots, w_m\}\)</span>,有:</p>
<ul>
<li>对于任何一个文章的集合<span class="math">\(D_n = \{d_{i_1}, \dots, d_{i_2}\}\)</span>, 我们总可以用Closure操作把它变成一个合理的,极大的Formal Concept。即:对于<span class="math">\(D_n\)</span>我们先寻找这些文章的共通单词的集合<span class="math">\(D_n^{\downarrow}\)</span>, 再通过这个单词的集合去寻找包含这些单词的文章集合<span class="math">\({D_n^{\downarrow\uparrow}}\)</span>,最终得到的文章-单词组合<span class="math">\(\{D_n^{\downarrow\uparrow}, D_n^{\downarrow}\}\)</span>就是一个极大的Formal Concept。这个寻找共通单词,共通文章的操作就是Closure。在本文里,我们会把寻找共同单词的操作写成<span class="math">\(\downarrow\)</span>, 寻找共通文章则是<span class="math">\(\uparrow\)</span>, 因此一个Closure操作就是<span class="math">\(\downarrow\uparrow\)</span>.</li>
<li>根据上面的特性,我们显然从一个空集合<span class="math">\(\emptyset\)</span>开始,向里面不停的添加文章,然后用Closure计算每次的极大Formal Concept,就可以最终得到所有的Formal Concept。 但这样得到的Formal Concept显然是会重复的,因此我们还要去掉重复生成的部分。显然,采用深度探索单纯的计算全部之后去重是一个最直接但最低效的做法。</li>
</ul>
<p>在这一篇文章里,我们会根据</p>
<blockquote>
<p>Parallel Recursive Algorithm for FCA, Petr Krajca, Jan Outrata and Vilem Vychodil, The International Conference on Concept Lattices and Their Applications, 2008.</p>
</blockquote>
<p>里面的描述,实现一个单线程的Formal Concept列举算法。</p>
<h2>Basic Representation</h2>
<p>在实现算法之前,让我们先来考虑如何表示一个Formal Concept。从上面的总结里就可以看出,Closure算法需要大量的集合操作,因此我们选择使用一个数字来表示一篇文章或者一个单词,而一个<a href="https://docs.rs/bit-vec/0.6.3/bit_vec/"><em>Bit Vector</em></a>则能够表示一个文章、单词的集合。用Rust的代码来说,定义是这样的:</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span> <span class="nc">FormalConcept</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">extent</span>: <span class="nc">BitVec</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">intent</span>: <span class="nc">BitVec</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">FormalConceptLattice</span><span class="o"><</span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="n">K</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">object_map</span>: <span class="nc">HashMap</span><span class="o"><</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">T</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">attribute_map</span>: <span class="nc">HashMap</span><span class="o"><</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">K</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">concepts</span>: <span class="nb">Vec</span><span class="o"><</span><span class="n">FormalConcept</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">Relation</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">row</span>: <span class="nc">HashMap</span><span class="o"><</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">BitVec</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">col</span>: <span class="nc">HashMap</span><span class="o"><</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">BitVec</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这里,一个<code>FormalConcept</code>包含了<code>extent</code>(文章的集合)以及一个intent(单词的集合),这些文章和单词均用数字来表示,而它们实际的内容则保存在<code>FormalConceptLattice</code>的<code>object_map</code>和<code>attribute_map</code>里,在列举过程中我们并不需要实际的内容,因此这两个Map仅仅在输出的时候有用处。</p>
<p>在上一篇文章里我们已经提到,一个记录文章-单词关系的表通常是这样的</p>
<table>
<thead>
<tr>
<th>doc\word</th>
<th>word1</th>
<th>word2</th>
<th>word3</th>
<th>word 4</th>
<th>...</th>
</tr>
</thead>
<tbody>
<tr>
<td>doc1</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>0</td>
<td>...</td>
</tr>
<tr>
<td>doc2</td>
<td>1</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>...</td>
</tr>
<tr>
<td>doc3</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>1</td>
<td>...</td>
</tr>
<tr>
<td>doc4</td>
<td>1</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>...</td>
</tr>
<tr>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
<td>..</td>
</tr>
</tbody>
</table>
<p><code>Relation</code>就是用来表示输入内容的,我们把每一行(文章)和每一列(单词)的01关系都单独记录下来,这样在计算过程中可以简单的使用它们而不必重复计算。</p>
<p>在Formal Concept的论文里,<a href="https://archive.ics.uci.edu/ml/datasets/Tic-Tac-Toe+Endgame">Tic-Tac-Toe</a>,<a href="https://archive.ics.uci.edu/ml/datasets/mushroom">Mushroom</a>等是常用的Dataset,不过在这篇文章里,我们使用下面的代码来生成随机的关系表作为数据:</p>
<div class="highlight"><pre><span></span><code><span class="c1">// number of objects</span>
<span class="kd">let</span><span class="w"> </span><span class="n">noo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">500</span><span class="p">;</span><span class="w"></span>
<span class="c1">// number of attributes</span>
<span class="kd">let</span><span class="w"> </span><span class="n">noa</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">50</span><span class="w"></span>
<span class="c1">// density</span>
<span class="kd">let</span><span class="w"> </span><span class="n">p</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">0.1</span><span class="w"></span>
<span class="fm">println!</span><span class="p">(</span><span class="s">"{} {} {}"</span><span class="p">,</span><span class="w"> </span><span class="n">noo</span><span class="p">,</span><span class="w"> </span><span class="n">noa</span><span class="p">,</span><span class="w"> </span><span class="n">p</span><span class="p">);</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">rng</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">rand</span>::<span class="n">thread_rng</span><span class="p">();</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="n">noo</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">print!</span><span class="p">(</span><span class="s">"{} "</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="n">noa</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="fm">print!</span><span class="p">(</span><span class="s">"{} "</span><span class="p">,</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">rng</span><span class="p">.</span><span class="n">gen_range</span><span class="p">(</span><span class="mi">0</span><span class="k">f32</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="k">f32</span><span class="p">)</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">p</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">);</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">""</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这里<span class="math">\(p\)</span>是用来指定关系的密度,<span class="math">\(p\)</span>越大,每个Object包含的Attribute就越多,当然Concept的数量也就越多,这个数量通常是爆发式增长的,如果<span class="math">\(p=1.0\)</span>那么总是一个全列举问题,因此所有的Formal Concept列举算法的效率都会变得一样。</p>
<h2>Closure Operation</h2>
<p>下面我们先来实现Formal Concept里最重要的Closure操作:</p>
<div class="highlight"><pre><span></span><code><span class="c1">// calculate extent -> intent operator</span>
<span class="c1">// input: a set of object</span>
<span class="c1">// output: a set of attributes which all objects share</span>
<span class="c1">// operation: calculate the intersection of attributes for all objects</span>
<span class="k">fn</span> <span class="nf">down</span><span class="p">(</span><span class="n">r</span>: <span class="kp">&</span><span class="nc">Relation</span><span class="p">,</span><span class="w"> </span><span class="n">extent</span>: <span class="kp">&</span><span class="nc">BitVec</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">BitVec</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">assert_eq!</span><span class="p">(</span><span class="n">extent</span><span class="p">.</span><span class="n">len</span><span class="p">(),</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="n">extent</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">iter</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">enumerate</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">fold</span><span class="p">(</span><span class="n">BitVec</span>::<span class="n">from_elem</span><span class="p">(</span><span class="n">r</span><span class="p">.</span><span class="n">col</span><span class="p">.</span><span class="n">len</span><span class="p">(),</span><span class="w"> </span><span class="kc">true</span><span class="p">),</span><span class="w"> </span><span class="o">|</span><span class="k">mut</span><span class="w"> </span><span class="n">f0</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">flag</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">flag</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">f0</span><span class="p">.</span><span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">[</span><span class="o">&</span><span class="n">i</span><span class="p">]);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="n">f0</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="c1">// calculate intent -> extent operator</span>
<span class="c1">// input: a set of attributes </span>
<span class="c1">// output: a set of objects containing all input attributes</span>
<span class="c1">// operation: calculate the intersection of objects for all attributes</span>
<span class="k">fn</span> <span class="nf">up</span><span class="p">(</span><span class="n">r</span>: <span class="kp">&</span><span class="nc">Relation</span><span class="p">,</span><span class="w"> </span><span class="n">intent</span>: <span class="kp">&</span><span class="nc">BitVec</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">BitVec</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">assert_eq!</span><span class="p">(</span><span class="n">intent</span><span class="p">.</span><span class="n">len</span><span class="p">(),</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">col</span><span class="p">.</span><span class="n">len</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="n">intent</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">iter</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">enumerate</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">fold</span><span class="p">(</span><span class="n">BitVec</span>::<span class="n">from_elem</span><span class="p">(</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">(),</span><span class="w"> </span><span class="kc">true</span><span class="p">),</span><span class="w"> </span><span class="o">|</span><span class="k">mut</span><span class="w"> </span><span class="n">f0</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">flag</span><span class="p">)</span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">flag</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">f0</span><span class="p">.</span><span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">r</span><span class="p">.</span><span class="n">col</span><span class="p">[</span><span class="o">&</span><span class="n">i</span><span class="p">]);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="n">f0</span><span class="w"></span>
<span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">closure</span><span class="p">(</span><span class="n">r</span>: <span class="kp">&</span><span class="nc">Relation</span><span class="p">,</span><span class="w"> </span><span class="n">extent</span>: <span class="kp">&</span><span class="nc">BitVec</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">FormalConcept</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">attributes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">down</span><span class="p">(</span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="n">extent</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">objects</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">up</span><span class="p">(</span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">attributes</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">FormalConcept</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">extent</span>: <span class="nc">objects</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">intent</span>: <span class="nc">attributes</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这里首先实现了之前提到的<span class="math">\(\uparrow\)</span>(up)和<span class="math">\(\downarrow\)</span>(down)操作。对于up操作,给定一组Object,由于<code>Relation</code>里已经包含了每一行的信息,它们共通的单词显然就是全部集合的Intersection操作。这个操作每一次Closure操作都会发生数次,而选用BitVector正是因为这个——通常一个普通的集合,比如<code>collections::HashSet</code>都是用数列直接记录已经存在的数字,进行Intersection时对这些数字进行比较的算法总是<span class="math">\(O(min(|Self|, |Other|))\)</span>的,当集合包含很多元素时,这个操作很耗时间。BitVector总是用Bit的位置来表示数字是否存在,因此对于总元素数为<span class="math">\(N\)</span>的集合,它的长度是<span class="math">\(N/8\)</span> bytes。更重要的是,BitVector之间的集合运算并不需要比较/删除操作,而是简单的比特运算就足够了—— Intersection是and,Union则是or。这能够极大的提高Closure的运算速度。</p>
<p>对于下面的例子:</p>
<table>
<thead>
<tr>
<th>example data</th>
<th>att1</th>
<th>att2</th>
<th>att3</th>
<th>att4</th>
<th>att5</th>
<th>att6</th>
<th>att7</th>
</tr>
</thead>
<tbody>
<tr>
<td>obj1</td>
<td>1</td>
<td>0</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>obj2</td>
<td>1</td>
<td>1</td>
<td>1</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>obj3</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>obj4</td>
<td>0</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>1</td>
<td>0</td>
<td>1</td>
</tr>
</tbody>
</table>
<p>我们对Closure给一个输入 <span class="math">\(Intend = \{obj1\}\)</span>,</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">initial_extent</span><span class="o">=</span><span class="w"> </span><span class="n">BitVec</span>::<span class="n">from_elem</span><span class="p">(</span><span class="mi">4</span><span class="p">,</span><span class="w"> </span><span class="kc">false</span><span class="p">);</span><span class="w"></span>
<span class="n">initial_extent</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="kc">true</span><span class="p">);</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">test_concept</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">closure</span><span class="p">(</span><span class="o">&</span><span class="n">relation</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">initial_extent</span><span class="p">);</span><span class="w"></span>
<span class="fm">println!</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">test_concept</span><span class="p">)</span><span class="w"></span>
</code></pre></div>
<p>可以得到下面的结果:</p>
<div class="highlight"><pre><span></span><code><span class="n">FormalConcept</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">extent</span>: <span class="mi">1100</span><span class="p">,</span><span class="w"> </span><span class="n">intent</span>: <span class="mi">1010000</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>对于<code>obj1</code>,我们可以找的attribute的集合为<span class="math">\(\{att1, att2\}\)</span>, 根据这个集合可以得到一组object为<span class="math">\(\{obj1, obj2\}\)</span>,所以上面的计算给出了正确的结果。</p>
<h2>Formal Concept Enumeration</h2>
<p>下面我们就可以用Closure来进行Formal Concept的枚举了。我们首先把intent的集合设为包含所有的attributes <span class="math">\(B=\{w_1, \dots, w_m\}\)</span>,然后从<span class="math">\(\{B^{\uparrow},B^{\uparrow\downarrow}\}\)</span>开始列举,每一次我们都向extent里面添加一个新的object。(事实上,intent和extent互换并不影响结果,因此不论从空集还是全集出发都能得到Formal Concept Lattice)</p>
<p>这个算法是这样的:</p>
<div class="highlight"><pre><span></span><code><span class="nv">Function</span> <span class="nv">Enumerate</span><span class="ss">(</span><span class="nv">Concept</span> {<span class="nv">A</span>, <span class="nv">B</span>}, <span class="nv">current</span> <span class="nv">index</span> <span class="nv">j</span><span class="ss">)</span>:
<span class="mi">1</span>. <span class="k">if</span> <span class="nv">B</span> <span class="nv">is</span> <span class="nv">already</span> <span class="nv">maximal</span>, <span class="k">exit</span>
<span class="mi">2</span>. <span class="k">for</span> <span class="nv">i</span> <span class="nv">in</span> <span class="nv">j</span> .. <span class="nv">number</span> <span class="nv">of</span> <span class="nv">object</span>
<span class="mi">2</span>.<span class="mi">1</span> <span class="k">if</span> <span class="nv">B</span> <span class="nv">already</span> <span class="nv">contains</span> <span class="nv">object</span> <span class="nv">i</span>, <span class="k">continue</span>
<span class="mi">2</span>.<span class="mi">2</span> <span class="nv">New</span> <span class="nv">Concept</span> {<span class="nv">C</span>, <span class="nv">D</span>} <span class="o">=</span> <span class="nv">Closure</span><span class="ss">(</span><span class="nv">Concept</span> {<span class="nv">A</span>, <span class="nv">B</span>}<span class="ss">)</span>
<span class="mi">2</span>.<span class="mi">3</span> <span class="k">if</span> <span class="nv">D</span>[<span class="mi">0</span> .. <span class="nv">i</span>] <span class="o">!=</span> <span class="nv">B</span>[<span class="mi">0</span> .. <span class="nv">i</span>], <span class="k">continue</span>
<span class="mi">2</span>.<span class="mi">4</span> <span class="nv">Run</span> <span class="nv">Enumerate</span><span class="ss">(</span><span class="nv">Concept</span>{<span class="nv">C</span>, <span class="nv">D</span>}, <span class="nv">i</span><span class="o">+</span><span class="mi">1</span><span class="ss">)</span>
</code></pre></div>
<p>用图表示出来就是这样:</p>
<p><img alt="Algorithm" src="https://www.horsal.dev/images/2021-08-16-algorithm.png"></p>
<p>首先让我们来看一看基本的Depth-first Search的算法:</p>
<p>每一步中,给定一个Concept,和已经添加过的extent的index j,我们对当前Concept依次添加<span class="math">\(d_j, d_{j+1}, \dots\)</span>,然后对每一个Concept进行递归。</p>
<p>具体来说,在Extent是<span class="math">\(\emptyset\)</span>时,我们首先每次添加一个Extent,比如第一次添加的是<span class="math">\(d_1\)</span>。之后我们可以对这个添加之后的Extent进行closure操作来计算真正的Concept,在这里我们得到了<span class="math">\(\{(w_1, w_3), (d_1,d_2)\}\)</span>,可以看到在计算Concept之后,这个Extent并不是我们最初给定的那个Extent了。对于这个Extent,由于这是我们添加<span class="math">\(d_1\)</span>之后的来的结果,它必定包含<span class="math">\(d_1\)</span>。(对于第<span class="math">\(n\)</span>此操作来说,由于我们之前添加了<span class="math">\(d_1 \cdots d_n\)</span>,所以它必定包含这些extent)因此,对于这个新的Concept,我们从<span class="math">\(d_2\)</span>开始添加,不过由于<span class="math">\(d_2\)</span>已经自然生成了,因此这里我们跳过它,添加<span class="math">\(d_3\)</span>。而添加之后得到的结果是<span class="math">\(\{(\emptyset),(d_1,d_2,d_3,d_4)\}\)</span>,即已经最大了,所以可以停止这个分支。</p>
<p>而对于添加<span class="math">\(d_2\)</span>的分支,可以看到我们再次添加<span class="math">\(d_3\)</span>时,同样得到了一个<span class="math">\(\{(\emptyset),(d_1,d_2,d_3,d_4)\}\)</span>,但这里的结果跟之前时重复的,因此不应该输出。判断重复的方法则很简答:</p>
<blockquote>
<p>对于一个新生成的Concept,它只能引入我们刚刚添加的extent之后的extent。如果它引入了一个之前的extent,我们显然已经在前面的分支里考虑过了,因此这个结果一定时重复的。</p>
</blockquote>
<p>因此,对于这个<span class="math">\(d_2, d_3\)</span>的分支,它计算之后多出了一个<span class="math">\(d_1\)</span>,但包含<span class="math">\(d_1\)</span>的情况在我们添加<span class="math">\(d_1\)</span>的分支里已经处理过了,因此这个结果总是重复的。</p>
<p>需要注意的是,这篇文章里我们用的是一个简化版的算法,这个算法并不保证Concept的顺序,如果你需要保证生成顺序,那么可以参考下面的文章:</p>
<blockquote>
<p>Kuznetsov S.: Learning of Simple Conceptual Graphs from Positive and Negative Examples. PKDD 1999, pp. 384–391.</p>
</blockquote>
<p>理解了上面的内容之后,我们就能把它写成代码了:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">enumerate_fc</span><span class="p">(</span><span class="n">cur_concept</span>: <span class="kp">&</span><span class="nc">FormalConcept</span><span class="p">,</span><span class="w"> </span><span class="n">cur_obj</span>: <span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">r</span>: <span class="kp">&</span><span class="nc">Relation</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">all</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">cur_obj</span><span class="o">..</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">!</span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">obj</span><span class="p">).</span><span class="n">unwrap_or</span><span class="p">(</span><span class="kc">false</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">extent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">extent</span><span class="p">.</span><span class="n">set</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span><span class="w"> </span><span class="kc">true</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">concept</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">closure</span><span class="p">(</span><span class="n">r</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">extent</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// mask: 11111111111111111 00000000000</span>
<span class="w"> </span><span class="c1">// ----------------- -----------</span>
<span class="w"> </span><span class="c1">// 0 ... cur_obj-1 cur_obj ...</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">mask</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BitVec</span>::<span class="n">from_fn</span><span class="p">(</span><span class="n">r</span><span class="p">.</span><span class="n">row</span><span class="p">.</span><span class="n">len</span><span class="p">(),</span><span class="w"> </span><span class="o">|</span><span class="n">i</span><span class="o">|</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kc">true</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="kc">false</span><span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">cur_left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">cur_concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">cur_left</span><span class="p">.</span><span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">mask</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">new_left</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">concept</span><span class="p">.</span><span class="n">extent</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">new_left</span><span class="p">.</span><span class="n">and</span><span class="p">(</span><span class="o">&</span><span class="n">mask</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">cur_left</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">new_left</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// new concept generated</span>
<span class="w"> </span><span class="n">enumerate_fc</span><span class="p">(</span><span class="o">&</span><span class="n">concept</span><span class="p">,</span><span class="w"> </span><span class="n">obj</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">r</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>上面的代码中生成了一个Mask,因为算法要求比较前N个数字的集合是否相同,而这次使用的BitVector库<a href="https://docs.rs/bit-vec/0.6.3/bit_vec/">bit-vec</a>并没有提供这个功能,才有了这个做法,但这么做实际上平白无故的多做了一次and。显然,理想的做法是直接比较BitVector的前N位——比如内部是用<code>u8</code>来储存的话,只需要比较前<code>(cur_obj-1)/8</code>byte以及<code>(cur_obj-1)%8</code>的Bit就足够了。</p>
<p>让我们初始化一个Concept,然后对上面的测试例子执行一下:</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">initial_intent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">BitVec</span>::<span class="n">from_elem</span><span class="p">(</span><span class="n">relation</span><span class="p">.</span><span class="n">col</span><span class="p">.</span><span class="n">len</span><span class="p">(),</span><span class="w"> </span><span class="kc">true</span><span class="p">);</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">initial_extent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">up</span><span class="p">(</span><span class="o">&</span><span class="n">relation</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">initial_intent</span><span class="p">);</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">initial_concept</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">FormalConcept</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">extent</span>: <span class="nc">initial_extent</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">intent</span>: <span class="nc">initial_intent</span><span class="p">,</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
<span class="n">enumerate_fc</span><span class="p">(</span><span class="o">&</span><span class="n">initial_concept</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">relation</span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>可以得到:</p>
<div class="highlight"><pre><span></span><code><span class="n">read</span> <span class="n">data</span><span class="p">:</span> <span class="mf">134.3</span><span class="err">µ</span><span class="n">s</span>
<span class="n">construct</span> <span class="n">initial</span> <span class="n">concept</span><span class="p">:</span> <span class="mf">239.1</span><span class="err">µ</span><span class="n">s</span>
<span class="n">enumerate</span><span class="p">:</span> <span class="mf">167.5</span><span class="err">µ</span><span class="n">s</span>
<span class="mi">0000</span> <span class="mi">1111111</span>
<span class="mi">1100</span> <span class="mi">1010000</span>
<span class="mi">1111</span> <span class="mi">0000000</span>
<span class="mi">0100</span> <span class="mi">1111000</span>
<span class="mi">0101</span> <span class="mi">0100000</span>
<span class="mi">0010</span> <span class="mi">0000011</span>
<span class="mi">0011</span> <span class="mi">0000001</span>
<span class="mi">0001</span> <span class="mi">0100101</span>
</code></pre></div>
<p>可以看到,我们得到了所有的Formal Concept。
不过, 这个简单的算法的速度并不是很理想,比如对于<code>3000 object, 50 attributes</code>, <span class="math">\(p=0.1\)</span>的数据来说,它的速度是:(release编译,不输出concept的内容)</p>
<div class="highlight"><pre><span></span><code><span class="n">read</span> <span class="n">data</span><span class="p">:</span> <span class="mf">20.7916</span><span class="n">ms</span>
<span class="n">construct</span> <span class="n">initial</span> <span class="n">concept</span><span class="p">:</span> <span class="mf">216.2</span><span class="err">µ</span><span class="n">s</span>
<span class="n">enumerate</span><span class="p">:</span> <span class="mf">457.6723555</span><span class="n">s</span>
</code></pre></div>
<p>实际上,同样<code>50 attributes</code>, <span class="math">\(p=0.1\)</span>的情况下,执行时间和object数量的关系:</p>
<p><img alt="executiontime" src="https://www.horsal.dev/images/2021-08-16-figure.png"></p>
<p>可以看到时间是指数式增长的,在<code>5000 object</code>的情况下,已经需要数十分钟来执行了。</p>
<h2>Conclusion</h2>
<p>在这篇文章里,我们简单描述了如何去计算一个Formal Concept Lattice。不过,由于算法并没有进行特别聪明的剪枝,速度上面并没有想象的那么理想。在下一篇文章里,我们将会对上面的算法并行化,提高它的执行速度。</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Write a Niconico Video Player, in Rust2021-08-15T00:00:00+09:002021-08-15T00:00:00+09:00Horsaltag:www.horsal.dev,2021-08-15:/write-a-niconico-video-player-in-rust.html<h2>Introduction</h2>
<p>Niconicoのフローは以下のような感じ:</p>
<p><img alt="Overview" src="https://www.horsal.dev/images/2021-08-15-overview.png"></p>
<ol>
<li>ログイン時取得したCookieを使って、<code>https://www.nicovideo.jp/watch/so*</code>をGetする。取得したページに<code>data-api-data</code>というJsonデータが埋められている。</li>
<li>上記の<code>data-api-data</code>を整備して、ほしい映像の解像度情報や認証情報をサーバに送る。するとそれに応じて映像のHLSアドレス(i.e. m3u8)やHeartbeatするためのSession情報が戻ってくる。</li>
<li>HLSストリーム自体にセッションKeyの認証などないため、基本<a href="https://www.ffmpeg.org/"><code>ffplay</code></a>や<a href="https://mpv.io/">MPV</a>など使えばこのHLSアドレスから映像取れるが、定期的にHeartbeatしないと、サーバ側でセッションを削除するので途中からエラーが出る。ちなみに<a href="https://ytdl-org.github.io/youtube-dl/index.html"><code>youtube-dl</code></a>はよく<code>403</code>エラー起きるのもこれのせい。</li>
<li>なので、上記HLS再生中、定期的にサーバにHeartbeatのリクエストの投げる。</li>
</ol>
<h2>Get <code>data-api-data</code></h2>
<p>今回は認証の部分に触れないので、(ブラウザで)ニコニコにすでにログインでき、それのCookieを取れている状態を想定している。ちなみに、Cookieを取るにはログインの状態でFirefoxやChromeで適当にページを開くと、デベロッパーツールのネットワークから最初の<code>get</code>リクエストを見れば、リクエストにヘッダーにCookieがついている …</p><h2>Introduction</h2>
<p>Niconicoのフローは以下のような感じ:</p>
<p><img alt="Overview" src="https://www.horsal.dev/images/2021-08-15-overview.png"></p>
<ol>
<li>ログイン時取得したCookieを使って、<code>https://www.nicovideo.jp/watch/so*</code>をGetする。取得したページに<code>data-api-data</code>というJsonデータが埋められている。</li>
<li>上記の<code>data-api-data</code>を整備して、ほしい映像の解像度情報や認証情報をサーバに送る。するとそれに応じて映像のHLSアドレス(i.e. m3u8)やHeartbeatするためのSession情報が戻ってくる。</li>
<li>HLSストリーム自体にセッションKeyの認証などないため、基本<a href="https://www.ffmpeg.org/"><code>ffplay</code></a>や<a href="https://mpv.io/">MPV</a>など使えばこのHLSアドレスから映像取れるが、定期的にHeartbeatしないと、サーバ側でセッションを削除するので途中からエラーが出る。ちなみに<a href="https://ytdl-org.github.io/youtube-dl/index.html"><code>youtube-dl</code></a>はよく<code>403</code>エラー起きるのもこれのせい。</li>
<li>なので、上記HLS再生中、定期的にサーバにHeartbeatのリクエストの投げる。</li>
</ol>
<h2>Get <code>data-api-data</code></h2>
<p>今回は認証の部分に触れないので、(ブラウザで)ニコニコにすでにログインでき、それのCookieを取れている状態を想定している。ちなみに、Cookieを取るにはログインの状態でFirefoxやChromeで適当にページを開くと、デベロッパーツールのネットワークから最初の<code>get</code>リクエストを見れば、リクエストにヘッダーにCookieがついている。そのCookieをそのままコピーすればよい。</p>
<p><img alt="Cookie" src="https://www.horsal.dev/images/2021-08-15-cookie.png"></p>
<p>ちなみに、この辺のツールはよくNetscape形式の<code>cookies.txt</code>を使うが、今回はコピーしたものそのまま使えば良い。</p>
<p>コード書く前に、まずプロジェクト作ろう。</p>
<div class="highlight"><pre><span></span><code>cargo new niconico-video-player
</code></pre></div>
<p>また、<code>Cargo.toml</code>に以下のPackageを追加しましょう。</p>
<div class="highlight"><pre><span></span><code><span class="k">[dependencies]</span>
<span class="n">html-escape</span> <span class="o">=</span> <span class="s">"0.2"</span>
<span class="n">reqwest</span> <span class="o">=</span> <span class="s">"0.11"</span>
<span class="n">regex</span> <span class="o">=</span> <span class="s">"1"</span>
<span class="n">serde</span> <span class="o">=</span> <span class="s">"1"</span>
<span class="n">serde_derive</span> <span class="o">=</span> <span class="s">"1"</span>
<span class="n">serde_json</span> <span class="o">=</span> <span class="s">"1"</span>
<span class="n">lazy_static</span> <span class="o">=</span> <span class="s">"1"</span>
<span class="n">clap</span> <span class="o">=</span> <span class="s">"3.0.0-beta.2"</span>
<span class="n">tokio</span> <span class="o">=</span> <span class="p">{</span> <span class="n">version</span> <span class="o">=</span> <span class="s">"1"</span><span class="p">,</span> <span class="n">features</span> <span class="o">=</span> <span class="p">[</span><span class="s">"full"</span><span class="p">]</span> <span class="p">}</span>
</code></pre></div>
<p>通信するには<code>reqwest</code>、Jsonの解析は<code>serde_json</code>を使うので、最低限にこの2つが必要。
早速Requestを投げてみる</p>
<div class="highlight"><pre><span></span><code><span class="k">use</span><span class="w"> </span><span class="n">reqwest</span>::<span class="n">header</span>::<span class="n">COOKIE</span><span class="p">;</span><span class="w"></span>
<span class="cp">#[tokio::main]</span><span class="w"></span>
<span class="k">async</span><span class="w"> </span><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="p">(),</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">std</span>::<span class="n">error</span>::<span class="n">Error</span><span class="o">>></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">cookie</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">""</span><span class="p">;</span><span class="w"> </span><span class="c1">// Your cookie here</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">reqwest</span>::<span class="n">Client</span>::<span class="n">new</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">client</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="p">.</span><span class="n">url</span><span class="p">).</span><span class="n">header</span><span class="p">(</span><span class="n">COOKIE</span><span class="p">,</span><span class="w"> </span><span class="n">cookie</span><span class="p">).</span><span class="n">send</span><span class="p">().</span><span class="k">await</span><span class="o">?</span><span class="p">.</span><span class="n">text</span><span class="p">.</span><span class="k">await</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="n">res</span><span class="p">)</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p><code>header</code>の<code>cookie</code>に上で取得したCookieを入れよう。コードに埋め込むのはどうかと思うならばファイルから読み込むか<code>Clap</code>を使ってコマンドラインから取得するのも悪くない。
当然、テストなら最初から自分のアカウントで試す必要もないので、ログインしなくても見れる動画であれば、Cookieなしでも問題ない。</p>
<p>上記実行してみると、無事HTMLファイルを取れるはず。その中で検索してみると、こんな感じの行を見つかるはず:</p>
<div class="highlight"><pre><span></span><code> <span class="p"><</span><span class="nt">div</span> <span class="na">id</span><span class="o">=</span><span class="s">"js-initial-watch-data"</span> <span class="na">data-api-data</span><span class="o">=</span><span class="s">"{&quot;ads&quot;:null,&quot;category&quot;:null,&quot;channel&q</span> <span class="err">...</span>
</code></pre></div>
<p>この部分はRegexを使えば簡単に取れる、例えば</p>
<div class="highlight"><pre><span></span><code><span class="n">lazy_static</span><span class="o">!</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">static</span><span class="w"> </span><span class="k">ref</span><span class="w"> </span><span class="n">REG</span>: <span class="nc">regex</span>::<span class="n">Regex</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Regex</span>::<span class="n">new</span><span class="p">(</span><span class="s">"data-api-data=</span><span class="se">\"</span><span class="s">([^</span><span class="se">\"</span><span class="s">]+)</span><span class="se">\"</span><span class="s">"</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="k">match</span><span class="w"> </span><span class="n">REG</span><span class="p">.</span><span class="n">captures</span><span class="p">(</span><span class="o">&</span><span class="n">html</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">data</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">//取れた中身</span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="nb">None</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">panic!</span><span class="p">(</span><span class="s">"unknown page format"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>ただし見ればわかるように、中身はHTMLのコードになっているので、使う前に一回unescapeが必要だ。幸い<code>html-escape</code>がこの機能提供している:</p>
<div class="highlight"><pre><span></span><code><span class="nb">Some</span><span class="p">(</span><span class="n">data</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// data[0]は全体の文字列</span>
<span class="w"> </span><span class="c1">// data[1]からは各グループ</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">escaped_api_data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&</span><span class="n">data</span><span class="p">.</span><span class="n">unwrap</span><span class="p">()[</span><span class="mi">1</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">output</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">String</span>::<span class="n">new</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">raw_api_data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">decode_html_entities_to_string</span><span class="p">(</span><span class="n">escaped_api_data</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="n">output</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// jsonをパース</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">data_api</span>: <span class="nc">DataAPI</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">serde_json</span>::<span class="n">from_str</span><span class="p">(</span><span class="n">raw_api_data</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="p">},</span><span class="w"></span>
</code></pre></div>
<p><code>DataAPI</code>は色んな情報入ってるが、今回必要なのは<code>media</code>の部分だけ(更に言うと中の<code>session</code>さえあれば何とかなる)。なので、その部分をStructとして書き出す:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#[derive(Deserialize)]</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">DataAPI</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">media</span>: <span class="nc">Media</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="cp">#[derive(Deserialize)]</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">Media</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">delivery</span>: <span class="nc">Delivery</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="cp">#[derive(Deserialize)]</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">Media</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">recipeId</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">movie</span>: <span class="nc">Movie</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="cp">#[derive(Deserialize)]</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">Movie</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">contentId</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">audios</span>: <span class="nb">Vec</span><span class="o"><</span><span class="n">Audio</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">videos</span>: <span class="nb">Vec</span><span class="o"><</span><span class="n">Video</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">session</span>: <span class="nc">Session</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="o">....</span><span class="w"></span>
</code></pre></div>
<p>実際にページをアクセスしてみて、そのJsonに合わせてひたすら書くだけで良い。ちなみに、<code>data_api.media.delivery.movie.videos</code>に各StreamのBitrateが書いているので、あとの設定でその中のIDを指定すればその画質で再生できるようになる。</p>
<h2>Get HLS URL</h2>
<p>次は上記の情報を<code>data_api.media.delivery.movie.session.urls[0].url</code>に送信して実際のHLSを取得できるが、その前に送信するデータを整備しないといけない。</p>
<p>送信データはこんな感じ:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
<span class="nt">"session"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"recipe_id"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"content_id"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"content_type"</span><span class="p">:</span> <span class="s2">"movie"</span><span class="p">,</span>
<span class="nt">"content_src_id_sets"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nt">"content_src_ids"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nt">"src_id_to_mux"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"video_src_ids"</span><span class="p">:</span> <span class="p">[],</span>
<span class="nt">"audio_src_ids"</span><span class="p">:</span> <span class="p">[]</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="nt">"timing_constraint"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"keep_method"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"heartbeat"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"lifetime"</span><span class="p">:</span> <span class="mi">0</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nt">"protocol"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"parameters"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"http_parameters"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"parameters"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"hls_parameters"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"use_well_known_port"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"use_ssl"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"transfer_preset"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"segment_duration"</span><span class="p">:</span> <span class="mi">0</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nt">"content_uri"</span><span class="p">:</span> <span class="s2">""</span><span class="p">,</span>
<span class="nt">"session_operation_auth"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"session_operation_auth_by_signature"</span><span class="p">:</span> <span class="p">{</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nt">"content_auth"</span><span class="p">:</span> <span class="p">{</span>
<span class="p">},</span>
<span class="nt">"client_info"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"player_id"</span><span class="p">:</span> <span class="s2">""</span>
<span class="p">},</span>
<span class="nt">"priority"</span><span class="p">:</span> <span class="mi">0</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p><code>content_src_id_sets</code>は取得したいStreamの組み合わせ、<code>video_src_ids</code>にほしい画質のIDを書けば、サーバ側が指定した画質・音質に応じてストリームを合成して送信してくれる。これ以外の項目はすべて<code>data_api</code>に入っているので、そっちを見ながら埋めて良い。</p>
<p>例えば、Rustで生成したいは:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#[derive(Serialize)]</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">VideoSessionRequest</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">session</span>: <span class="nc">RequestVideo</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="cp">#[derive(Serialize)]</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">RequestVideo</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">recipe_id</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">content_id</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">content_type</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">content_src_id_sets</span>: <span class="nb">Vec</span><span class="o"><</span><span class="n">ContentSrcSet</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">timing_constraint</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">keep_method</span>: <span class="nc">KeepMethod</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>あとはこのJsonをサーバに送信するのみ:</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">client</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="fm">format!</span><span class="p">(</span><span class="s">"{}?_format=json"</span><span class="p">,</span><span class="n">data_api</span><span class="p">.</span><span class="n">media</span><span class="p">.</span><span class="n">delivery</span><span class="p">.</span><span class="n">movie</span><span class="p">.</span><span class="n">session</span><span class="p">.</span><span class="n">urls</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">url</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">body</span><span class="p">(</span><span class="n">serde_json</span>::<span class="n">to_string</span><span class="p">(</span><span class="o">&</span><span class="n">VideoSessionRequest</span>::<span class="n">from</span><span class="p">(</span><span class="n">data_api</span><span class="p">))</span><span class="o">?</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">send</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="k">await</span><span class="o">?</span><span class="w"></span>
</code></pre></div>
<p>成功すれば、サーバからこんな感じのJsonが貰える:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
<span class="nt">"meta"</span><span class="p">:</span> <span class="p">{</span>
<span class="nt">"status"</span><span class="p">:</span> <span class="mi">201</span><span class="p">,</span>
<span class="nt">"message"</span><span class="p">:</span> <span class="s2">"created"</span>
<span class="p">},</span>
<span class="nt">"data"</span><span class="p">:</span> <span class="p">{</span>
<span class="err">...</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p><code>.data.session.content_uri</code>に実際のアドレスが入っている。</p>
<h2>Heartbeat</h2>
<p>次は再生が中断されないように定期的にHeartbeatを行う。まずHeartbeatのアドレスは<code>https://api.dmc.nico/api/sessions/{}?_format=json&_method=PUT</code>, <code>{}</code>に上サーバから貰ったjsonの<code>.data.session.id</code>。送信する中身は<code>.data</code>と同じで問題ないので、Rustで書くならこんな感じ:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#[derive(Deserialize)]</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">struct</span> <span class="nc">VideoSessionInfo</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">meta</span>: <span class="nc">Meta</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="c1">// for heartbeat</span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">data</span>: <span class="nc">serde_json</span>::<span class="n">Value</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>送信内容はずっと変わらないので、送信前にまとめて整備すればよい。実際の送信は<code>tokio</code>でSpawnして、<code>timer</code>で定期的に行う:</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">hb_content</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">serde_json</span>::<span class="n">to_string</span><span class="p">(</span><span class="o">&</span><span class="n">video_session_info</span><span class="p">.</span><span class="n">data</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="n">tokio</span>::<span class="n">spawn</span><span class="p">(</span><span class="k">async</span><span class="w"> </span><span class="k">move</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// 2分一回Heratbeat</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">timer</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tokio</span>::<span class="n">time</span>::<span class="n">interval</span><span class="p">(</span><span class="n">std</span>::<span class="n">time</span>::<span class="n">Duration</span>::<span class="n">from_secs</span><span class="p">(</span><span class="mi">120</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">client</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">reqwest</span>::<span class="n">Client</span>::<span class="n">new</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">timer</span><span class="p">.</span><span class="n">tick</span><span class="p">().</span><span class="k">await</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">json</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">client</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">post</span><span class="p">(</span><span class="n">hb_url</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">body</span><span class="p">(</span><span class="n">hb_content</span><span class="p">.</span><span class="n">clone</span><span class="p">())</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">send</span><span class="p">().</span><span class="k">await</span><span class="p">.</span><span class="n">unwrap</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">text</span><span class="p">().</span><span class="k">await</span><span class="p">.</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">});</span><span class="w"></span>
</code></pre></div>
<p>Heartbeatが成功すれば、上記のアドレスを<code>mpv</code>や<code>ffplay</code>に渡せば普通に再生できるので、<code>tokio</code>の<code>Command</code>を使えば良い:</p>
<div class="highlight"><pre><span></span><code><span class="n">tokio</span>::<span class="n">process</span>::<span class="n">Command</span>::<span class="n">new</span><span class="p">(</span><span class="s">"mpv"</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">arg</span><span class="p">(</span><span class="n">video_session_info</span><span class="p">.</span><span class="n">data</span><span class="p">.</span><span class="n">session</span><span class="p">.</span><span class="n">content_uri</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">spawn</span><span class="p">()</span><span class="o">?</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">wait</span><span class="p">()</span><span class="w"></span>
</code></pre></div>
<p><code>wait</code>を使うことで、<code>mpv</code>などの再生を待つことできるので、Heartbeatもその間ずっと行われる。</p>
<p>これで無事ニコニコの再生をできるようになるはず。さらにやるなら、各リクエストにいい感じのHeader(少なくともUser-agent)をつけたり、コメントを取得してMPVの字幕に変換したりもできるが、今回はとりあえずここまで。</p>Things Behind ATH-DSR9bt(1)2021-08-11T00:00:00+09:002021-08-11T00:00:00+09:00Horsaltag:www.horsal.dev,2021-08-11:/things-behind-ath-dsr9bt1.html<h2>Introduction</h2>
<p>2014年、オーディオテクニカからフルデジタルのヘッドホン<a href="https://www.audio-technica.co.jp/product/ATH-DN1000USB">ATH-DN1000USB</a>が発売された。札幌のヨドバシで実機を聞いてから、その解像度をずっと忘れられなかった。ただし、当時としても値段がかなり高く(7万くらい)、私と同じ手を出せなかった人も多いでしょう。その結果、一時期話題になっていたが、あまり売れていなかった。</p>
<p>2年後、DN1000USBと同じ<a href="https://trigencednote.wixsite.com/japanese/dnote">Dnote</a>技術を使った<a href="https://www.audio-technica.co.jp/product/ATH-DSR9BT">DSR-7/9bt</a>が発売された。Bluetooth付き、デザインも更にかっこよく、何より値段は3万~6万、もはや売れる要素しかないと思いきや、普通に売れた程度だった。<a href="https://www.sony.jp/high-resolution/about/">ハイレゾ</a>のような怪しい概念をすんなりと受け入れられるオーディオ市場なのに、なぜ<em>フルデジタル</em>は無視されるだろう?レビューや掲示板を見ていると、フルデジタルがよく言われるのは</p>
<ul>
<li>結局アナログに変換されるでしょう?ならフルデジタルの意味はなんだ?</li>
<li>仕組み上アンプ入りのヘッドホンと何が違うんだ?</li>
<li>所詮無線だし、有線に勝てるわけがない</li>
</ul>
<p>最後はともかく、上の2つについて私も似てる疑問があった。Dnoteの説明もオーテクのインタビューも詳細を説明されていないため、今回は色んな特許文献を見ながら、フルデジタルの原理、その特徴について述べていく。</p>
<h2>The origin of Full-digital Audio …</h2><h2>Introduction</h2>
<p>2014年、オーディオテクニカからフルデジタルのヘッドホン<a href="https://www.audio-technica.co.jp/product/ATH-DN1000USB">ATH-DN1000USB</a>が発売された。札幌のヨドバシで実機を聞いてから、その解像度をずっと忘れられなかった。ただし、当時としても値段がかなり高く(7万くらい)、私と同じ手を出せなかった人も多いでしょう。その結果、一時期話題になっていたが、あまり売れていなかった。</p>
<p>2年後、DN1000USBと同じ<a href="https://trigencednote.wixsite.com/japanese/dnote">Dnote</a>技術を使った<a href="https://www.audio-technica.co.jp/product/ATH-DSR9BT">DSR-7/9bt</a>が発売された。Bluetooth付き、デザインも更にかっこよく、何より値段は3万~6万、もはや売れる要素しかないと思いきや、普通に売れた程度だった。<a href="https://www.sony.jp/high-resolution/about/">ハイレゾ</a>のような怪しい概念をすんなりと受け入れられるオーディオ市場なのに、なぜ<em>フルデジタル</em>は無視されるだろう?レビューや掲示板を見ていると、フルデジタルがよく言われるのは</p>
<ul>
<li>結局アナログに変換されるでしょう?ならフルデジタルの意味はなんだ?</li>
<li>仕組み上アンプ入りのヘッドホンと何が違うんだ?</li>
<li>所詮無線だし、有線に勝てるわけがない</li>
</ul>
<p>最後はともかく、上の2つについて私も似てる疑問があった。Dnoteの説明もオーテクのインタビューも詳細を説明されていないため、今回は色んな特許文献を見ながら、フルデジタルの原理、その特徴について述べていく。</p>
<h2>The origin of Full-digital Audio</h2>
<p>実はこの<em>フルデジタル</em>の概念は新しいものではなく、結構昔から提案されていた。特許検索してみると、こんなものが</p>
<blockquote>
<blockquote>
<p>JPS52121316A Pulse code modulated signal reproducing speaker (ソニー株式会社)</p>
</blockquote>
</blockquote>
<p>この特許はなんと1976年のもの、しかも今の文献にも引用されちゃんとしたフルデジタルの大元だ。
なので、Dnoteを語る前に、まずは当時の考え方を見ていきましょう。</p>
<h3>PCM Audio, and Basic Structure of Headphones</h3>
<p>ヘッドホンを語る前に、まずいくつ基本概念を理解しないといけない。最初は音声保存方式のPCM(パルス符号変調)だ:</p>
<p><img alt="PCM Audio" src="https://www.horsal.dev/images/2021-08-11-pcm.png"></p>
<p>赤い線が音声信号の場合、一定の時間<span class="math">\(T\)</span>毎にこの線の高さ(信号の強さ)を記録し、この高さを<span class="math">\(N\)</span>段階量子化すれば、信号の近似を得られる。この記録の方法はPCMだ。
ハイレゾの「ハイ」っていうのも、元の音声を記録した時、<span class="math">\(N\)</span>を増やす・<span class="math">\(T\)</span>を減らすを意味している。
まとめると、音声はこんな感じでユーザの所に届いている:</p>
<p><img alt="General Arch" src="https://www.horsal.dev/images/2021-08-11-arch.png"></p>
<p>量子化された信号は最終的に(近似の)連続信号に戻され、ヘッドホンによって再現されるが、そのヘッドホンの一般的な構造はこんな感じだ:</p>
<p><img alt="Headphone" src="https://www.horsal.dev/images/2021-08-11-headphone.png"></p>
<p>アナログの信号は電気としてボイスコイルに流れて、ボイスコイルがそれに応じて磁気を発生し前後に動く。ボイスコイルについている振動板も同時に動いて空気を動かし、電気信号による振動を空気の振動(音声)変換する。</p>
<p>もっと知りたい人は<a href="https://www.audio-technica.co.jp/headphone/navi/whatis/">オーテクがいい感じの説明</a>を用意しているのでぜひ一度見てみてください。</p>
<p>この図を見ればわかると思うが、</p>
<ul>
<li>電気信号が如何に元の信号に近いか</li>
<li>振動板が如何に忠実に振動するか</li>
</ul>
<p>によって音声の正確さが決まるため、今までどのオーディオ会社も高性能の振動板やよりノイズ少ない回路を作っていた。ただし、以下の問題はずっと残っていた:</p>
<blockquote>
<p>一回量子化された信号は、もう元に戻せない。ならば、その戻す処理は無駄ではないか?</p>
</blockquote>
<h3>Full-digital Audio, First Try</h3>
<p>その戻す処理が無駄なら、一層それをなくせばいいのでは?とソニーが考え出した。
前述の通り、信号は1つ1つの数値でしかないため、そのタイミングで振動板がその強さになっていれば、もう記録された情報をすべて表現したと言えるだろう。なので、<code>JPS52121316A</code>の特許はこんなことをしていた:</p>
<blockquote>
<blockquote>
<p>強さ毎にボイスコイルを用意して、信号の強さに応じて対応のボイスコイルを鳴らす</p>
</blockquote>
</blockquote>
<p>絵で書くとこんな感じ:</p>
<p><img alt="Full-digital Audio" src="https://www.horsal.dev/images/2021-08-11-sony.png"></p>
<p>つまり、ボイスコイル1は強さ1しか出せない、ボイスコイル2は強さ2しか出せない...
そうすると、ある強さが必要の場合、そのボイスコイルを鳴らすだけでいけるのだ、電圧も電流も調整する必要はない。こう見ると、あるボイスコイルに対して状態は<code>on</code>と<code>off</code>しかないため、デジタルそのままと言えるだろう。</p>
<p>また、特に注意してほしいのは今の音声の量子化ビット数は16bit以上もあるので、最低でも<span class="math">\(2^{16}=65536\)</span>のパターンがある。<span class="math">\(65536\)</span>個のボイスコイルを用意できるわけがないが、代わりに、ビット毎のボイスコイルを用意することが可能だ。ボイスコイルの強さを<span class="math">\(1,2,4,8,16,\dots\)</span>の順に作ると、1つのボイスコイルは量子化の1ビットに相当するので、16個のボイスコイルで16ビットの音声を表現できるようになる。</p>
<p>ちなみに、ボイスコイルの強さは線を巻く回数によって決めているので、<code>8bit</code>ならまだいいが、今の<code>24bit</code>や<code>32bit</code>になるとさすがに対応できなくなるだろう。</p>
<p>そうすると、今のフルデジタルヘッドホンはどう実現したのだろうか?次回のブログではDnote技術の特許を見ながら、実現方式を説明する。</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Enum, Generic and Templates2020-02-23T00:00:00+09:002020-02-23T00:00:00+09:00Horsaltag:www.horsal.dev,2020-02-23:/enum-generic-and-templates.html<p>在很久之前,我曾经<a href="https://www.horsal.dev/ooc-generics-and-flawed-design.html">写过(或者说,翻译过)一篇关于OOC里泛型的博客</a>,在那个时候,我对OOC的泛型设计是持否定态度的——相比起OOC的动态泛型,那时的我认为类似C++的泛型更加好用。类型在编译时是确定的,因此编译器可以进行静态类型检查,同时没有执行时的性能损失,也不需要在使用时cast,不会出现错误……总之,似乎没有理由去选择OOC的设计。
在那之后的2~3年里,我也一直都是这么认为的。</p>
<p>当然,Rust也是这样的,因此这几年我也一直很满足,直到最近遇到的问题。</p>
<h2>An Example of Deserialization</h2>
<p>让我们先来考虑一个简单的场景,有某个服务用Json传送信息,里面包含了一个服务器列表,服务器有几种类型,每一种有不同的属性,比如:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
<span class="nt">"server_list"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"server_a"</span><span class="p">,</span>
<span class="nt">"role"</span><span class="p">:</span> <span class="s2">"front"</span><span class="p">,</span>
<span class="nt">"scale"</span><span class="p">:</span> <span class="mi">10</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"server_b"</span><span class="p">,</span>
<span class="nt">"role"</span><span class="p">:</span> <span class="s2">"worker"</span><span class="p">,</span>
<span class="nt">"is_debug"</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="nt">"restart_time"</span><span class="p">:</span> <span class="s2">"23 …</span></code></pre></div><p>在很久之前,我曾经<a href="https://www.horsal.dev/ooc-generics-and-flawed-design.html">写过(或者说,翻译过)一篇关于OOC里泛型的博客</a>,在那个时候,我对OOC的泛型设计是持否定态度的——相比起OOC的动态泛型,那时的我认为类似C++的泛型更加好用。类型在编译时是确定的,因此编译器可以进行静态类型检查,同时没有执行时的性能损失,也不需要在使用时cast,不会出现错误……总之,似乎没有理由去选择OOC的设计。
在那之后的2~3年里,我也一直都是这么认为的。</p>
<p>当然,Rust也是这样的,因此这几年我也一直很满足,直到最近遇到的问题。</p>
<h2>An Example of Deserialization</h2>
<p>让我们先来考虑一个简单的场景,有某个服务用Json传送信息,里面包含了一个服务器列表,服务器有几种类型,每一种有不同的属性,比如:</p>
<div class="highlight"><pre><span></span><code><span class="p">{</span>
<span class="nt">"server_list"</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"server_a"</span><span class="p">,</span>
<span class="nt">"role"</span><span class="p">:</span> <span class="s2">"front"</span><span class="p">,</span>
<span class="nt">"scale"</span><span class="p">:</span> <span class="mi">10</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"server_b"</span><span class="p">,</span>
<span class="nt">"role"</span><span class="p">:</span> <span class="s2">"worker"</span><span class="p">,</span>
<span class="nt">"is_debug"</span><span class="p">:</span> <span class="kc">false</span><span class="p">,</span>
<span class="nt">"restart_time"</span><span class="p">:</span> <span class="s2">"23:55"</span><span class="p">,</span>
<span class="nt">"restart_type"</span><span class="p">:</span> <span class="s2">"everyday"</span>
<span class="p">},</span>
<span class="p">{</span>
<span class="nt">"name"</span><span class="p">:</span> <span class="s2">"server_c"</span><span class="p">,</span>
<span class="nt">"role"</span><span class="p">:</span> <span class="s2">"backup"</span><span class="p">,</span>
<span class="nt">"scale"</span><span class="p">:</span> <span class="s2">"100"</span><span class="p">,</span>
<span class="nt">"storage_limit"</span><span class="p">:</span> <span class="s2">"24G"</span><span class="p">,</span>
<span class="nt">"log_level"</span><span class="p">:</span> <span class="s2">"debug"</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
<p>直接操作json肯定不是好选项,大部分情况下用serde先Deserialize是个不错的办法。</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span> <span class="nc">Server</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="n">server_list</span>: <span class="nb">Vec</span><span class="o"><..</span><span class="p">.</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="o">....</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">server_list</span><span class="w"> </span>: <span class="nc">Server</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">serde_json</span>::<span class="n">from_str</span><span class="p">(</span><span class="o">&</span><span class="n">json_str</span><span class="p">)</span><span class="o">?</span><span class="p">;</span><span class="w"></span>
<span class="o">....</span><span class="w"></span>
</code></pre></div>
<p>现在问题就来了,<code>server_list</code>显然是一个Vec,但它的内容不是一致的——里面其实有数个不同的类型。
并且这种写法并不少见,json,xml,yaml等等都可以这么做。
如果不同类型的属性名称是不同的,那么我们可以把它们全部合并成一个巨大的struct,然后根据role来判断需要哪些field:</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span> <span class="nc">ServerItem</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">name</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">role</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">scale</span>: <span class="nb">Option</span><span class="o"><</span><span class="kt">i64</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">is_debug</span>: <span class="nb">Option</span><span class="o"><</span><span class="kt">bool</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">restart_time</span>: <span class="nb">Option</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">restart_type</span>: <span class="nb">Option</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">storage_limit</span>: <span class="nb">Option</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">log_level</span>: <span class="nb">Option</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">&</span><span class="n">server_list</span><span class="p">.</span><span class="n">server_list</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">item</span><span class="p">.</span><span class="n">role</span><span class="p">.</span><span class="n">as_str</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="s">"front"</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="s">"worker"</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="s">"backup"</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">unreachable!</span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这样我们就能统一的访问这些成员了。当然,每一次访问都需要判断<code>role</code>,并且要处理大量的<code>Option</code>,导致代码看起来很冗长。(Rust的<code>Option</code>的Zero-cost是指内存上的,但并不代表代码上写起来是zore-cost的)</p>
<p>并且,另外一个更重要的问题是——如果不同种类的属性之间有冲突,这个办法就没法用了。比如这里的<code>scale</code>,在front里他是一个数字,然而在backup里他是一个字符串。这样处理起来就麻烦多了。
当然,serde也能处理这种情况:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">any_to_str</span><span class="o"><</span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="n">S</span><span class="o">></span><span class="p">(</span><span class="n">data</span>: <span class="kp">&</span><span class="nc">T</span><span class="p">,</span><span class="w"> </span><span class="n">s</span>: <span class="nc">S</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="n">S</span>::<span class="nb">Ok</span><span class="p">,</span><span class="w"> </span><span class="n">S</span>::<span class="n">Error</span><span class="o">></span><span class="w"></span>
<span class="k">where</span><span class="w"></span>
<span class="w"> </span><span class="n">S</span>: <span class="nc">Serializer</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">T</span>: <span class="nc">std</span>::<span class="n">fmt</span>::<span class="n">Debug</span><span class="p">,</span><span class="w"></span>
<span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">s</span><span class="p">.</span><span class="n">serialize_str</span><span class="p">(</span><span class="o">&</span><span class="fm">format!</span><span class="p">(</span><span class="s">"{:?}"</span><span class="p">,</span><span class="w"> </span><span class="o">&</span><span class="n">data</span><span class="p">))</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">ServerItem</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="cp">#[serde(deserialize_with=</span><span class="s">"any_to_string"</span><span class="cp">)]</span><span class="w"></span>
<span class="w"> </span><span class="n">scale</span>: <span class="nb">Option</span><span class="o"><</span><span class="nb">String</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这样,任何类型的<code>scale</code>都会转换成字符串,我们可以在后面的处理中根据需要再<code>parse</code>回数字。
很显然的,这种做法效率很低,并且会导致代码进一步的复杂,如果未来消息里不停的有这种情况,我们要不停的修改这个巨大的struct,并且跟着修改各种对应的parse。并且,随着类型的增加,这个巨大struct会失去维护性——从字面上根本看不出哪些类型拥有哪些属性,我们也无法在deserialize时检查数据是不是正确的了(因为它们全都是Option的)。</p>
<h2>Enum Varints</h2>
<p>一个比较常见的解决办法就是用Enum了,Rust的Enum Variants可以像Struct一样拥有自己的成员,因此,对上面的例子我们可以这样写:</p>
<div class="highlight"><pre><span></span><code><span class="cp">#[serde(tag = </span><span class="s">"role"</span><span class="cp">)]</span><span class="w"></span>
<span class="k">enum</span> <span class="nc">ServerItem</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">front</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">name</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">scale</span>: <span class="kt">i64</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="n">worker</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">name</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">is_debug</span>: <span class="kt">bool</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">restart_time</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">restart_type</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="n">backup</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">name</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">scale</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">storage_limit</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">log_level</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这样,我们可以把列表Parse成一个<code>ServerItem</code>的Vec了,每一个属性都是只跟当前的类型有关,不再需要类型转换和Option了。在处理Vec的时候会变得很轻松。</p>
<p>不过,其实还有一个问题——处理的时候我们依然需要判断类型,就像这样:</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">&</span><span class="n">server_list</span><span class="p">.</span><span class="n">server_list</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">ServerItem</span>::<span class="n">front</span><span class="p">{</span><span class="n">name</span><span class="p">,</span><span class="w"> </span><span class="n">scale</span><span class="p">,</span><span class="w"> </span><span class="o">..</span><span class="p">.}</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="n">ServerItem</span>::<span class="n">worker</span><span class="p">{</span><span class="n">name</span><span class="p">,</span><span class="w"> </span><span class="o">..</span><span class="p">.}</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{},</span><span class="w"></span>
<span class="w"> </span><span class="n">serverItem</span>::<span class="n">backup</span><span class="p">{</span><span class="n">name</span><span class="p">,</span><span class="w"> </span><span class="n">scale</span><span class="p">,</span><span class="w"> </span><span class="o">..</span><span class="p">.}</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{},</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>在第一次遇到一个ServerItem的时候,判断类型并没有什么问题,然而就算我们已经知道了它的类型,每次用到它还是需要重新来一次:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">process_front</span><span class="p">(</span><span class="n">item</span>: <span class="kp">&</span><span class="nc">ServerItem</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="o">&</span><span class="n">ServerItem</span>::<span class="n">front</span><span class="p">{</span><span class="w"> </span><span class="n">name</span><span class="p">,</span><span class="w"> </span><span class="n">scale</span><span class="p">,</span><span class="w"> </span><span class="o">..</span><span class="p">.}</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{},</span><span class="w"></span>
<span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="fm">unreachable!</span><span class="p">()</span><span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="c1">// 我们已经知道这是一个front</span>
<span class="n">process_front</span><span class="p">(</span><span class="o">&</span><span class="n">item</span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>可以想象到每次改变scope,我们都要重新确认item到底是什么,但我们早就知道了——因此这除了让代码变长之外并没有什么意义。为了避免这种情况,我们需要一些办法。</p>
<h2>Enum::as_struct</h2>
<p>一个很直白的方法就是:对enum,我们准备很多<code>as_..</code>的方法,把每个variant都转换成对应的struct。实际上,<a href="https://docs.serde.rs/serde_json/enum.Value.html">serde_json的Value就是这么做的</a>。</p>
<div class="highlight"><pre><span></span><code><span class="k">enum</span> <span class="nc">ServerItem</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">Front</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span><span class="w"> </span>
<span class="k">struct</span> <span class="nc">Worker</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">Backup</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">ServerItem</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">is_front</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">bool</span> <span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">is_worker</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">bool</span> <span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">is_backup</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">bool</span> <span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">as_front</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Option</span><span class="o"><</span><span class="n">Front</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">as_worker</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Option</span><span class="o"><</span><span class="n">Worker</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">as_backup</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Option</span><span class="o"><</span><span class="n">Backup</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这样,我们在处理之前,可以把它们转换成对应的类型:</p>
<div class="highlight"><pre><span></span><code><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">front</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">item</span><span class="p">.</span><span class="n">as_front</span><span class="p">();</span><span class="w"></span>
<span class="n">process_front</span><span class="p">(</span><span class="o">&</span><span class="n">front</span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>这样下来,后面的处理就变得简洁多了。这也是目前主流的做法。
但还有一个问题,这种处理能不能变得更简洁一些?</p>
<h2>Enum Variants as Type</h2>
<p>一个很直接的想法就是,让每个Enum Variant都成为单独的类型,这样我们就能把参数定义成这个variant,或者用泛型来处理了,比如:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">process_front</span><span class="p">(</span><span class="n">front_item</span>: <span class="nc">ServerItem</span>::<span class="n">Front</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>显然如果能够这么做,那么上面的问题大都不存在了,我们甚至不需要这种函数,因为在上面的循环里直接处理就已经很清晰了:</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">server_list</span><span class="p">.</span><span class="n">server_list</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// process item</span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">ServerItem</span>::<span class="n">Front</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// process item directly</span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>在这里,每个item在match之前就已经带着类型了,这里的match仅仅是一个guard,并不涉及类型转换。按照这个设计,下面的写法也是正确的:</p>
<div class="highlight"><pre><span></span><code><span class="k">enum</span> <span class="nc">Foo</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">A</span><span class="w"> </span><span class="p">(</span><span class="kt">i32</span><span class="p">,</span><span class="w"> </span><span class="kt">i64</span><span class="w"> </span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">B</span><span class="w"> </span><span class="p">(</span><span class="nb">String</span><span class="p">,</span><span class="w"> </span><span class="kt">i8</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">C</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="kd">let</span><span class="w"> </span><span class="n">foo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Foo</span>::<span class="n">A</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="mi">20</span><span class="p">};</span><span class="w"></span>
<span class="k">match</span><span class="w"> </span><span class="n">foo</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">A</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">B</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// handle foo</span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="n">C</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// do nothing</span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>到这里,所有熟悉Rust的人都会看出问题——这跟目前的类型系统是有矛盾的。A和B是不同的类型,虽然foo的类型是确定的,但在<code>A|B</code>的Arm下我们并不知道它到底是哪一种,因此也无法取出它的内部数据。就算写成<code>A(bar, baz) | B (bar, baz)</code>,这里的bar和baz的类型依然是冲突的,它的类型在编译期不确定,自然也没法这么使用。(纵使给他们不同的名字,我们也不知道到底那个Arm match了,因此每个变量都是Option的,我们还是要挨个判断)</p>
<p>其实,Rust的开发者们<a href="https://github.com/rust-lang/rfcs/pull/1450">从2016年就想给Enum Variants加类型了</a>,但上面这个问题一直是绊脚石。2018年<a href="https://github.com/rust-lang/rfcs/pull/1450">有人重新提起了这个问题</a>,也并没有获得很多正面反馈。</p>
<h2>Enum Variants and Generics</h2>
<p>这让我想起了过去的OOC。实际上OOC的Generics看起来就像是专门用来解决这个问题的。用Rust的语言来说,其实OOC打算实现这么一个东西:</p>
<p>对于这么一个定义:</p>
<div class="highlight"><pre><span></span><code><span class="k">enum</span> <span class="nc">Foo</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">A</span><span class="p">(</span><span class="kt">i32</span><span class="p">,</span><span class="w"> </span><span class="kt">i64</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">B</span><span class="p">(</span><span class="nb">String</span><span class="p">,</span><span class="w"> </span><span class="kt">i8</span><span class="p">),</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>编译器会把它翻译成:</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span> <span class="nc">A</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">_ano1</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">_ano2</span>: <span class="kt">i64</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">B</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">_ano1</span>: <span class="nb">String</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">_ano2</span>: <span class="kt">i8</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="c1">// 每一个Enum都会有自己的Trait,不过它们的定义都是一样的</span>
<span class="k">trait</span><span class="w"> </span><span class="n">Foo</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">whoami</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">TypeId</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">Foo</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">A</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">whoami</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">TypeId</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">//注意,这里的内容其实是在编译期就确定了的,</span>
<span class="w"> </span><span class="c1">//因此这个函数并没有执行开销(inline之后就是一个usize的常数)</span>
<span class="w"> </span><span class="n">std</span>::<span class="n">any</span>::<span class="n">TypeId</span>::<span class="n">of</span><span class="o"><</span><span class="n">A</span><span class="o">></span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">Foo</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">B</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">whoami</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">TypeId</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span>::<span class="n">any</span>::<span class="n">TypeId</span>::<span class="n">of</span><span class="o"><</span><span class="n">B</span><span class="o">></span><span class="p">()</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>因此,实际上Foo并不是真正的类型(这里仅仅使用Rust的语言来描述,我们只能用Trait,实际上OOC的定义要更自然一些,更接近一个Meta类型)。当我们使用它的variants时,其实是这样的:</p>
<p>比如下面的代码:</p>
<div class="highlight"><pre><span></span><code><span class="c1">// 实际上,foo的类型是Box<dyn Foo>,它的“实际类型”是A。</span>
<span class="kd">let</span><span class="w"> </span><span class="n">foo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Foo</span>::<span class="n">A</span><span class="p">(</span><span class="mi">64</span><span class="p">,</span><span class="w"> </span><span class="mi">32</span><span class="p">);</span><span class="w"></span>
<span class="k">match</span><span class="w"> </span><span class="n">foo</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// 编译器有foo的所有信息,显然这里是可以判定的,但cast则是由用户完成的。</span>
<span class="w"> </span><span class="c1">// 这意味着用户可以故意的把一个A cast成一个B,但这会导致运行时Panic。</span>
<span class="w"> </span><span class="n">Foo</span>::<span class="n">A</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">process</span><span class="p">(</span><span class="n">foo</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">A</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">Foo</span>::<span class="n">B</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">process2</span><span class="p">(</span><span class="n">foo</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">B</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>其实会被翻译成:</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">foo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Box</span>::<span class="n">new</span><span class="p">(</span><span class="n">A</span><span class="w"> </span><span class="p">{</span><span class="mi">64</span><span class="p">,</span><span class="w"> </span><span class="mi">32</span><span class="p">})</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">Foo</span><span class="o">></span><span class="p">;</span><span class="w"></span>
<span class="k">match</span><span class="w"> </span><span class="n">foo</span><span class="p">.</span><span class="n">whoami</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span>::<span class="n">any</span>::<span class="n">TypeId</span>::<span class="n">of</span><span class="o"><</span><span class="n">A</span><span class="o">></span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">_tmp_foo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">foo</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">Any</span><span class="o">></span><span class="p">).</span><span class="n">downcast_ref</span>::<span class="o"><</span><span class="n">A</span><span class="o">></span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="c1">//这时,_tmp_foo的类型已经是A了。</span>
<span class="w"> </span><span class="n">process</span><span class="p">(</span><span class="n">_tmp_foo</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span>::<span class="n">any</span>::<span class="n">TypeId</span>::<span class="n">of</span><span class="o"><</span><span class="n">B</span><span class="o">></span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">_tmp_foo</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">foo</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">Any</span><span class="o">></span><span class="p">).</span><span class="n">downcast_ref</span>::<span class="o"><</span><span class="n">B</span><span class="o">></span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">process</span><span class="p">(</span><span class="n">_tmp_foo</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>当然,这并没有解决所有的问题(尤其是Rust存在的binding问题),但对于大部分的情况,它足够强壮,也足够优雅了——我们有了variants的类型,没有失去类型检查,编译器可以解决绝大部分的转换问题,除了稍微有一点运行时的损耗(但这是必不可少的)。</p>
<p>所以,每次会想起在OOC里发生的争论,我对会回过头来看Rust里的设计,C++的Template是否真的比OOC的Generic优雅?运行时的检查和Cast是否比确定性的生成要受限制?每次用到泛型时写一次cast是否真的比编译器的静态检查要冗长?</p>
<p>三年前,我或许会毫不犹豫的回答“是”,但现在,我又没法下结论了。</p>Back to Mac2018-01-03T00:00:00+09:002018-01-03T00:00:00+09:00Horsaltag:www.horsal.dev,2018-01-03:/back-to-mac.html<p>在过去的一段时间里,我一直在用ZTE的安卓手机<a href="http://www.ztemobile.jp/products/axon7.html">Axon 7</a>,我并不讨厌Android,也没有感觉到网上所说的卡顿或者内存不足等问题,不过系统更新太慢&弹窗还是让人觉得用起来并不舒服。实际上,日本版的Axon7直到去年的7月才<a href="https://prtimes.jp/main/html/rd/p/000000011.000024971.html">更新了Android7.1.1</a>,并且似乎并没有后续更新的计划,我并不期待新版本的功能,但缺乏Security Patch还是让人颇有些不安的。于是,趁着iPhone发布新版本,我换了iPhone8。</p>
<p>实际上,在iPhone之前,我已经买了iPad,因为它给了我很大的惊喜。在ヨドバシカメラ的店头随手试了一下iPad Pro + Apple Pencil之后,流畅的书写感觉给我留下了很深的印象,以至于我当时就买了下来。后来半年的时间里iPad也没有让我失望,配合<a href="https://itunes.apple.com/jp/app/goodnotes-4/id778658393?mt=8">GoodNotes4</a>,不论是读论文还是会议笔记,iPad都没有出现任何问题。由于在iPad上书写几乎对环境没有要求(不需要光源,不需要桌子或垫板),在睡觉之前甚至可以趴在床上写日记。毫无疑问这是我这些年里买的最不会后悔的产品了。</p>
<p>在换了iPhone之后,我把很久之前买的Mac mini mid2012翻了出来,自从把环境全部转移到了ArchLinux+i3之后Mac mini就一直没有开过机。跟Linux比起来macOS的优势并不明显,尤其是低配版的Mac mini并没有SSD …</p><p>在过去的一段时间里,我一直在用ZTE的安卓手机<a href="http://www.ztemobile.jp/products/axon7.html">Axon 7</a>,我并不讨厌Android,也没有感觉到网上所说的卡顿或者内存不足等问题,不过系统更新太慢&弹窗还是让人觉得用起来并不舒服。实际上,日本版的Axon7直到去年的7月才<a href="https://prtimes.jp/main/html/rd/p/000000011.000024971.html">更新了Android7.1.1</a>,并且似乎并没有后续更新的计划,我并不期待新版本的功能,但缺乏Security Patch还是让人颇有些不安的。于是,趁着iPhone发布新版本,我换了iPhone8。</p>
<p>实际上,在iPhone之前,我已经买了iPad,因为它给了我很大的惊喜。在ヨドバシカメラ的店头随手试了一下iPad Pro + Apple Pencil之后,流畅的书写感觉给我留下了很深的印象,以至于我当时就买了下来。后来半年的时间里iPad也没有让我失望,配合<a href="https://itunes.apple.com/jp/app/goodnotes-4/id778658393?mt=8">GoodNotes4</a>,不论是读论文还是会议笔记,iPad都没有出现任何问题。由于在iPad上书写几乎对环境没有要求(不需要光源,不需要桌子或垫板),在睡觉之前甚至可以趴在床上写日记。毫无疑问这是我这些年里买的最不会后悔的产品了。</p>
<p>在换了iPhone之后,我把很久之前买的Mac mini mid2012翻了出来,自从把环境全部转移到了ArchLinux+i3之后Mac mini就一直没有开过机。跟Linux比起来macOS的优势并不明显,尤其是低配版的Mac mini并没有SSD,启动terminal都要花费数十秒的时间。另外没有使用Mac mini的另一个重要原因是没有使用它的动力 -- 单独一台macOS电脑并没有什么优势,传送文件依然需要scp或者网盘,编辑文档也没有比Windows机器甚至Linux号多少。如果说单独一台macOS电脑比其他环境优越的地方,可能只有Dvorak键盘配置起来比其他系统简单一个数量级。在Windows下面,想使用Dvorak的IME几乎只能靠外部软件或者修改注册表,我为此在one drive里保存了一堆注册表文件,用来切换Dvorak和qwert键盘。在Linux也至少需要用<code>setxkbmap</code>来修改,并且并不怎么稳定。在macOS下只需要在默认IME下面原则Dvorak键盘就足够了 -- 不仅仅是系统自带的输入法,Google IME也有这个功能。不过,在买了可编程键盘之后,这个设置也变得无所谓了。</p>
<p>对于iOS,问题也是一样的,虽然新版的iPad pro大理的宣传了“生产力”,但如果出差只有iPad,还是会有不少问题,比如
iPad不支持Dvorak,因此对于Dvorak使用者,apple推出的smart connector键盘就完全没有意义(目前两款键盘均不支持按键编程),同时市面上一大堆Bluetooth和Lightning的键盘也同时失去了意义。而普通的可编程键盘用起来也不怎么舒服 -- 苹果提供了Lightning to USB外设,但同时限制了电流导致有些键盘接上去是没法用的。但是苹果却不公布可用的产品列表,因此买iPad用键盘几乎等同于抽奖。另外iOS虽然提供了Files应用,但却无法向USB stick里拷贝文件,外出需要到便利店打印的时候基本都需要USB stick来传送文件,只有iPad的时候事情就变的很麻烦了。</p>
<p>不过同时有了iOS和macOS之后一切就好多了,macOS上没有编辑完的文档可以同步到iPad上继续修改,在iPad上批注或者签名的PDF也可以Airdrop到电脑上确认。不知道什么时候开始有的Handoff功能也相当方便,所有的苹果设备之间可以共享剪贴板,也可以打开别的设备上的页面或者程序。尤其在没有网络的地方写论文的时候,可以通过iPhone查参考文献,用iPad阅读,然后把Google scholar上的bibtex条目直接复制到macOS上,在别的系统下面从来没有过这么舒服的体验。因此,我决定再次回到macOS上处理日常事务。另外如果今年Mac mini能够更新的话,我很乐意买一台。</p>
<hr>
<p>其实,在刚回到macOS的时候,我曾经试图吧博客的Static Page Generator从Pelican换成其他的 -- 原因是Pelican似乎无法识别我的Markdown。不过可惜的是别的generator都不怎么符合我的心意,毕竟我只需要一个写Markdown,然后转化成html的工具而已,提供一堆选项让切换模版都很难的话就适得其反了。</p>
<p>在写这篇博客的时候,我偶尔看到无法识别markdown是因为没有安装python的Markdown package,但由于即没有Warning也没有在主页上有提示,想找到答案还是挺困难的。</p>Writing A Threadpool in Rust2017-08-26T00:00:00+09:002017-08-26T00:00:00+09:00Horsaltag:www.horsal.dev,2017-08-26:/writing-a-threadpool-in-rust.html<p>多线程一直是我相当不相碰的东西,总觉得看起来很棒,用起来却一点都不放心——尤其是过去用Delphi体验了多线程之后。实际上到了多线程里根本就没法定位那里出了错误,因此大部分时间压根不是在“调试”,而是告诉用户怎么用才能避免这个错误。在给OOC写MultiTheard Generating的时候我也紧紧用了最简单的ThreadPool,并且因为互斥锁多线程还没有单线程快。总之,这么多年我一直有意的躲避多线程的问题。</p>
<p>而Rust一直在宣传它在多线程上的优势——所有权如何的好,在Servo里面他们如何用Rust写了个漂亮的Parallel Parser。加上最近一年一直再用Rust写游戏的服务器,于是我决定回头看看Rust下的多线程体验到底如何。</p>
<h2>ThreadPool</h2>
<p>相比简单的用多线程算个加法,或者写个<a href="http://benchmarksgame.alioth.debian.org/">The Computer Language Benchmarks Game</a>的测试程序来说,ThreadPool可能更适合练手。在这里,我会实现下面这个结构的ThreadPool:</p>
<p><img alt="ThreadPool" src="https://www.horsal.dev/images/threadpool.png"></p>
<p>简单来说,主线程通过Channel向ThreadPool发送Job,然后Threadpool里的每一个子线程在空闲时都会尝从Channel里获取Job,然后执行它。执行完毕之后,Job的返回结果会通过另一个Channel传送给主线程。实际上,在C或Delphi里,实现这么一个东西似乎并不是很困难,然而,Rust因为独特的所有权制度,代码比思路要复杂一些,下面,让我们来看看怎么实现这个ThreadPool。</p>
<h2>Jobs for the Job</h2>
<p>首先 …</p><p>多线程一直是我相当不相碰的东西,总觉得看起来很棒,用起来却一点都不放心——尤其是过去用Delphi体验了多线程之后。实际上到了多线程里根本就没法定位那里出了错误,因此大部分时间压根不是在“调试”,而是告诉用户怎么用才能避免这个错误。在给OOC写MultiTheard Generating的时候我也紧紧用了最简单的ThreadPool,并且因为互斥锁多线程还没有单线程快。总之,这么多年我一直有意的躲避多线程的问题。</p>
<p>而Rust一直在宣传它在多线程上的优势——所有权如何的好,在Servo里面他们如何用Rust写了个漂亮的Parallel Parser。加上最近一年一直再用Rust写游戏的服务器,于是我决定回头看看Rust下的多线程体验到底如何。</p>
<h2>ThreadPool</h2>
<p>相比简单的用多线程算个加法,或者写个<a href="http://benchmarksgame.alioth.debian.org/">The Computer Language Benchmarks Game</a>的测试程序来说,ThreadPool可能更适合练手。在这里,我会实现下面这个结构的ThreadPool:</p>
<p><img alt="ThreadPool" src="https://www.horsal.dev/images/threadpool.png"></p>
<p>简单来说,主线程通过Channel向ThreadPool发送Job,然后Threadpool里的每一个子线程在空闲时都会尝从Channel里获取Job,然后执行它。执行完毕之后,Job的返回结果会通过另一个Channel传送给主线程。实际上,在C或Delphi里,实现这么一个东西似乎并不是很困难,然而,Rust因为独特的所有权制度,代码比思路要复杂一些,下面,让我们来看看怎么实现这个ThreadPool。</p>
<h2>Jobs for the Job</h2>
<p>首先,让我们来设计Job,在这篇文章里,Job总是一个没有参数的函数,用代码来说,是这样:</p>
<div class="highlight"><pre><span></span><code><span class="k">type</span> <span class="nc">Job</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="nb">Fn</span><span class="p">()</span><span class="w"> </span>-> <span class="nc">T</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="o">'</span><span class="nb">static</span><span class="o">></span><span class="w"></span>
</code></pre></div>
<p>由于我们需要把返回值传回给主线程,因此Job的返回值需要Send属性。这里,没有用<code>FnOnce</code>而是<code>Fn</code>仅仅是因为目前Rust的Box并不支持Box<FnOnce>(),并且我并不打算用<code>BoxFn</code>来增加文章的复杂度。
对于每一个线程,Job大概是这样使用的:</p>
<div class="highlight"><pre><span></span><code><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">get_a_job</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">send_to_main_thread</span><span class="p">(</span><span class="n">f</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>我们每个线程都会不停的尝试获取job,执行它,然后获取下一个。相信到这里已经有不少人发现了问题:这个子线程似乎永远都无法结束。因为<code>get_a_job</code>显然只能返回Ok或阻塞——如果当Channel为空它就出错的话,一旦没有新任务,所有的子线程都会退出,之后这个ThreadPool就再也没法用了。于是,为了让这个子线程能退出,我们给Job添加一个用来表示结束的状态:</p>
<div class="highlight"><pre><span></span><code><span class="k">enum</span> <span class="nc">Message</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">Work</span><span class="p">(</span><span class="nb">Box</span><span class="o"><</span><span class="nb">Fn</span><span class="p">()</span><span class="w"> </span>-> <span class="nc">T</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="o">'</span><span class="nb">static</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">Terminate</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">type</span> <span class="nc">Job</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Message</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>这样,一旦Job是<code>Terminate</code>,线程就知道ThreadPool已经准备退出了。于是,我们可以把子线程写成这样:</p>
<div class="highlight"><pre><span></span><code><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">get_a_job</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">f</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">Message</span>::<span class="n">Work</span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">f</span><span class="p">(),</span><span class="w"></span>
<span class="w"> </span><span class="n">Message</span>::<span class="n">Terminate</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="k">break</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>有了这个定义之后,让我们来看看ThreadPool该怎么设计。</p>
<p>从前一节的图片里,我们知道这个ThrealPolo需要两个Channel,一个用来接受新Job,另一个用来发送Job的返回结果,因此,ThrealPool可以写成这样:</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span> <span class="nc">ThreadPool</span><span class="w"> </span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">sender</span>: <span class="nc">mpsc</span>::<span class="n">Sender</span><span class="o"><</span><span class="n">Job</span><span class="o"><</span><span class="n">T</span><span class="o">>></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="n">result</span>: <span class="nc">mpsc</span>::<span class="n">Receiver</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">threads</span>: <span class="nb">Vec</span><span class="o"><</span><span class="nb">Option</span><span class="o"><</span><span class="n">thread</span>::<span class="n">JoinHandle</span><span class="o"><</span><span class="p">()</span><span class="o">>>></span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这里,我们用了<code>std::sync::mpsc</code>,而threads则是用来储存所有子线程的,毕竟在结束的时候我们还要关闭他们的句柄。下面,我们尝试生成这个ThreadPool。由于子线程数是静态的,几乎所有的工作都可以在<code>new</code>里面完成,我们的大致思路是这样:</p>
<ul>
<li>生成两个Channel, A和B,A用来接受Job,B用来发送结果</li>
<li>生成n个线程,每一个线程里:</li>
<li>保留A的Receiver和B的Sender</li>
<li>通过A的Receiver接受Job,执行,通过Sender发送</li>
</ul>
<p>看起来并不是很难,让我们把它变成代码:</p>
<div class="highlight"><pre><span></span><code><span class="k">impl</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="n">ThreadPool</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="k">where</span><span class="w"> </span><span class="n">T</span>: <span class="nb">Send</span> <span class="o">+</span><span class="w"> </span><span class="o">'</span><span class="nb">static</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="o">....</span><span class="w"></span>
<span class="w"> </span><span class="c1">// n是线程数,s是每一个线程获取新Job时的等待时间</span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">n</span>: <span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">s</span>: <span class="kt">u64</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">Self</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// 用来接收Job的channel</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="n">tx</span><span class="p">,</span><span class="w"> </span><span class="n">rx</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mpsc</span>::<span class="n">channel</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="c1">// 用来发送结果的Channel</span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="n">tx1</span><span class="p">,</span><span class="w"> </span><span class="n">rx1</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mpsc</span>::<span class="n">channel</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">rx</span><span class="w"> </span>: <span class="nc">Arc</span><span class="o"><</span><span class="n">Mutex</span><span class="o"><</span><span class="n">mpsc</span>::<span class="n">Receiver</span><span class="o"><</span><span class="n">Job</span><span class="o"><</span><span class="n">T</span><span class="o">>>>></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Arc</span>::<span class="n">new</span><span class="p">(</span><span class="n">Mutex</span>::<span class="n">new</span><span class="p">(</span><span class="n">rx</span><span class="p">));</span><span class="w"></span>
</code></pre></div>
<p>首先我们定义了必要的channel,需要注意的是,对于channel,通常我们认为Sender可以有多个,但Receiver只有一个,因此在设计时Sender可以自然应对多线程,但Receiver则不。对于我们这里的情况,可以手动对Receiver加一个互斥锁,在Rust里可以使用<code>std::sync::Mutex</code>。</p>
<p>下面就是生成n个线程了:</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="mi">0</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="n">n</span><span class="p">).</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">rx</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">rx</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">tx1</span><span class="w"> </span>: <span class="nc">mpsc</span>::<span class="n">Sender</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tx1</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">thread</span>::<span class="n">spawn</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">||</span><span class="w"> </span>
<span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">rx</span><span class="p">.</span><span class="n">lock</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">f</span><span class="p">.</span><span class="n">recv</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">f</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">Message</span>::<span class="n">Work</span><span class="p">(</span><span class="n">f</span><span class="p">)</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">r</span><span class="w"> </span>: <span class="nc">T</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">f</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">tx1</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">r</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">},</span><span class="w"></span>
<span class="w"> </span><span class="n">Message</span>::<span class="n">Terminate</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="k">break</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">thread</span>::<span class="n">sleep</span><span class="p">(</span><span class="n">Duration</span>::<span class="n">from_millis</span><span class="p">(</span><span class="n">s</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">))</span><span class="w"></span>
<span class="p">}).</span><span class="n">collect</span>::<span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="n">_</span><span class="o">>></span><span class="p">();</span><span class="w"></span>
</code></pre></div>
<p>这里,<code>thread::spawn</code>用来生成新的线程,每个线程所作的事情就如之前描述的一样。最后,把这一切合起来,就可以得到一个ThreadPool了:</p>
<div class="highlight"><pre><span></span><code><span class="n">ThreadPool</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">sender</span>: <span class="nc">tx</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">result</span>: <span class="nc">rx1</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">threads</span>: <span class="nc">v</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>有了这个ThreadPool之后,我们还需要一个方法来随时向里添加新Job,不过这件事情非常简单——只需要通过sender发送就够了:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">add</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">f</span>: <span class="nc">Job</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">sender</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">f</span><span class="p">).</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>为了简单,我们并没有处理<code>send</code>返回的错误,在实际应用里,<code>add</code>可以返回<code>Result</code>。</p>
<h2>Finishing and Drop</h2>
<p>实际上,到此为止,大部分工作已经结束了,最后还有一个小事情:ThreadPool没法自己结束,因此我们需要手动实现这一部分。在其他语言里析构可能是理所当然的,不过在Rust里,除了C/C++的Wrapper之外,这种事情并不常见。Rust提供了<code>Drop</code> Trait来实现析构,<code>Drop</code>里的<code>drop</code>函数会在内存释放前自动执行。因此,我们只需要这么做:</p>
<div class="highlight"><pre><span></span><code><span class="k">impl</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="nb">Drop</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">ThreadPool</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">drop</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">&</span><span class="bp">self</span><span class="p">.</span><span class="n">threads</span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">sender</span><span class="p">.</span><span class="n">send</span><span class="p">(</span><span class="n">Message</span>::<span class="n">Terminate</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">threads</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="n">t</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">t</span><span class="p">.</span><span class="n">take</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">t</span><span class="p">.</span><span class="n">join</span><span class="p">().</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>在这里,我们先对每一个线程发送结束命令,然后等待他们结束(<code>join</code>)。在实际应用里,线程可能会因为Job比较耗时而无法处理Terminate命令,Drop也会卡在<code>Join</code>的部分,不过可惜的是Rust的Thread并没有提供non-blocking的方法,因此你可能需要<code>Future-rs</code>来实现non-blocking的join。</p>
<h2>Try It</h2>
<p>到这里,主要部分已经完成了,下面我们来测试一下这个ThreadPool的效果如何:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">tp</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ThreadPool</span>::<span class="n">new</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="mi">100</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">tp</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">Message</span>::<span class="n">Work</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="nb">Box</span>::<span class="n">new</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"Thread: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">);</span><span class="w"> </span><span class="n">i</span><span class="o">*</span><span class="mi">100</span><span class="w"> </span><span class="p">})));</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">c</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">loop</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nb">Ok</span><span class="p">(</span><span class="n">s</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tp</span><span class="p">.</span><span class="n">result</span><span class="p">.</span><span class="n">recv</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"Result: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">s</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">c</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">thread</span>::<span class="n">sleep</span><span class="p">(</span><span class="n">Duration</span>::<span class="n">from_millis</span><span class="p">(</span><span class="mi">10</span><span class="p">));</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">c</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">break</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>线程池有10个子线程,每个Job会输出一行文字,并返回一个数字,编译执行后,结果看起来像这样:</p>
<div class="highlight"><pre><span></span><code><span class="n">Thread</span><span class="o">:</span> <span class="mi">0</span>
<span class="n">Thread</span><span class="o">:</span> <span class="mi">1</span>
<span class="n">Result</span><span class="o">:</span> <span class="mi">0</span>
<span class="n">Result</span><span class="o">:</span> <span class="mi">100</span>
<span class="n">Thread</span><span class="o">:</span> <span class="mi">2</span>
<span class="n">Thread</span><span class="o">:</span> <span class="mi">3</span>
<span class="n">Result</span><span class="o">:</span> <span class="mi">200</span>
<span class="n">Result</span><span class="o">:</span> <span class="mi">300</span>
<span class="n">Thread</span><span class="o">:</span> <span class="mi">4</span>
<span class="n">Thread</span><span class="o">:</span> <span class="mi">5</span>
<span class="n">Thread</span><span class="o">:</span> <span class="mi">6</span>
<span class="n">Result</span><span class="o">:</span> <span class="mi">400</span>
<span class="n">Result</span><span class="o">:</span> <span class="mi">500</span>
<span class="n">Result</span><span class="o">:</span> <span class="mi">600</span>
<span class="o">......</span>
</code></pre></div>
<h2>Conclusion</h2>
<p>在文章里,我尽量平白的描述这个过程,不过如果在没有基础的情况下自己去写的话,可能会遇到很多有趣的问题,比如:实际上<code>thread.join</code>会消耗掉JoinHandle的,因此,如果用<code>Vec<JoinHandle></code>来储存线程句柄的话,会无法编译通过——因为<code>drop</code>是by ref的。在这篇文章里,我用了跟<a href="https://doc.rust-lang.org/book/">The Book</a>同样的方法:添加一个Option来解决这个问题。不过,The Boox并不代表着最好的解决办法,比如我们还可以:</p>
<div class="highlight"><pre><span></span><code><span class="k">while</span> <span class="nv">let</span> <span class="nv">Some</span><span class="ss">(</span><span class="nv">e</span><span class="ss">)</span> <span class="o">=</span> <span class="nv">self</span>.<span class="nv">threads</span>.<span class="nv">pop</span><span class="ss">()</span> {
<span class="nv">e</span>.<span class="nv">join</span><span class="ss">()</span>.<span class="nv">unwrap</span><span class="ss">()</span><span class="c1">;</span>
}
</code></pre></div>
<p>对我来说,这个办法看起来更加简洁和优雅——但不知为什么The Book没有这么做。</p>Formal Concept Analysis with Rust (1) - Introduction2016-08-16T00:00:00+09:002016-08-16T00:00:00+09:00Horsaltag:www.horsal.dev,2016-08-16:/formal-concept-analysis-with-rust-1-introduction.html<h2>Introduction</h2>
<p>不知不觉,从开始使用Rust到现在已经有接近一年了,Rust也渐渐成为了目前用起来感觉最舒服的语言。虽然Rust号称因为有确定性析构,所以不会在内存释放上有效率损失,不过在实际使用中,总是能够感觉到再有大量小内存分配和释放的时候运行效率和内存效率都比C要稍微低一些。这次,让我们用Rust通过实现一套<a href="https://en.wikipedia.org/wiki/Formal_concept_analysis">形式概念分析</a>的算法,来看看Rust在全列举程序上的表现到底如何。</p>
<h2>Formal Concept Lattice</h2>
<p>这次我们将把重心放在实现上,对与Formal Concept的我们只介绍在程序中需要使用的特征,如果打算了解更详细的理论内容,可以参考Lattice Theory的综述性论文,比如:</p>
<blockquote>
<blockquote>
<p>Wille, Rudolf. </p>
<p>"Restructuring lattice theory: an approach based on hierarchies of concepts." </p>
<p>Ordered sets. Springer Netherlands, 1982. 445-470.</p>
</blockquote>
</blockquote>
<p>简单来说,对于一组文章<span class="math">\(D={d_1, d_2, \dots, d_n}\)</span>,和一些事先提取出来的文字<span class="math">\(W …</span></p><h2>Introduction</h2>
<p>不知不觉,从开始使用Rust到现在已经有接近一年了,Rust也渐渐成为了目前用起来感觉最舒服的语言。虽然Rust号称因为有确定性析构,所以不会在内存释放上有效率损失,不过在实际使用中,总是能够感觉到再有大量小内存分配和释放的时候运行效率和内存效率都比C要稍微低一些。这次,让我们用Rust通过实现一套<a href="https://en.wikipedia.org/wiki/Formal_concept_analysis">形式概念分析</a>的算法,来看看Rust在全列举程序上的表现到底如何。</p>
<h2>Formal Concept Lattice</h2>
<p>这次我们将把重心放在实现上,对与Formal Concept的我们只介绍在程序中需要使用的特征,如果打算了解更详细的理论内容,可以参考Lattice Theory的综述性论文,比如:</p>
<blockquote>
<blockquote>
<p>Wille, Rudolf. </p>
<p>"Restructuring lattice theory: an approach based on hierarchies of concepts." </p>
<p>Ordered sets. Springer Netherlands, 1982. 445-470.</p>
</blockquote>
</blockquote>
<p>简单来说,对于一组文章<span class="math">\(D={d_1, d_2, \dots, d_n}\)</span>,和一些事先提取出来的文字<span class="math">\(W={w_1, w_2, \dots, w_n}\)</span>, 每一篇文章都可以包含若干文字,同时每一个文字可以属于若干篇文章。基于这个关系,我们可以想象:如果文章和文字的集合选取得当,那么我们可以得到一个文章-文字集合<span class="math">\(C=(A,B), A \subseteq D, B \subseteq W\)</span>,其中A中文章的共通文字集合恰好是B,而反之,所有包含B的文章集合又恰好是A。用聚类的语言来说,就是我们根据特征B选取了一个Cluster A。那么,对于所有可能的这种<span class="math">\((A,B)\)</span>的集合,则就是一个根据文字对文章进行的聚类分析。而这里,每一个<span class="math">\(C=(A,B)\)</span>称为一个<em>Formal Concept</em>(形式概念)。</p>
<p>现在让我们来思考两个形式概念之间可能存在的关系,对于两个形式概念<span class="math">\(C_1 = (A_1, B_1)\)</span>, <span class="math">\(C_2 = (A_2, B_2)\)</span>,如果<span class="math">\(A_1 \subset A_2\)</span>,也就是说<span class="math">\(A_2\)</span>除了有<span class="math">\(A_1\)</span>的文章之外,还包含了别的文章,那么显然的,<span class="math">\(A_2\)</span>所有文章的共通文字肯定会更少(<strong>注意:而不是不会更多</strong>,原因是如果<span class="math">\(B_2\)</span>跟<span class="math">\($B_1\)</span>是一样的,那么根据Formal Concept的性质,我们可以知道<span class="math">\(A_1 = A_2\)</span>)。因此,我们可以根据<span class="math">\(A\)</span> 或者 <span class="math">\(B\)</span> 的包含关系来创建一个<a href="https://en.wikipedia.org/wiki/Lattice_(order)">束</a>,这个束成为Formal Concept Lattic。</p>
<p>当我们有了一个Formal Concept Lattice,就相当于我们把文章进行了分层聚类,最顶层包含了所有的文章和它们共通的文字(文字可能是空集),越下面共通的文字越多而每一个Formal Concept包含的文章越少,直到最下面,我们得到所有共通的文字和包含这些文字的文章的集合。在实际应用中,Formal Concept Lattice可以用来分析话题的关注度,提取抽象概念,以及通过额外的关联(比如文字合集的互信息量)来提取关键信息。不过,在这篇文章里,我们不会设计应用,而是把重心放在构建Formal Concept Lattice本身上。</p>
<h2>Building Formal Concept Lattice</h2>
<p>假设我们已经有了文章和单词的包含关系,比如像下面的表:</p>
<table>
<thead>
<tr>
<th>doc\word</th>
<th>word1</th>
<th>word2</th>
<th>word3</th>
<th>word 4</th>
<th>...</th>
</tr>
</thead>
<tbody>
<tr>
<td>doc1</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>0</td>
<td>...</td>
</tr>
<tr>
<td>doc2</td>
<td>1</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>...</td>
</tr>
<tr>
<td>doc3</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>1</td>
<td>...</td>
</tr>
<tr>
<td>doc4</td>
<td>1</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>...</td>
</tr>
<tr>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
</tr>
</tbody>
</table>
<p>每一行代表了<em>一篇文章-所有单词</em>的对应关系,0代表文章里没有出现这个单词,1则反之。要建立一个Formal Concept Lattice,只需要完成下面两件事:</p>
<ul>
<li>列举出所有可能的Formal Concept</li>
<li>对于所有可能的Formal Concept,将它们按照包含关系建立成Lattice。</li>
</ul>
<p>由于并不是任意的<em>文章集合-单词集合</em>都是Formal Concept,因此我们需要筛选出那些满足性质的组合。比如,在上面这个表中, <span class="math">\((\{doc1, doc2\}, \{word2\})\)</span>并不是一个Formal Concept,原因是虽然word2是doc1和doc2的共通单词,但实际上doc1,doc2和doc3共同包含word2,因此正确的Formal Concept应该是<span class="math">\((\{doc1, doc2, doc3\}, \{word2\})\)</span>。</p>
<p>那么假设我们有了文章集合<span class="math">\(\{doc1, doc2\}\)</span>,那么该如果获得一个Formal Concept?答案很简单,由于Formal Concept是唯一的,因此我们首先寻找<span class="math">\(\{doc1, doc2\}\)</span>的共通单词,这里是<span class="math">\(\{word2\}\)</span>,然后,我们只需要寻找所有包含<span class="math">\(\{word2\}\)</span>的文章即可,在这里,我们可以看到是<span class="math">\(\{doc1, doc2, doc3\}\)</span>,这样,我们就能获得一个Formal Concept。而这个寻找单词-寻找文章的操作,在Formal Concept Analysis里称作闭包(Closure)。很容易想象,有了闭包操作,我们只需要对所有的可能的文章组合取闭包,然后去掉重复,就可以得到所有的Formal Concept了。这就是目前所有Formal Concept Enumeration算法的基本思想。总结起来,算法可以简单归纳成一个简单的深度优先搜索:</p>
<div class="highlight"><pre><span></span><code><span class="nv">Function</span> <span class="nv">EnumerateConcept</span><span class="ss">(</span><span class="nv">concept</span><span class="ss">)</span>
<span class="k">for</span> <span class="nv">any</span> <span class="nv">document</span> <span class="nv">not</span> <span class="nv">in</span> <span class="nv">concept</span>.<span class="nv">concept</span> <span class="k">do</span>
<span class="nv">concept</span>.<span class="nv">document</span> <span class="o">=</span> <span class="nv">concept</span>.<span class="nv">document</span><span class="o">+</span> <span class="nv">document</span>
<span class="nv">new_concept</span> <span class="o">=</span> <span class="nv">closure</span><span class="ss">(</span><span class="nv">concept</span>.<span class="nv">document</span><span class="ss">)</span>
<span class="k">if</span> <span class="nv">new_concept</span> <span class="nv">is</span> <span class="nv">not</span> <span class="nv">duplicate</span> <span class="k">do</span>
<span class="nv">add</span> <span class="nv">new_concept</span> <span class="nv">to</span> <span class="nv">concept_list</span>
<span class="k">end</span> <span class="k">do</span>
<span class="nv">EnumerateConcept</span><span class="ss">(</span><span class="nv">new_concept</span><span class="ss">)</span>
<span class="k">end</span> <span class="k">do</span>
<span class="k">end</span> <span class="nv">func</span>
</code></pre></div>
<p>有了所有的Formal Concept之后,我们可以着手开始建立Lattice了,不过,由于Lattice包含关系的限制,我们并不能想图一样简单的对所有Formal Concept Pair遍历然后创建连接,而是需要先把Formal Concept按照文章数的多少排序,然后按照大小顺序来简历Lattice。如果你对Lattice不熟,那么有可能不明白为什么,没关系,让我们看一个例子:</p>
<p>假设我们已经有三个Formal Concept(只列出的文章集合):
<span class="math">\(C_1 = \{doc1, doc2, doc3\}\)</span>, <span class="math">\(C_2 = \{doc1, doc3\}\)</span>, <span class="math">\(C_3 = \{doc3\}\)</span>,
那么Lattice应该是<span class="math">\(C_1 - C_2 - C_3\)</span>,其中,<span class="math">\(C_1\)</span> 和 <span class="math">\(C_3\)</span>不能直接相连,因为按照顺序关系,其中还有一个<span class="math">\(C_2\)</span>。</p>
<p>看到这里,你可能已经明白了,如果我们遍历所有可能的组合,那么我们并不知道这两个Concept之间是否有别的“中间节点”,但是,由于包含关系的性质(较小的一定无法包含较大的),我们只要按照大小排序,那么对于任何两个Concept,可能的中间节点总是在它们之间 —— 这样我们就判定两个节点是否能直接相连了。更明确一些,实际上,在列举过程中我们可以自然的得到足够的信息,因此并没有必要真的去排序然后遍历,这些我们会在后面的文章里讨论。</p>
<h3>Summary</h3>
<p>在这篇文章里我们简单介绍的Formal Concept Lattice的性质和相关的算法。在接下来的文章中,我们将会依次实现下面的功能:</p>
<ul>
<li>Formal Concept的数据结构</li>
<li>Formal Concept的列举算法</li>
<li>Formal Concept Lattice的创建算法</li>
<li>基于Formal Concept Lattice的简单数据挖掘</li>
</ul>
<p>在下一章里,我们将会着手实现最基本的数据结构,在实现中,我们会遇到并解决一些Rust独有的Borrow-checker问题。</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Description of LZ772016-02-26T00:00:00+09:002016-02-26T00:00:00+09:00Horsaltag:www.horsal.dev,2016-02-26:/description-of-lz77.html<h2>Basic Definitions</h2>
<p>In this article, <span class="math">\(D\)</span> is used to represent an i-elements data array. <span class="math">\(d_0, d_1, ... , d_i\)</span> are the elements of <span class="math">\(D\)</span>.
Any continuous subset of <span class="math">\(D\)</span> is called <strong>slices</strong>, represented by <span class="math">\(S(i, j)\)</span>, where <span class="math">\(i\)</span> and <span class="math">\(j\)</span> is the start and end position.
For two slices <span class="math">\(S_1\)</span> and …</p><h2>Basic Definitions</h2>
<p>In this article, <span class="math">\(D\)</span> is used to represent an i-elements data array. <span class="math">\(d_0, d_1, ... , d_i\)</span> are the elements of <span class="math">\(D\)</span>.
Any continuous subset of <span class="math">\(D\)</span> is called <strong>slices</strong>, represented by <span class="math">\(S(i, j)\)</span>, where <span class="math">\(i\)</span> and <span class="math">\(j\)</span> is the start and end position.
For two slices <span class="math">\(S_1\)</span> and <span class="math">\(S_2\)</span>, if <span class="math">\(S_1\)</span> and <span class="math">\(S_2\)</span> are the same, this pair of slices is called a match. <span class="math">\(|S_1|\)</span> and <span class="math">\(|S_2|\)</span>
are their lengths.
If data <span class="math">\(D\)</span> and a slice <span class="math">\(S\)</span> have more than <span class="math">\(k\)</span> same elements from the start, we say <span class="math">\(D\)</span> and <span class="math">\(S\)</span> is a <span class="math">\(k\)</span>-match.</p>
<h2>General Description</h2>
<p>Given a data array <span class="math">\(D\)</span>, LZ77 compresses the data by replacing the repeated part with a position and length indactor.
For example, if we have the following data:</p>
<div class="highlight"><pre><span></span><code>1 2 3 4 5 1 2 3 4 8 9 0
</code></pre></div>
<p>It is easy to point out that the slice <code>1 2 3 4</code> appears twice. The slice tt the very first is called <span class="math">\(S_1(0,3)\)</span> and another is <span class="math">\(S_2(5,8)\)</span>.
It means that there is no need to store this slice twice,
but a slice and just a indactor pointing where we can find this silce. Here, to index <span class="math">\(S_2(i_2,j_2)\)</span> by <span class="math">\(S_1(i_1,j_1)\)</span>,
we use the following indactor:</p>
<div class="highlight"><pre><span></span><code>i_2 - i_1, |S_2|
</code></pre></div>
<p>The first term of this indactor represents the distance of <span class="math">\(S_1\)</span> from current position. The second term is the length of <span class="math">\(S_2\)</span>.
Then, the example we give at first can be compressed to:</p>
<div class="highlight"><pre><span></span><code>1 2 3 4 5 5 4 8 9 0
</code></pre></div>
<p>However, by looking at the above compressed data, we do not know which part is compressed and which part is raw. Thus, to address this
problem, we introduce the <strong>compress flag</strong>. Compress flag is a 8-bit byte <span class="math">\([b_1, b_2, b_3, b_4, b_5, b_6, b_7, b_8]\)</span>. Each bit is
assigned to a part of following data. If the bit is <span class="math">\(0\)</span>, this part is compressed, and vice versa. One thing need to be pointed out is that we always
check the bits from low to high, so <span class="math">\(b_8\)</span> is always assigned to the first part following this flag.</p>
<p>Apparently, the simplest way of using <strong>compress flag</strong> is as follows:</p>
<ul>
<li>If current bit is <span class="math">\(1\)</span>, the following one byte in raw data.</li>
<li>If current bit is <span class="math">\(0\)</span>, the following two byte represent a indactor.</li>
</ul>
<p>Now we can rewrite the compress example:</p>
<div class="highlight"><pre><span></span><code><span class="mf">0</span><span class="n">xDF</span> <span class="mf">1</span> <span class="mf">2</span> <span class="mf">3</span> <span class="mf">4</span> <span class="mf">5</span> <span class="mf">4</span> <span class="mf">8</span> <span class="mf">9</span> <span class="mf">0</span><span class="n">x01</span> <span class="mf">0</span>
<span class="mf">0</span><span class="n">xDF</span> <span class="o">=</span> <span class="err">`</span><span class="mf">0</span><span class="n">b11011111</span><span class="err">`</span><span class="mf">.</span>
<span class="mf">0</span><span class="n">x01</span> <span class="o">=</span> <span class="err">`</span><span class="mf">0</span><span class="n">b00000001</span><span class="err">`</span><span class="mf">.</span>
</code></pre></div>
<p>You can manually decompress and validate this compressed array.</p>
<h2>Algorithm</h2>
<p>We have talked about the general idea. However, there are several remaining problems:</p>
<ul>
<li>How far should we look back?</li>
<li>What is the bound of the length of match?</li>
<li>How do we know that we reach the end of data?</li>
</ul>
<p>Generally speaking, the shorter we look back, the faster the algorithm will be. The most common solution is set the upbound of
the distance we can look back. If we reach the max distance and there is still no match, then current byte is marked as raw.
We can also say the totally same thing for the upper bound of the length of match. However, for lower bound, things become
different, because the indactor contains two bytes. If we compress a one-byte slice, we will get a longer data.
Thus, the lower bound is usually fixed as 3, which is just larger than the length of indactor.
The final problem is that how should we mark this stream as end. There are variant ways to achieve this. Here we will use
a simple way: a 0-length compressed slice with position at 0 represents the end of data. This type of marks is safe because
we can not get a 0-length compressed slice. Back to the example in previous section, now it becomes:</p>
<div class="highlight"><pre><span></span><code><span class="mf">0</span><span class="n">xDF</span> <span class="mf">1</span> <span class="mf">2</span> <span class="mf">3</span> <span class="mf">4</span> <span class="mf">5</span> <span class="mf">4</span> <span class="mf">8</span> <span class="mf">9</span> <span class="mf">0</span><span class="n">x01</span> <span class="mf">0</span> <span class="mf">0</span> <span class="mf">0</span>
<span class="mf">0</span><span class="n">xDF</span> <span class="o">=</span> <span class="err">`</span><span class="mf">0</span><span class="n">b11011111</span><span class="err">`</span><span class="mf">.</span>
<span class="mf">0</span><span class="n">x01</span> <span class="o">=</span> <span class="err">`</span><span class="mf">0</span><span class="n">b00000001</span><span class="err">`</span><span class="mf">.</span>
</code></pre></div>
<h3>Compress</h3>
<p>Now we will actually talk about the compress/decompress method. Slightly different from our previous discuss, when we try to find
a match, we do not search in the raw data, but in a special slice, called <strong>window</strong>. A windows <span class="math">\(W\)</span> is a fixed-size slice. The content
of window can be changed during the compression/decompression. The advantage of using window is that:</p>
<ul>
<li>We do not need to touch the data.</li>
<li>Window is a better choice when we are working on stream data.</li>
<li>We can set the initial value of window according dictonary to get better compression ratio.</li>
</ul>
<p>Here we give the algorithms as follows:</p>
<ul>
<li>Initialize window.</li>
<li>Set current position p = 0, current window position w = 0</li>
<li>Initialize flag byte.</li>
<li>for 1 to 8 do:<ul>
<li>find match between data and window from current position.</li>
<li>if match exists<ul>
<li>set current bit to 0, write the position and length of match</li>
</ul>
</li>
<li>else <ul>
<li>set current bit to 1, write current byte.</li>
</ul>
</li>
</ul>
</li>
<li>Repeat until the end of data.</li>
<li>Write the end mark.</li>
</ul>
<h3>Decompress</h3>
<p>Decompress is based on the same idea of compress. Thus, we won't talk about it too much but just give the algorithm:</p>
<ul>
<li>Initialize window.</li>
<li>Set current position p = 0, current window position w = 0</li>
<li>Read flag byte.</li>
<li>for 1 to 8 do:<ul>
<li>if current bit is 0<ul>
<li>read position and lengeth, copy the slice(position, length) of window to output.</li>
</ul>
</li>
<li>if current bit is 1<ul>
<li>read one byte and write to output.</li>
</ul>
</li>
</ul>
</li>
<li>Repeat until reach the end mark.</li>
</ul>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Dive into Rust(1) - Iterator2016-02-15T00:00:00+09:002016-02-15T00:00:00+09:00Horsaltag:www.horsal.dev,2016-02-15:/dive-into-rust1-iterator.html<p>大概三个月之前,这三个月,我一直在使用Rust。回想起来,在离开Object Pascal家族之后,过去五六年里也已经尝试了很多的新语言,不管是D,还是因为谷歌的名头而开始用的Go,又或者是Nim,Crystal或者Zimbu,都没能让我真正的满意,而<a href="https://github.com/ooc-lang/rock">OOC</a>虽然在语法上让人十分舒服,但它使用了libGC,并且对多线程和Closure的支持并不到位。其实,在改进OOC的过程中,有人提出过给OOC加入RAII,并且尝试把一部分内容分配在了栈上,不过很可惜,OOC的编译器在设计当初并没有考虑这些事情,不到几个小时这种改进就遇到了问题。</p>
<p>在刚刚接触Rust的时候曾经写过一篇<a href="https://www.horsal.dev/a-first-look-at-rust.html">关于Rust的文章</a>,里面阐述了我对Rust的第一印象——省心。把代码写出来,编译器就会指出所有的错误,在这几十天里,我明显感觉到在调试程序时莫名其妙的错误少多了——我还清楚的记得之前在OOC里因为一个Generics的bug导致<code>ArrayList<T></code>里的元素过早被回收,结果就是程序有时候正常,有时候乱码,有时候Segmentation Fault,而这种问题在Rust里几乎不会出现。</p>
<p>当然,Rust这么做的代价就是语法上的繁杂。Rust的语法并不算复杂,大部分关键词的意义都很明确,也没有存在歧义的设计,然而说它繁杂却不为过,一个简单的例子就是在声明一个用到了Borrow的Struct时,编译器要求必须指明所有Borrow的Lifetime,因此定义和声明中就出现了大量的Lifetime标识符 …</p><p>大概三个月之前,这三个月,我一直在使用Rust。回想起来,在离开Object Pascal家族之后,过去五六年里也已经尝试了很多的新语言,不管是D,还是因为谷歌的名头而开始用的Go,又或者是Nim,Crystal或者Zimbu,都没能让我真正的满意,而<a href="https://github.com/ooc-lang/rock">OOC</a>虽然在语法上让人十分舒服,但它使用了libGC,并且对多线程和Closure的支持并不到位。其实,在改进OOC的过程中,有人提出过给OOC加入RAII,并且尝试把一部分内容分配在了栈上,不过很可惜,OOC的编译器在设计当初并没有考虑这些事情,不到几个小时这种改进就遇到了问题。</p>
<p>在刚刚接触Rust的时候曾经写过一篇<a href="https://www.horsal.dev/a-first-look-at-rust.html">关于Rust的文章</a>,里面阐述了我对Rust的第一印象——省心。把代码写出来,编译器就会指出所有的错误,在这几十天里,我明显感觉到在调试程序时莫名其妙的错误少多了——我还清楚的记得之前在OOC里因为一个Generics的bug导致<code>ArrayList<T></code>里的元素过早被回收,结果就是程序有时候正常,有时候乱码,有时候Segmentation Fault,而这种问题在Rust里几乎不会出现。</p>
<p>当然,Rust这么做的代价就是语法上的繁杂。Rust的语法并不算复杂,大部分关键词的意义都很明确,也没有存在歧义的设计,然而说它繁杂却不为过,一个简单的例子就是在声明一个用到了Borrow的Struct时,编译器要求必须指明所有Borrow的Lifetime,因此定义和声明中就出现了大量的Lifetime标识符,同时,对于一个类型和它的Borrow,Trait是不能通用的,也就是说就算对一个类型实现了某个Trait,如果需要用到它的Borrow,那么还需要重新实现一次,然而在绝大多数情况下,Borrow的声明不过是对应类型的复制粘贴而已。虽然从安全性的角度来看,这么做是理所当然的,但至于为了安全性是不是值得作出这点牺牲就因人而异了。</p>
<p>好吧,对于Rust这种做法的争论已经够多了,这里也不需要再来一次,因此让我们结束这个话题,来看看Rust的方方面面。</p>
<h2>An Easy Start</h2>
<p>让我们先从一个尽可能简单的例子开始,假设有一个struct储存了一些数据,现在需要对它实现一个Iterator,不过这个Iterator有点特殊,struct里的数据需要按照一个函数<code>index(t+1) = f(index(t))</code>的顺序取出来。用代码来说,就像是这个样子:</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span> <span class="nc">MyIter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">data</span>: <span class="kp">&</span><span class="o">'</span><span class="na">a</span> <span class="p">[</span><span class="kt">u8</span><span class="p">],</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">rdata</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="mi">10</span><span class="p">,</span><span class="w"> </span><span class="mi">20</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="mi">9</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">];</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">MyIter</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">data</span>: <span class="kp">&</span><span class="nc">rdata</span><span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>下面需要对<code>MyIter</code>实现Iterator了,当然,为了实现Iterator我们需要一个Index来指示当前的位置。最直接的做法就是在MyIter里记录当前位置:</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span> <span class="nc">MyIter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">data</span>: <span class="kp">&</span><span class="o">'</span><span class="na">a</span> <span class="p">[</span><span class="kt">u8</span><span class="p">],</span><span class="w"></span>
<span class="w"> </span><span class="n">position</span>: <span class="kt">usize</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>不过这么做是非常危险的,原因很简单,如果把<code>position</code>放进<code>MyIter</code>里,我们很难保证position总是正确的。<code>position</code>可以被除了Iterator之外的函数修改,并且在一次遍历完成之后还要想法法把<code>position</code>置0——这些都会导致Bug出现,因此在大部分程序里直接针对MyIter声明position并不是一个好办法。很自然的,我们可以把二者分开:</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span> <span class="nc">MyIter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">data</span>: <span class="kp">&</span><span class="o">'</span><span class="na">a</span> <span class="p">[</span><span class="kt">u8</span><span class="p">],</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">Iter</span><span class="w"> </span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">iter</span>: <span class="nc">MyIter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">position</span>: <span class="kt">usize</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="nb">IntoIterator</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">MyIter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">type</span> <span class="nc">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">u8</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">type</span> <span class="nc">IntoIter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Iter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">into_iter</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">Self</span>::<span class="n">IntoIter</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">MyIterIntoIterator</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">iter</span>: <span class="nc">self</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">position</span>: <span class="mi">0</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="nb">Iterator</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">Iter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">type</span> <span class="nc">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">u8</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">next</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Option</span><span class="o"><</span><span class="bp">Self</span>::<span class="n">Item</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">positoion</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">f</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">position</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">position</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">iter</span><span class="p">.</span><span class="n">data</span><span class="p">.</span><span class="n">len</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">iter</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="bp">self</span><span class="p">.</span><span class="n">position</span><span class="p">])</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">None</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>在这里,我们新定义了一个<code>Iter</code>,然后我们仅对Iter实现Iterator,这样,在每次遍历MyIter时,其实是对新创建的Iter进行遍历,这样就不需要担心position不够安全了。同时,在Rust中,<code>for i in data</code>这种类新的循环不但会被自动unwrap成基于<code>Iterator.next</code>的循环,如果<code>data</code>实现了<code>IntoIterator</code>Trait那么编译器还会自动把data转换成<code>data.into_iter()</code>。因此,我们需要做的就是对<code>MyIter</code>实现<code>IntoIterator</code>,然后对<code>Iter</code>实现<code>Iterator</code>就够了。</p>
<p>到这里,最初的程序就已经可以运行了。不过这里还残留着一个问题,对比<code>vec</code>,对于一个<code>vec</code>,<code>for each</code>循环可以这么写:</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="fm">vec!</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">5</span><span class="p">];</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="c1">// do your work</span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这种写法会消耗掉data,如果在循环之后试图使用data,则会出现<code>use of moved variable</code>错误。当我们不打算消耗掉data的时候,可以对data的borrow进行循环:</p>
<div class="highlight"><pre><span></span><code><span class="kd">let</span><span class="w"> </span><span class="n">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="fm">vec!</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="mi">5</span><span class="p">];</span><span class="w"></span>
<span class="k">for</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="o">&</span><span class="n">data</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这样data仅仅是一个borrow,在循环结束后依然可以对data进行别的操作。返回到最初的<code>MyIter</code>例子中,现在没法对MyIter的borrow进行循环,因为<code>&MyIter</code>并没有实现<code>IntoIteroter</code>,也没有<code>Iteroter</code>。为了实现这个功能,我们还需要实现一个对Borrow的版本:</p>
<div class="highlight"><pre><span></span><code><span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="nb">IntoIterator</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="o">&'</span><span class="na">a</span><span class="w"> </span><span class="n">MyIter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">type</span> <span class="nc">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">u8</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">type</span> <span class="nc">IntoIter</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">MyIterIntoIteratorRef</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">into_iter</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">Self</span>::<span class="n">IntoIter</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">MyIterIntoIteratorRef</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">iter</span>: <span class="nc">self</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">position</span>: <span class="mi">0</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">MyIterIntoIteratorRef</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">iter</span>: <span class="kp">&</span><span class="o">'</span><span class="na">a</span> <span class="nc">MyIter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">position</span>: <span class="kt">usize</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="nb">Iterator</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">MyIterIntoIteratorRef</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">type</span> <span class="nc">Item</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="kt">u8</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">fn</span> <span class="nf">next</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Option</span><span class="o"><</span><span class="bp">Self</span>::<span class="n">Item</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">positoion</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">f</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">position</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">position</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">iter</span><span class="p">.</span><span class="n">data</span><span class="p">.</span><span class="n">len</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">Some</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">iter</span><span class="p">.</span><span class="n">data</span><span class="p">[</span><span class="bp">self</span><span class="p">.</span><span class="n">position1</span><span class="p">])</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="nb">None</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>在这里,仅仅是再之前的类型前面加上一个<code>&</code>而已,但很可惜,由于二者被当做不同的类型,所有相关的函数都得重新实现一遍,里面包括用于获得下一个index的<code>self.f</code>。再仔细考虑一下,如果我们还打算对<code>&mut MyIter</code>做循环的话,还要实现一个<code>&mut MyIter</code>的版本,然而这三个版本的代码几乎是相同的,但因为种种限制,我们却几乎无法作出更好的抽象。实际上,在标准库里每一个支持for each的类型也都实现了这三种Iterator。不过,需要注意的是,这里的例子仅仅是为了展示Iterator而有意采用了类似libstd里的写法,在实际应用当中,绝大多数情况下都没有必要自行实现这些内容,比如实际上<code>&[T]</code>已经实现了<code>iter(&self)</code>,因此我们只需要在<code>into_iter</code>里直接调用<code>iter()</code>就足够了。</p>A First Look at Rust2015-11-24T00:00:00+09:002015-11-24T00:00:00+09:00Horsaltag:www.horsal.dev,2015-11-24:/a-first-look-at-rust.html<p>过去的一年半多,我一直沉迷与<a href="https://ooc-lang.org">OOC</a>,原因倒是很简单,OOC是目前为止我所能见到的最容易理解和最容易书写的语言。并且另外一个极其重要的地方是,它可以编译成C代码。<strong>编译成C代码</strong>,也就意味着优化可以交给高度发展的C语言编译器来做,听起来似乎适合十分高效的方法。</p>
<p>最近几年类似的语言越来越多,从很久很久之前就存在却一直没出名的<code>Haxe</code>,还有最近的<code>Nim-lang</code>,以及采用了类似ruby语法的<code>Crystal</code>,甚至包括编译成C++的<code>felix</code>。这些语言都号称自己考虑了速度(运行速度),至少从编译成C/C++的层面上。</p>
<p>可惜的是,在改进OOC编译器<a href="https://github.com/fasterthanlime/rock">rock</a>的过程中,我遇到了越来越多的问题,这些问题让喜欢速度的人泄气。一个最明显的事情是,这些语言几乎都用了GC,不论是libGC还是自己写的,并且更重要的是,很多语言特性是基于GC设计的——比如闭包,比如iterator的unwrap,在有没GC的情况下,这些东西的设计要复杂的多。在OOC里,由于Generics不是Template,更多的东西开始依存GC,在用了它一年后,当我真正开始在工作里使用的时候,这些问题开始出现,我开始打算关闭GC,但很显然这是不可能的。编译器会把一切搞不清楚的事情踢给GC。</p>
<p>在这个时候 …</p><p>过去的一年半多,我一直沉迷与<a href="https://ooc-lang.org">OOC</a>,原因倒是很简单,OOC是目前为止我所能见到的最容易理解和最容易书写的语言。并且另外一个极其重要的地方是,它可以编译成C代码。<strong>编译成C代码</strong>,也就意味着优化可以交给高度发展的C语言编译器来做,听起来似乎适合十分高效的方法。</p>
<p>最近几年类似的语言越来越多,从很久很久之前就存在却一直没出名的<code>Haxe</code>,还有最近的<code>Nim-lang</code>,以及采用了类似ruby语法的<code>Crystal</code>,甚至包括编译成C++的<code>felix</code>。这些语言都号称自己考虑了速度(运行速度),至少从编译成C/C++的层面上。</p>
<p>可惜的是,在改进OOC编译器<a href="https://github.com/fasterthanlime/rock">rock</a>的过程中,我遇到了越来越多的问题,这些问题让喜欢速度的人泄气。一个最明显的事情是,这些语言几乎都用了GC,不论是libGC还是自己写的,并且更重要的是,很多语言特性是基于GC设计的——比如闭包,比如iterator的unwrap,在有没GC的情况下,这些东西的设计要复杂的多。在OOC里,由于Generics不是Template,更多的东西开始依存GC,在用了它一年后,当我真正开始在工作里使用的时候,这些问题开始出现,我开始打算关闭GC,但很显然这是不可能的。编译器会把一切搞不清楚的事情踢给GC。</p>
<p>在这个时候,恰好Rust站了出来,静态析构,没有野指针…… 简直就是为有着Compile to C语言苦恼的人设计的。于是我打算在这篇文章里瞄一眼Rust,来看看它是不是我想找的东西。</p>
<p>首先从官方的例子开始,打开<a href="https://rust-lang.org">Rust的主页</a>就会看到。直接拷贝过来,就是这个样子:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">program</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"+ + * - /"</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">accumulator</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">token</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">program</span><span class="p">.</span><span class="n">chars</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">token</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="sc">'+'</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">accumulator</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="sc">'-'</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">accumulator</span><span class="w"> </span><span class="o">-=</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="sc">'*'</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">accumulator</span><span class="w"> </span><span class="o">*=</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="sc">'/'</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">accumulator</span><span class="w"> </span><span class="o">/=</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="cm">/* ignore everything else */</span><span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"The program </span><span class="se">\"</span><span class="s">{}</span><span class="se">\"</span><span class="s"> calculates the value {}"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">program</span><span class="p">,</span><span class="w"> </span><span class="n">accumulator</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>看起来跟现代语言并没有太大差别,至少这个例子还算比较容易阅读,让我们来把这段代码改成类似函数式的写法:</p>
<div class="highlight"><pre><span></span><code><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">program</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"+ + * - /"</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">program</span><span class="p">.</span><span class="n">chars</span><span class="p">().</span><span class="n">fold</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">x1</span><span class="w"> </span><span class="o">|</span><span class="w"> </span>
<span class="w"> </span><span class="k">match</span><span class="w"> </span><span class="n">x1</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="sc">'+'</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="sc">'-'</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="sc">'*'</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="sc">'/'</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="o">=></span><span class="w"> </span><span class="n">x</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"The program </span><span class="se">\"</span><span class="s">{}</span><span class="se">\"</span><span class="s"> calculates the value {}"</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">program</span><span class="p">,</span><span class="w"> </span><span class="n">res</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这段代码对OOC的用户来说相当亲切,它们实在有些相似,比如相同的lambda语法 <code>| arguments | program</code>,几乎相同的match语法<code>match expr { case => expr }</code>。</p>
<p>不过如果仅仅是这样,恐怕Rust不会这么吸引人,下面让我们来看一个稍微复杂点的例子。</p>
<p>这个例子来自<a href="http://benchmarksgame.alioth.debian.org/">Computer Language Benchmark Game</a>的Binary Tree,这也是我最喜欢的一个例子,几乎在了解任何语言时我写的第一个小代码都是Binary Tree。它包含了一些基本的东西——构造体(或类),递归,循环。先来看看我写的Binary Tree,后面会有详细的解说。</p>
<div class="highlight"><pre><span></span><code><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">env</span><span class="p">;</span><span class="w"></span>
<span class="k">struct</span> <span class="nc">Tree</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">left</span>: <span class="nb">Option</span><span class="o"><</span><span class="nb">Box</span><span class="o"><</span><span class="n">Tree</span><span class="o">>></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">right</span>: <span class="nb">Option</span><span class="o"><</span><span class="nb">Box</span><span class="o"><</span><span class="n">Tree</span><span class="o">>></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">item</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="n">Tree</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">depth</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"> </span><span class="n">i</span>: <span class="kt">i32</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">Tree</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">depth</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">Tree</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">item</span><span class="w"> </span>: <span class="nc">i</span><span class="p">,</span><span class="w"> </span><span class="n">left</span>: <span class="nb">None</span><span class="p">,</span><span class="w"> </span><span class="n">right</span>: <span class="nb">None</span> <span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">Tree</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">item</span><span class="w"> </span>: <span class="nc">i</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">left</span>: <span class="nb">Some</span><span class="p">(</span><span class="nb">Box</span>::<span class="n">new</span><span class="p">(</span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">))),</span><span class="w"></span>
<span class="w"> </span><span class="n">right</span>: <span class="nb">Some</span><span class="p">(</span><span class="nb">Box</span>::<span class="n">new</span><span class="p">(</span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="p">))),</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">item_check</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span>-> <span class="kt">i32</span> <span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">item</span><span class="w"> </span><span class="o">+</span><span class="w"> </span>
<span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">left</span><span class="p">.</span><span class="n">as_ref</span><span class="p">().</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">t</span><span class="p">.</span><span class="n">item_check</span><span class="p">()).</span><span class="n">unwrap_or</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="o">-</span><span class="w"></span>
<span class="w"> </span><span class="bp">self</span><span class="p">.</span><span class="n">right</span><span class="p">.</span><span class="n">as_ref</span><span class="p">().</span><span class="n">map</span><span class="p">(</span><span class="o">|</span><span class="w"> </span><span class="n">t</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">t</span><span class="p">.</span><span class="n">item_check</span><span class="p">()).</span><span class="n">unwrap_or</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">const</span><span class="w"> </span><span class="n">MINDEP</span><span class="w"> </span>: <span class="kt">i32</span> <span class="o">=</span><span class="w"> </span><span class="mi">4</span><span class="p">;</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">depth</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">env</span>::<span class="n">args</span><span class="p">().</span><span class="n">nth</span><span class="p">(</span><span class="mi">1</span><span class="p">).</span><span class="n">unwrap_or</span><span class="p">(</span><span class="s">"10"</span><span class="p">.</span><span class="n">to_string</span><span class="p">()).</span><span class="n">parse</span>::<span class="o"><</span><span class="kt">i32</span><span class="o">></span><span class="p">().</span><span class="n">unwrap_or</span><span class="p">(</span><span class="mi">10</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"Running program with depth = {}"</span><span class="p">,</span><span class="w"> </span><span class="n">depth</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">stretch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">depth</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"stretch tree of depth {}</span><span class="se">\t</span><span class="s"> check: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">stretch</span><span class="p">,</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">stretch</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">).</span><span class="n">item_check</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">long_lived</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">depth</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">MINDEP</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="n">depth</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">).</span><span class="n">filter</span><span class="p">(</span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">).</span><span class="n">map</span><span class="p">(</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">|</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">MINDEP</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">),</span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">MINDEP</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">).</span><span class="n">fold</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">xt</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="n">x1</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">xt</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">x1</span><span class="p">).</span><span class="n">item_check</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="n">x1</span><span class="p">).</span><span class="n">item_check</span><span class="p">())));</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">iters</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">check</span><span class="p">)</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}</span><span class="se">\t</span><span class="s"> trees of depth {}</span><span class="se">\t</span><span class="s"> check: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">iters</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">check</span><span class="p">);</span><span class="w"> </span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"long lived tree of depth {}</span><span class="se">\t</span><span class="s"> check: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">depth</span><span class="p">,</span><span class="w"> </span><span class="n">long_lived</span><span class="p">.</span><span class="n">item_check</span><span class="p">());</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这段程序很短,算上空行也不过总共44行,让我们来看看每一部分都有什么有趣的地方。</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span> <span class="nc">Tree</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">left</span>: <span class="nb">Option</span><span class="o"><</span><span class="nb">Box</span><span class="o"><</span><span class="n">Tree</span><span class="o">>></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">right</span>: <span class="nb">Option</span><span class="o"><</span><span class="nb">Box</span><span class="o"><</span><span class="n">Tree</span><span class="o">>></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">item</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>这是一个很容易理解的structure定义,让人高兴的是新语言越来越多的使用pascal式的<code>variable : type</code>而不是难于理解的<code>type variable</code>。下面就是具有Rust特点的东西了,跟C,D等语言不同,递归定义时并没有用类似<code>left: &Tree</code>的形式,原因很简单——left和right有可能是空的,而rust不允许这种空指针。为了解决这个问题,Rust提供了一个叫做<code>Option</code>的特殊类型(enum),如果没有内容,那么<code>Option</code>是<code>None</code>,否则就是<code>Some(T)</code>,这样做的好处是不用再考虑<code>nil.item_check()</code>这种可能引起Segmental fault的形式了。</p>
<p>接下来看到的是<code>Box</code>,当然Box也不过是储存Heap上的一个指针而已,在C++等语言里,<code>Box</code>跟<code>&Tree</code>似乎并没有太大差别。不过在Rust里,<code>&Tree</code>并不是<code>Tree的指针</code>,而是<code>Tree的Borrow</code>,或许你没看明白,没问题,让我们动手把<code>Box</code>修改成<code>&</code>,看看会发生什么:</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span> <span class="nc">Tree</span><span class="w"> </span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">left</span>: <span class="nb">Option</span><span class="o"><&'</span><span class="na">a</span><span class="w"> </span><span class="n">Tree</span><span class="o"><'</span><span class="na">a</span><span class="o">>></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">right</span>: <span class="nb">Option</span><span class="o"><&'</span><span class="na">a</span><span class="w"> </span><span class="n">Tree</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="o">></span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">item</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="k">impl</span><span class="w"> </span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="n">Tree</span><span class="w"> </span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">depth</span>: <span class="kt">i32</span><span class="p">,</span><span class="w"> </span><span class="n">i</span>: <span class="kt">i32</span><span class="p">)</span><span class="w"> </span>-> <span class="nc">Tree</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">depth</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">Tree</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">item</span><span class="w"> </span>: <span class="nc">i</span><span class="p">,</span><span class="w"> </span><span class="n">left</span>: <span class="nb">None</span><span class="p">,</span><span class="w"> </span><span class="n">right</span>: <span class="nb">None</span> <span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">Tree</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="n">item</span><span class="w"> </span>: <span class="nc">i</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="n">left</span>: <span class="nb">Some</span><span class="p">(</span><span class="o">&</span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">)),</span><span class="w"></span>
<span class="w"> </span><span class="n">right</span>: <span class="nb">Some</span><span class="p">(</span><span class="o">&</span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="p">)),</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="err">………………</span><span class="w"> </span><span class="err">下略</span><span class="w"> </span><span class="err">………………</span><span class="w"></span>
</code></pre></div>
<p>修改完之后程序有了很大的变化,除了把<code>Box</code>改成<code>&</code>之外,我们还添加了lifetime标识。如果之前或多或少知道rust,那么肯定知道rust是如何管理内存的——每一个变量都有一个生命期,超过生命期之后这个变量就会被销毁。因此,对于struct里的变量这种无法推断生命期的东西,需要在代码里指明这些变量到底能存在多长时间。不过很可惜,纵使修改成这个样,这段代码依然无法编译通过——会出现下面的错误:</p>
<div class="highlight"><pre><span></span><code><span class="mi">15</span><span class="o">:</span><span class="mi">29</span><span class="o">:</span> <span class="mi">15</span><span class="o">:</span><span class="mi">60</span> <span class="n">error</span><span class="o">:</span> <span class="n">borrowed</span> <span class="n">value</span> <span class="n">does</span> <span class="n">not</span> <span class="n">live</span> <span class="n">long</span> <span class="n">enough</span>
<span class="mi">15</span> <span class="n">left</span><span class="o">:</span> <span class="n">Some</span><span class="o">(&</span><span class="n">Tree</span><span class="o">::</span><span class="k">new</span><span class="o">(</span><span class="n">depth</span> <span class="o">-</span> <span class="mi">1</span><span class="o">,</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">i</span> <span class="o">-</span> <span class="mi">1</span><span class="o">)),</span>
<span class="o">^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</span>
<span class="mi">10</span><span class="o">:</span><span class="mi">48</span><span class="o">:</span> <span class="mi">19</span><span class="o">:</span><span class="mi">6</span> <span class="n">note</span><span class="o">:</span> <span class="n">reference</span> <span class="n">must</span> <span class="n">be</span> <span class="n">valid</span> <span class="k">for</span> <span class="n">the</span> <span class="n">lifetime</span> <span class="err">'</span><span class="n">a</span> <span class="k">as</span> <span class="n">defined</span> <span class="n">on</span> <span class="n">the</span> <span class="n">block</span> <span class="n">at</span> <span class="mi">10</span><span class="o">:</span><span class="mi">47</span><span class="o">...</span>
</code></pre></div>
<p>简单来说,在new函数里面,我们定义的所有变量在new函数结束时就全部被析构了,因此我们没法在其他地方用它。为了解决这个问题,我们需要把<code>Tree</code>分配在Heap上,并且保证它能活得够长。(当然,这并不代表这么做是不可能的,但在这里我们不讨论)</p>
<p>这个话题一旦展开就不得不附带上冗长的解释,毕竟lifetime是rust里最独特的东西,如果想更详细理解lifetime,rust的<a href="https://doc.rust-lang.org/">官方文档</a>是一个好地方。总之,希望你能通过这个不够详细的解释理解<code>&</code>跟<code>Box</code>的区别。</p>
<p>在说明了lifetime这个概念之后,下面的事情就变得简单多了, <code>pub fn new(depth: i32, i: i32) -> Tree</code>是一个"构造函数",构造函数有引号是因为rust里并没有语言层级的构造函数,<code>new</code>仅仅是一个约定而已。函数的定义跟<code>Ada</code>有些类似,相信所有人第一眼都能看明白这个函数的意义。</p>
<p>下面让我们来看<code>main</code>函数,除去大量的<code>println</code>,重要的代码只有一句:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">MINDEP</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="n">depth</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">).</span><span class="n">filter</span><span class="p">(</span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">).</span><span class="n">map</span><span class="p">(</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">|</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">MINDEP</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">),</span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">MINDEP</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">).</span><span class="n">fold</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">xt</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="n">x1</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">xt</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">x1</span><span class="p">).</span><span class="n">item_check</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="n">x1</span><span class="p">).</span><span class="n">item_check</span><span class="p">())));</span><span class="w"></span>
</code></pre></div>
<p>这一句稍微有些函数式的感觉,简单解释,我们找出<code>MINDEP</code>到<code>depth + 1</code>之间的所有偶数,对于每一个偶数,求从1到<code>(1 << (depth - x + MINDEP)) + 1</code>循环,并求对应Tree::item_check的和。相信熟悉函数式的人能够很快搞明白每一句的意思:map把没一个偶数变成一个Tuple,而fold则对区间求和。如果用更普通一点的写法,那么是这样:</p>
<div class="highlight"><pre><span></span><code><span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">MINDEP</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="n">depth</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">iterations</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">MINDEP</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">check</span><span class="w"> </span>: <span class="kt">i32</span> <span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">j</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="n">iterations</span><span class="o">+</span><span class="mi">1</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">check</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">j</span><span class="p">).</span><span class="n">itemCheck</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">check</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="n">j</span><span class="p">).</span><span class="n">itemCheck</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}</span><span class="se">\t</span><span class="s">trees of depth {}</span><span class="se">\t</span><span class="s"> check: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">iterations</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">2</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">check</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">2</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
</code></pre></div>
<p>可以看到,三行代码可以展开成12行。就如同函数式宗教的信者们所一直在宣讲的一样,相比与循环,map和fold可能更加简洁直观。</p>
<p>不过这些并不是重点,重点是我们看到这些代码里压根没有出现<code>free()</code>这种东西,完全就如同任何一个有GC的语言,定义,然后使用,不必担心哪些东西会吃掉内存。更重要的是Rust压根没有使用GC——也就是说不会有什么东西会突然停掉你的程序然后扫描内存,也不会有<code>gc_malloc</code>这种函数会在你使用的时候花费半个小时去扫描并释放空间,所有的析构都是静态的,也就是相当与自动在C代码里插入了<code>free</code>语句。</p>
<p>这种做法的好处显而易见,不会有什么不确定的东西影响程序的运行,也不会有无法释放的内存。让我们继续修改下这个程序,让它变成多线程:</p>
<div class="highlight"><pre><span></span><code><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="p">{</span><span class="n">env</span><span class="p">,</span><span class="w"> </span><span class="n">thread</span><span class="p">};</span><span class="w"></span>
<span class="err">…………</span><span class="w"> </span><span class="err">中略</span><span class="w"> </span><span class="err">…………</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">MINDEP</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="n">depth</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">).</span><span class="n">filter</span><span class="p">(</span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">).</span><span class="n">map</span><span class="p">(</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">|</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">MINDEP</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">),</span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">thread</span>::<span class="n">spawn</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">||</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">MINDEP</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">).</span><span class="n">fold</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">xt</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="n">x1</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">xt</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">x1</span><span class="p">).</span><span class="n">item_check</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="n">x1</span><span class="p">).</span><span class="n">item_check</span><span class="p">())))).</span><span class="n">collect</span>::<span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="n">_</span><span class="o">>></span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="p">(</span><span class="n">iters</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">check</span><span class="p">)</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="p">{</span><span class="w"> </span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}</span><span class="se">\t</span><span class="s"> trees of depth {}</span><span class="se">\t</span><span class="s"> check: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">iters</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">check</span><span class="p">.</span><span class="n">join</span><span class="p">().</span><span class="n">ok</span><span class="p">().</span><span class="n">unwrap_or</span><span class="p">(</span><span class="mi">0</span><span class="p">));</span><span class="w"> </span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="err">…………</span><span class="w"> </span><span class="err">后略</span><span class="w"> </span><span class="err">…………</span><span class="w"></span>
</code></pre></div>
<p>可以看到修改的地方很少,仅仅是在原先的<code>1 .. (1 << (depth - x + MINDEP)) + 1</code>循环外面套了一个<code>thread::new</code>而已,这也是函数式的另一个好处,相比与用for循环来说,实现多线程非常简单。同时,如果有一个C++程序员,那么他很有可能会对<code>thread::new</code>里面的代码表示担心,比如会不会有data race。回到Rust上,Rust有一个<code>owner</code>的概念,也就是说任何一个变量都有一个"所有者",并且只能有一个,虽然前面提到了Rust拥有borrow这个概念,但编译器会限制同一时间只能有一个可修改内容的borrow。那么很显然,只要编译过后没有错误,那么这个程序就不会出现问题——当然你可能需要面对很多的编译错误。这通常是一个trade-off,不过对于多线程程序来说,很明显面对编译器的错误信息要简单的多。</p>
<p>当然,也可以用<code>channel</code>来传递消息:</p>
<div class="highlight"><pre><span></span><code><span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">env</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">thread</span><span class="p">;</span><span class="w"></span>
<span class="k">use</span><span class="w"> </span><span class="n">std</span>::<span class="n">sync</span>::<span class="n">mpsc</span><span class="p">;</span><span class="w"></span>
<span class="err">…………</span><span class="w"> </span><span class="err">中略</span><span class="w"> </span><span class="err">…………</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">long_lived</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">depth</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="n">tx</span><span class="p">,</span><span class="w"> </span><span class="n">rx</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mpsc</span>::<span class="n">channel</span>::<span class="o"><</span><span class="p">(</span><span class="kt">i32</span><span class="p">,</span><span class="w"> </span><span class="kt">i32</span><span class="p">,</span><span class="w"> </span><span class="kt">i32</span><span class="p">)</span><span class="o">></span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="n">MINDEP</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="n">depth</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">).</span><span class="n">filter</span><span class="p">(</span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="mi">0</span><span class="p">).</span><span class="n">map</span><span class="p">(</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">tx</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tx</span><span class="p">.</span><span class="n">clone</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="n">thread</span>::<span class="n">spawn</span><span class="p">(</span><span class="k">move</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">tx</span><span class="p">.</span><span class="n">send</span><span class="p">((</span><span class="mi">1</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">MINDEP</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">),</span><span class="w"> </span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o">..</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="w"> </span><span class="o"><<</span><span class="w"> </span><span class="p">(</span><span class="n">depth</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">x</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">MINDEP</span><span class="p">))</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="p">).</span><span class="n">fold</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">xt</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="n">x1</span><span class="w"> </span><span class="o">|</span><span class="w"> </span><span class="n">xt</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="n">x1</span><span class="p">).</span><span class="n">item_check</span><span class="p">()</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Tree</span>::<span class="n">new</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="w"> </span><span class="o">-</span><span class="n">x1</span><span class="p">).</span><span class="n">item_check</span><span class="p">()))))}).</span><span class="n">collect</span>::<span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="n">_</span><span class="o">>></span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">_</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">res</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="p">(</span><span class="n">iters</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">check</span><span class="p">)</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">rx</span><span class="p">.</span><span class="n">recv</span><span class="p">().</span><span class="n">unwrap</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="fm">println!</span><span class="p">(</span><span class="s">"{}</span><span class="se">\t</span><span class="s"> trees of depth {}</span><span class="se">\t</span><span class="s"> check: {}"</span><span class="p">,</span><span class="w"> </span><span class="n">iters</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">check</span><span class="p">);</span><span class="w"> </span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="err">…………</span><span class="w"> </span><span class="err">下略</span><span class="w"> </span><span class="err">…………</span><span class="w"></span>
</code></pre></div>
<p>唯一需要注意的地方是由于owner的限制,<code>tx(sender)</code>对于每个线程都与要一个克隆。</p>
<p>好了,到这里,这篇文章也算多少介绍了Rust的主要特性,是时候来回头看看它到底怎么样了。对我来说,Rust有一个最大的特征——安心。只要没有编译错误或者<code>fn main() { main() }</code>这种代码,就可以放心的认为自己的程序是正确的,在写了几天Rust之后,可以明显感觉到自己考虑的事情变少了,只要按照自己的想法写出来,剩下的不足全部都由编译器来指出。有一个提升生活质量的设计,我还能要求什么呢?</p>Closures in OOC2015-02-20T00:00:00+09:002015-02-20T00:00:00+09:00Horsaltag:www.horsal.dev,2015-02-20:/closures-in-ooc.html<p>接<a href="https://www.horsal.dev/complexity-behind-closure.html">上一篇Complexity Behind Closure</a>,这次来专注于Rock是如何在C里实现Closure的。</p>
<h2>Block as Blocks</h2>
<p>首先,需要指出的是,在C里面并不是完全没有办法使用Closure。Apple的GCC Fork里就给C添加了Block,用于实现Closure:</p>
<p>(Steal from the <a href="http://en.wikipedia.org/wiki/Blocks_(C_language_extension)">Wiki</a>)</p>
<div class="highlight"><pre><span></span><code><span class="cp">#include</span> <span class="cpf"><stdio.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><Block.h></span><span class="cp"></span>
<span class="k">typedef</span> <span class="kt">int</span> <span class="p">(</span><span class="o">^</span><span class="n">IntBlock</span><span class="p">)();</span>
<span class="n">IntBlock</span> <span class="nf">MakeCounter</span><span class="p">(</span><span class="kt">int</span> <span class="n">start</span><span class="p">,</span> <span class="kt">int</span> <span class="n">increment</span><span class="p">)</span> <span class="p">{</span>
<span class="n">__block</span> <span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="n">start</span><span class="p">;</span>
<span class="k">return</span> <span class="n">Block_copy</span><span class="p">(</span> <span class="o">^</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">ret</span> <span class="o">=</span> <span class="n">i</span><span class="p">;</span>
<span class="n">i</span> <span class="o">+=</span> <span class="n">increment</span><span class="p">;</span>
<span class="k">return</span> <span class="n">ret</span><span class="p">;</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span> <span class="p">{</span>
<span class="n">IntBlock …</span></code></pre></div><p>接<a href="https://www.horsal.dev/complexity-behind-closure.html">上一篇Complexity Behind Closure</a>,这次来专注于Rock是如何在C里实现Closure的。</p>
<h2>Block as Blocks</h2>
<p>首先,需要指出的是,在C里面并不是完全没有办法使用Closure。Apple的GCC Fork里就给C添加了Block,用于实现Closure:</p>
<p>(Steal from the <a href="http://en.wikipedia.org/wiki/Blocks_(C_language_extension)">Wiki</a>)</p>
<div class="highlight"><pre><span></span><code><span class="cp">#include</span> <span class="cpf"><stdio.h></span><span class="cp"></span>
<span class="cp">#include</span> <span class="cpf"><Block.h></span><span class="cp"></span>
<span class="k">typedef</span> <span class="kt">int</span> <span class="p">(</span><span class="o">^</span><span class="n">IntBlock</span><span class="p">)();</span>
<span class="n">IntBlock</span> <span class="nf">MakeCounter</span><span class="p">(</span><span class="kt">int</span> <span class="n">start</span><span class="p">,</span> <span class="kt">int</span> <span class="n">increment</span><span class="p">)</span> <span class="p">{</span>
<span class="n">__block</span> <span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="n">start</span><span class="p">;</span>
<span class="k">return</span> <span class="n">Block_copy</span><span class="p">(</span> <span class="o">^</span> <span class="p">{</span>
<span class="kt">int</span> <span class="n">ret</span> <span class="o">=</span> <span class="n">i</span><span class="p">;</span>
<span class="n">i</span> <span class="o">+=</span> <span class="n">increment</span><span class="p">;</span>
<span class="k">return</span> <span class="n">ret</span><span class="p">;</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span> <span class="p">{</span>
<span class="n">IntBlock</span> <span class="n">mycounter</span> <span class="o">=</span> <span class="n">MakeCounter</span><span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">);</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"First call: %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">mycounter</span><span class="p">());</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"Second call: %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">mycounter</span><span class="p">());</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"Third call: %d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">mycounter</span><span class="p">());</span>
<span class="cm">/* because it was copied, it must also be released */</span>
<span class="n">Block_release</span><span class="p">(</span><span class="n">mycounter</span><span class="p">);</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="cm">/* Output:</span>
<span class="cm"> First call: 5</span>
<span class="cm"> Second call: 7</span>
<span class="cm"> Third call: 9</span>
<span class="cm">*/</span>
</code></pre></div>
<p>除去在代码风格上的偏见后,这个扩展看起来不是很差,不过依然有很多限制,比如考虑下面的功能:</p>
<div class="highlight"><pre><span></span><code><span class="n">isFound</span> <span class="p">:</span><span class="o">=</span> <span class="bp">false</span>
<span class="n">filelist</span> <span class="n">each</span><span class="p">(</span><span class="o">|</span><span class="n">filename</span><span class="p">,</span> <span class="n">mode</span><span class="o">|</span>
<span class="n">f</span> <span class="p">:</span><span class="o">=</span> <span class="k">func</span> <span class="o">-></span> <span class="nb nb-Type">String</span><span class="p">{</span>
<span class="n">filename</span> <span class="n">split</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span> <span class="n">each</span><span class="p">(</span><span class="n">strip</span><span class="p">())</span> <span class="n">strip</span><span class="p">()[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="p">}</span>
<span class="n">last</span> <span class="p">:</span><span class="o">=</span> <span class="n">f</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
<span class="k">if</span><span class="p">(</span><span class="n">last</span> <span class="o">==</span> <span class="s2">"mysdk"</span><span class="p">){</span>
<span class="n">isFound</span> <span class="o">=</span> <span class="bp">true</span>
<span class="k">return</span> <span class="bp">true</span>
<span class="p">}</span>
<span class="bp">false</span>
<span class="p">)</span>
</code></pre></div>
<p>尝试这用Apple的block扩展来实现一下?看起来一点不现实。并且就算实现了,也只能在Apple的GCC Fork下面工作,显然并不是我们在追求的东西。</p>
<h2>Way to Variable</h2>
<p>好的,在开始解释OOC的Closure之前,让我们先来想想Closure要有什么特性:</p>
<ul>
<li>能够自由的使用“自由变量”。 这里的“自由”是指在语法上定义于Closure以前的变量,比如</li>
</ul>
<div class="highlight"><pre><span></span><code>void test(Int a){
Int b;
// closure
}
</code></pre></div>
<p>这时closure要有能力读写变量a和b。当然,这是闭包的基本要求。</p>
<ul>
<li>
<p>能够定义在任何地方。在Object Pascal/Delphi里,虽然我们能够定义nested function,但它的位置并不是自由的——当你在函数中间写了一个nested function时,编译器会要求你把它移动到顶端。但显然很多情况下我们需要的不仅仅是一个“临时”函数,我们还需要一个包含了之前定义过的所有变量的环境。</p>
</li>
<li>
<p>能够“携带”定义时的环境。考虑下面的情况:</p>
</li>
</ul>
<div class="highlight"><pre><span></span><code><span class="n">f</span> <span class="o">:=</span> <span class="n">func</span><span class="p">(</span><span class="n">a</span><span class="o">:</span> <span class="n">Int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Func</span><span class="o">-></span><span class="n">Int</span><span class="p">{</span>
<span class="n">b</span><span class="o">:</span> <span class="n">Int</span> <span class="o">=</span> <span class="n">a</span><span class="o">*</span><span class="mi">2</span>
<span class="kr">return</span> <span class="n">func</span><span class="o">-></span> <span class="n">Int</span> <span class="p">{</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span> <span class="p">}</span>
<span class="p">}</span>
<span class="n">f1</span> <span class="o">:=</span> <span class="n">f</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="n">f1</span><span class="p">()</span> <span class="n">toString</span><span class="p">()</span> <span class="n">println</span><span class="p">()</span>
<span class="n">f1</span> <span class="o">=</span> <span class="n">f</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
<span class="n">f1</span><span class="p">()</span> <span class="n">toString</span><span class="p">()</span> <span class="n">println</span><span class="p">()</span>
</code></pre></div>
<p>这个时候应该输出什么? 显然是9和12。但如果我们的闭包不能携带这个环境的话,是没法输出正确的结果的。</p>
<h2>Way in OOC</h2>
<p>好的,现在让我们来看看在OOC里,closure是怎么实现的,这次让我们从最简单的情况开始:</p>
<div class="highlight"><pre><span></span><code><span class="n">main</span><span class="p">:</span> <span class="k">func</span> <span class="p">{</span>
<span class="n">a</span> <span class="p">:</span><span class="o">=</span> <span class="mi">3</span>
<span class="n">f</span> <span class="p">:</span><span class="o">=</span> <span class="k">func</span> <span class="o">-></span> <span class="n">Int</span> <span class="p">{</span> <span class="n">a</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>一个闭包,它只用来返回之前定义过的变量的值。在这里,如果你执行<code>"#{f()}" println()</code>的话,会得到结果<code>3</code>。好的,让我们首先看看OOC生成了什么,然后再慢慢解释: (另外,需要注意为什么我们这里要加上main函数? 因为如果没有main的话,所有的变量都会变成全局,在闭包里本身就可以直接读写,也就失去的例子的意义)</p>
<div class="highlight"><pre><span></span><code> <span class="n">lang_Numbers__Int</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">3</span><span class="p">;</span>
<span class="n">__simpleclosure_simpleclosure_closure3_ctx</span><span class="o">*</span> <span class="n">__simpleclosure_ctx4</span> <span class="o">=</span> <span class="n">lang_Memory__gc_malloc</span><span class="p">(((</span><span class="n">lang_types__Class</span><span class="o">*</span><span class="p">)</span><span class="n">__simpleclosure_simpleclosure_closure3_ctx_class</span><span class="p">())</span><span class="o">-></span><span class="n">size</span><span class="p">);</span>
<span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">__simpleclosure_ctx4</span><span class="p">))</span> <span class="o">=</span> <span class="p">(</span><span class="n">__simpleclosure_simpleclosure_closure3_ctx</span><span class="p">)</span> <span class="p">{</span>
<span class="n">a</span>
<span class="p">};</span>
<span class="n">lang_types__Closure</span> <span class="n">__simpleclosure_closure5</span> <span class="o">=</span> <span class="p">(</span><span class="n">lang_types__Closure</span><span class="p">)</span> <span class="p">{</span>
<span class="n">simpleclosure____simpleclosure_simpleclosure_closure3_thunk</span><span class="p">,</span>
<span class="n">__simpleclosure_ctx4</span>
<span class="p">};</span>
<span class="n">lang_types__Closure</span> <span class="n">f</span> <span class="o">=</span> <span class="n">__simpleclosure_closure5</span><span class="p">;</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">lang_Numbers__Int</span> <span class="n">simpleclosure____simpleclosure_simpleclosure_closure3</span><span class="p">(</span><span class="n">lang_Numbers__Int</span> <span class="n">a</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">a</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">lang_Numbers__Int</span> <span class="n">simpleclosure____simpleclosure_simpleclosure_closure3_thunk</span><span class="p">(</span><span class="n">__simpleclosure_simpleclosure_closure3_ctx</span><span class="o">*</span> <span class="n">__context__</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">simpleclosure____simpleclosure_simpleclosure_closure3</span><span class="p">((</span><span class="o">*</span><span class="n">__context__</span><span class="p">).</span><span class="n">a</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div>
<p>对于简单的两行代码,rock生成了一大堆东西。好的,让我们开始看到底发生了什么。</p>
<ol>
<li>首先,OOC里closure是一个结构体,它包含两个部分:<code>context</code>和<code>function pointer</code>。function pointer很简单,就如同名字上说的,我们把closure当成一个普通函数,这个函数的指针就是function pointer。而context则包含了<em>所有</em>closure需要的变量。用C的代码来说的话,就像这样:</li>
</ol>
<div class="highlight"><pre><span></span><code><span class="k">typedef</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">Closure_Context</span><span class="o">*</span> <span class="n">context</span><span class="p">;</span>
<span class="kt">void</span> <span class="p">(</span><span class="o">*</span> <span class="n">thunk</span><span class="p">)(</span><span class="n">Closure_Context</span><span class="o">*</span><span class="p">);</span>
<span class="p">}</span> <span class="n">myclosure</span><span class="p">;</span>
<span class="k">typedef</span> <span class="k">struct</span> <span class="p">{</span>
<span class="n">Int</span> <span class="n">a</span><span class="p">;</span>
<span class="p">}</span> <span class="n">Closure_Context</span><span class="p">;</span>
</code></pre></div>
<ol>
<li>
<p>当我们声明一个closure时,首先,这个closure会变成一个普通的函数,假设它的名字是<code>closure_impl</code>,但不同的地方是,除了本身声明时的参数外,这个closure还接受一个<code>Closure_Context*</code>作为参数。随后,我们初始化一个Closure_Context,它的成员包含了所有这个closure里用到的外部变量。在刚才那个例子里,我们只有一个<code>Int a</code>。最后,把这两个指针组合成<code>myclosure</code>,我们的声明就完成了。</p>
</li>
<li>
<p>到了这里,相信你早就明白该怎么使用它了,只要简单的<code>myclosure->thunk(myclosure->context)</code>,一切就自然的完成了。让我们看看是不是真的这样:</p>
</li>
</ol>
<div class="highlight"><pre><span></span><code><span class="nl">main</span><span class="p">:</span> <span class="n">func</span> <span class="p">{</span>
<span class="nl">a</span> <span class="p">:</span><span class="o">=</span> <span class="mi">3</span>
<span class="nl">f</span> <span class="p">:</span><span class="o">=</span> <span class="n">func</span> <span class="o">-></span> <span class="n">Int</span> <span class="p">{</span> <span class="n">a</span> <span class="p">}</span>
<span class="n">f</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div>
<p>编译之后:</p>
<div class="highlight"><pre><span></span><code> <span class="n">lang_Numbers__Int</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">3</span><span class="p">;</span>
<span class="n">__simpleclosure_simpleclosure_closure3_ctx</span><span class="o">*</span> <span class="n">__simpleclosure_ctx4</span> <span class="o">=</span> <span class="n">lang_Memory__gc_malloc</span><span class="p">(((</span><span class="n">lang_types__Class</span><span class="o">*</span><span class="p">)</span><span class="n">__simpleclosure_simpleclosure_closure3_ctx_class</span><span class="p">())</span><span class="o">-></span><span class="n">size</span><span class="p">);</span>
<span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">__simpleclosure_ctx4</span><span class="p">))</span> <span class="o">=</span> <span class="p">(</span><span class="n">__simpleclosure_simpleclosure_closure3_ctx</span><span class="p">)</span> <span class="p">{</span>
<span class="n">a</span>
<span class="p">};</span>
<span class="n">lang_types__Closure</span> <span class="n">__simpleclosure_closure5</span> <span class="o">=</span> <span class="p">(</span><span class="n">lang_types__Closure</span><span class="p">)</span> <span class="p">{</span>
<span class="n">simpleclosure____simpleclosure_simpleclosure_closure3_thunk</span><span class="p">,</span>
<span class="n">__simpleclosure_ctx4</span>
<span class="p">};</span>
<span class="n">lang_types__Closure</span> <span class="n">f</span> <span class="o">=</span> <span class="n">__simpleclosure_closure5</span><span class="p">;</span>
<span class="p">((</span><span class="n">lang_Numbers__Int</span> <span class="p">(</span><span class="o">*</span><span class="p">)(</span><span class="kt">void</span><span class="o">*</span><span class="p">))</span> <span class="n">f</span><span class="p">.</span><span class="n">thunk</span><span class="p">)(</span><span class="n">f</span><span class="p">.</span><span class="n">context</span><span class="p">);</span>
</code></pre></div>
<p>虽然比起手写的代码看起来要复杂,但跟我们的基本思想是完全一致的。</p>
<h2>Pain in Context</h2>
<p>不过,到这里其实一切并没有结束,因为我有意忽略了一个重要的地方——如何确定我们的context? 很显然,每个closure都有不同的context,不但意味这每个closure的context都要有自己的定义,还意味着我们必须自行推断需要把那些变量放进context里。好吧,让我们先看看第一个问题。</p>
<p>假设我们有这么一个代码:</p>
<div class="highlight"><pre><span></span><code><span class="nl">foo</span><span class="p">:</span> <span class="n">func</span><span class="p">{</span>
<span class="nl">b</span><span class="p">:</span> <span class="n">Int</span> <span class="o">=</span> <span class="mi">1</span>
<span class="nl">f</span> <span class="p">:</span><span class="o">=</span> <span class="n">func</span><span class="p">{</span>
<span class="n">b</span> <span class="o">+</span> <span class="mi">1</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nl">bar</span><span class="p">:</span> <span class="n">func</span><span class="p">{</span>
<span class="n">a</span><span class="p">,</span> <span class="nl">b</span><span class="p">:</span> <span class="n">Int</span>
<span class="nl">f</span> <span class="p">:</span><span class="o">=</span> <span class="n">func</span><span class="p">{</span>
<span class="n">a</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">b</span> <span class="o">=</span> <span class="mi">2</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>那么根据刚才介绍的构造,每一个<code>f</code>都会变成一个<code>(context,thunk)</code>的集合。thunk没有任何问题——简单的把f写成函数取地址就够了,那么让我们想想context该怎么构造。如果直接写的话,那么会是这样:</p>
<div class="highlight"><pre><span></span><code><span class="k">typedef</span> <span class="k">struct</span><span class="p">{</span>
<span class="kt">int</span> <span class="n">b</span><span class="p">;</span>
<span class="p">}</span> <span class="n">foo_f_context</span><span class="p">;</span>
<span class="k">typedef</span> <span class="k">struct</span><span class="p">{</span>
<span class="kt">int</span><span class="o">*</span> <span class="n">a</span><span class="p">;</span>
<span class="kt">int</span><span class="o">*</span> <span class="n">b</span><span class="p">;</span>
<span class="p">}</span> <span class="n">bar_f_context</span><span class="p">;</span>
</code></pre></div>
<p>这样,我们可以通过<code>foo_f_context* -> b</code>来访问b(只读),而可以通过<code>bar_f_context* -> a</code>来读写变量a。对,这种最简单的方法就是OOC所采用了。原因很简单——OOC不是C,在编译代码时所有closure访问了哪些变量都已经确定下来,我们只需要扫描一次,然后生成对应了struct就足够了。因此你会在头文件里找到这样的定义:</p>
<div class="highlight"><pre><span></span><code><span class="k">struct</span> <span class="nc">___simpleclosure_simpleclosure_closure3_ctx</span> <span class="p">{</span>
<span class="n">lang_Numbers__Int</span> <span class="n">a</span><span class="p">;</span>
<span class="p">};</span>
</code></pre></div>
<p>因为一切都是自动生成的,我们并不担心会出错。并且,这种方式非常简单而且高效。不过,如果打算直接在C代码里使用闭包,那问题就要复杂的多了——毕竟我们不可能给每一个闭包手写一个结构体。纵使可以用Hashmap来自由的访问变量,如何收集变量是一个大问题。现在回头看来,似乎Apple挑选了一个最好的方式,在没有破坏C的结构之下引入了闭包。</p>Complexity Behind Closure2015-01-30T00:00:00+09:002015-01-30T00:00:00+09:00Horsaltag:www.horsal.dev,2015-01-30:/complexity-behind-closure.html<h2>Introduction</h2>
<p>今天,我在<a href="https://github.com/cogneco/ooc-kean">ooc-kean</a>上看到了一个注释:</p>
<div class="highlight"><pre><span></span><code><span class="n">minimum</span><span class="p">:</span> <span class="k">static</span> <span class="k">func</span> <span class="o">~</span><span class="n">multiple</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="n">This</span><span class="p">,</span> <span class="n">values</span><span class="p">:</span> <span class="o">...</span><span class="p">)</span> <span class="o">-></span> <span class="n">This</span> <span class="p">{</span>
<span class="o">//</span> <span class="n">FIXME</span><span class="p">:</span> <span class="n">This</span> <span class="n">creates</span> <span class="n">a</span> <span class="n">closure</span> <span class="n">that</span> <span class="n">causes</span> <span class="n">a</span> <span class="n">leak</span> <span class="n">every</span> <span class="n">time</span> <span class="n">this</span> <span class="n">function</span> <span class="k">is</span> <span class="n">called</span><span class="o">.</span>
<span class="n">values</span> <span class="n">each</span><span class="p">(</span><span class="o">|</span><span class="n">v</span><span class="o">|</span>
<span class="k">if</span> <span class="p">((</span><span class="n">v</span> <span class="k">as</span> <span class="n">This</span><span class="p">)</span> <span class="o"><</span> <span class="n">value</span><span class="p">)</span>
<span class="n">value</span> <span class="o">=</span> <span class="n">v</span>
<span class="p">)</span>
<span class="n">value</span>
<span class="p">}</span>
</code></pre></div>
<p>minimum是Float类型的一个扩展,这里用了<code>闭包</code>来实现对每个值的比较。如果这些发生在GC之下,那么一切都没有问题,因为任何内存(包括闭包的)都会被GC默默的回收。但OOC-Kean并没有打开GC,这里就出现了问题 …</p><h2>Introduction</h2>
<p>今天,我在<a href="https://github.com/cogneco/ooc-kean">ooc-kean</a>上看到了一个注释:</p>
<div class="highlight"><pre><span></span><code><span class="n">minimum</span><span class="p">:</span> <span class="k">static</span> <span class="k">func</span> <span class="o">~</span><span class="n">multiple</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="n">This</span><span class="p">,</span> <span class="n">values</span><span class="p">:</span> <span class="o">...</span><span class="p">)</span> <span class="o">-></span> <span class="n">This</span> <span class="p">{</span>
<span class="o">//</span> <span class="n">FIXME</span><span class="p">:</span> <span class="n">This</span> <span class="n">creates</span> <span class="n">a</span> <span class="n">closure</span> <span class="n">that</span> <span class="n">causes</span> <span class="n">a</span> <span class="n">leak</span> <span class="n">every</span> <span class="n">time</span> <span class="n">this</span> <span class="n">function</span> <span class="k">is</span> <span class="n">called</span><span class="o">.</span>
<span class="n">values</span> <span class="n">each</span><span class="p">(</span><span class="o">|</span><span class="n">v</span><span class="o">|</span>
<span class="k">if</span> <span class="p">((</span><span class="n">v</span> <span class="k">as</span> <span class="n">This</span><span class="p">)</span> <span class="o"><</span> <span class="n">value</span><span class="p">)</span>
<span class="n">value</span> <span class="o">=</span> <span class="n">v</span>
<span class="p">)</span>
<span class="n">value</span>
<span class="p">}</span>
</code></pre></div>
<p>minimum是Float类型的一个扩展,这里用了<code>闭包</code>来实现对每个值的比较。如果这些发生在GC之下,那么一切都没有问题,因为任何内存(包括闭包的)都会被GC默默的回收。但OOC-Kean并没有打开GC,这里就出现了问题——闭包需要保存它的临时环境,于是每个闭包都会创建一个结构体,但编译器并没有考虑去回收它。因此就有了这里这个FIXME。</p>
<p>好吧,或许你还不是很明白为什么,那么让我们来看看这段代码最终生成的C代码:</p>
<div class="highlight"><pre><span></span><code><span class="n">lang_Numbers__Float</span> <span class="n">lang_Numbers__Float_minimum_multiple</span><span class="p">(</span><span class="n">lang_Numbers__Float</span> <span class="n">value</span><span class="p">,</span> <span class="n">lang_VarArgs__VarArgs</span> <span class="n">values</span><span class="p">)</span> <span class="p">{</span>
<span class="n">__FloatExtension_FloatExtension_closure4_ctx</span><span class="o">*</span> <span class="n">__FloatExtension_ctx5</span> <span class="o">=</span> <span class="n">lang_Memory__gc_malloc</span><span class="p">(((</span><span class="n">lang_types__Class</span><span class="o">*</span><span class="p">)</span><span class="n">__FloatExtension_FloatExtension_closure4_ctx_class</span><span class="p">())</span><span class="o">-></span><span class="n">size</span><span class="p">);</span>
<span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">__FloatExtension_ctx5</span><span class="p">))</span> <span class="o">=</span> <span class="p">(</span><span class="n">__FloatExtension_FloatExtension_closure4_ctx</span><span class="p">)</span> <span class="p">{</span>
<span class="o">&</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="p">};</span>
<span class="n">lang_types__Closure</span> <span class="n">__FloatExtension_closure6</span> <span class="o">=</span> <span class="p">(</span><span class="n">lang_types__Closure</span><span class="p">)</span> <span class="p">{</span>
<span class="n">FloatExtension____FloatExtension_FloatExtension_closure4_thunk</span><span class="p">,</span>
<span class="n">__FloatExtension_ctx5</span>
<span class="p">};</span>
<span class="n">lang_VarArgs__VarArgs_each</span><span class="p">(</span><span class="n">values</span><span class="p">,</span> <span class="n">__FloatExtension_closure6</span><span class="p">);</span>
<span class="kr">return</span> <span class="n">value</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>让我们来慢慢解释这个函数。</p>
<ul>
<li>首先,对于没一个闭包(|x|..)编译器会首先生成一个闭包环境用来保存运行时的内容,这里,这个环境叫做<code>__FloatExtension_ctx5</code>,这是一个简单的结构体,随后,我们知道,每个closure都只能在当前Scope里有效,因此在每次使用之前,我们要为它分配新的内存,然后初始化。为什么要这么做?首先,我们可能打算模拟闭包那种随用随舍弃的特性,但这不是最大的原因。最大的问题是大部分Closure都是作为__First-class Function__使用的,如果之是定义之后简单调用的话,本来一切会很美好(我们可以简单的生成macro,或者在module里生成新的函数),但当他是一个指针时,在C语言的层面上,我们只能寻找一个替代品——这个替代品要能够使用当前scope已经存在的变量,函数,同时还能将一切反馈回来。这就是一个十分困难的问题。</li>
<li>在这里,编译器在closure的背后做了许多事情,这里初始化的<code>ctx5</code>包含了一个庞大的初始化函数(未在源代码里显示),这个初始化会根据Scope里每个函数的类型和状态来决定是用Pointer(by reference)还是value。随后我们创建了一个普通的C函数并取它的指针,但它并不直接跟我们声明的closure有一样的参数,而是接受一个叫做__context__的参数,这个参数包含了所有之前的创建几个内容。</li>
<li>最后,不管是我们直接调用闭包,还是把它当作指针使用,都可以通过统一的初始化函数生成<code>ctx</code>和<code>context</code>了。</li>
</ul>
<p>显然, 问题也就出在这里——为了使用context,我们分配的内存,但并没有考虑释放它。</p>
<h2>The First Encount</h2>
<p>第一个有的想法很单纯——既然没有释放,那么我们在Scope的最后加上一个<code>gc_free</code>不就行了么?那么让我们来试一试:</p>
<p>在生成closure之后,让我们添加一个Free的调用:</p>
<div class="highlight"><pre><span></span><code>ctxFreeCall := FunctionCall new("gc_free", token)
ctxFreeCall args add(VariableAccess new(ctxDecl, token))
trail addAfterInScope(this, ctxFreeCall)
</code></pre></div>
<p><code>trail addAfterInScope</code>会在当前Scope里当前Statement的后方生成对应的函数——当然,前提是编译器的栈没有问题。</p>
<p>然后,理所当然的,刚才的内存泄漏消失了:</p>
<div class="highlight"><pre><span></span><code><span class="n">lang_Numbers__Float</span> <span class="n">lang_Numbers__Float_minimum_multiple</span><span class="p">(</span><span class="n">lang_Numbers__Float</span> <span class="n">value</span><span class="p">,</span> <span class="n">lang_VarArgs__VarArgs</span> <span class="n">values</span><span class="p">)</span> <span class="p">{</span>
<span class="n">__FloatExtension_FloatExtension_closure4_ctx</span><span class="o">*</span> <span class="n">__FloatExtension_ctx5</span> <span class="o">=</span> <span class="n">lang_Memory__gc_malloc</span><span class="p">(((</span><span class="n">lang_types__Class</span><span class="o">*</span><span class="p">)</span><span class="n">__FloatExtension_FloatExtension_closure4_ctx_class</span><span class="p">())</span><span class="o">-></span><span class="n">size</span><span class="p">);</span>
<span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">__FloatExtension_ctx5</span><span class="p">))</span> <span class="o">=</span> <span class="p">(</span><span class="n">__FloatExtension_FloatExtension_closure4_ctx</span><span class="p">)</span> <span class="p">{</span>
<span class="o">&</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="p">};</span>
<span class="n">lang_types__Closure</span> <span class="n">__FloatExtension_closure6</span> <span class="o">=</span> <span class="p">(</span><span class="n">lang_types__Closure</span><span class="p">)</span> <span class="p">{</span>
<span class="n">FloatExtension____FloatExtension_FloatExtension_closure4_thunk</span><span class="p">,</span>
<span class="n">__FloatExtension_ctx5</span>
<span class="p">};</span>
<span class="n">lang_VarArgs__VarArgs_each</span><span class="p">(</span><span class="n">values</span><span class="p">,</span> <span class="n">__FloatExtension_closure6</span><span class="p">);</span>
<span class="n">lang_Memory__gc_free</span><span class="p">(</span><span class="n">__FloatExtension_ctx5</span><span class="p">);</span>
<span class="kr">return</span> <span class="n">value</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>不过,先别高兴,因为很快就有了新的问题——标准库的一些函数运行时抛出了<code>segmentation fault</code>。一个典型的例子是<code>lang/format</code>下面的一个函数,这个函数在字符串处理里有着关键的角色:</p>
<div class="highlight"><pre><span></span><code><span class="n">getEntityInfo</span><span class="p">:</span> <span class="n">inline</span> <span class="k">func</span> <span class="p">(</span><span class="n">info</span><span class="p">:</span> <span class="n">FSInfoStruct</span><span class="err">@</span><span class="p">,</span> <span class="n">va</span><span class="p">:</span> <span class="n">VarArgsIterator</span><span class="o">*</span><span class="p">,</span> <span class="n">start</span><span class="p">:</span> <span class="n">Char</span><span class="o">*</span><span class="p">,</span> <span class="n">end</span><span class="p">:</span> <span class="n">Pointer</span><span class="p">)</span> <span class="p">{</span>
<span class="o">/*</span> <span class="n">save</span> <span class="n">original</span> <span class="n">pointer</span> <span class="o">*/</span>
<span class="n">p</span> <span class="p">:</span><span class="o">=</span> <span class="n">start</span>
<span class="n">checkedInc</span> <span class="p">:</span><span class="o">=</span> <span class="k">func</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">p</span> <span class="o"><</span> <span class="n">end</span><span class="p">)</span> <span class="n">p</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">else</span> <span class="n">InvalidFormatException</span> <span class="n">new</span><span class="p">(</span><span class="n">start</span><span class="p">)</span> <span class="n">throw</span><span class="p">()</span>
<span class="p">}</span>
<span class="o">/*</span> <span class="n">Find</span> <span class="n">any</span> <span class="n">flags</span><span class="o">.</span> <span class="o">*/</span>
<span class="n">info</span> <span class="n">flags</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">while</span><span class="p">(</span><span class="n">p</span> <span class="k">as</span> <span class="n">Pointer</span> <span class="o"><</span> <span class="n">end</span><span class="p">)</span> <span class="p">{</span>
<span class="n">checkedInc</span><span class="p">()</span>
<span class="k">match</span><span class="p">(</span><span class="n">p</span><span class="err">@</span><span class="p">)</span> <span class="p">{</span>
<span class="n">case</span> <span class="s1">'#'</span> <span class="o">=></span> <span class="n">info</span> <span class="n">flags</span> <span class="o">|=</span> <span class="n">TF_ALTERNATE</span>
<span class="n">case</span> <span class="s1">'0'</span> <span class="o">=></span> <span class="n">info</span> <span class="n">flags</span> <span class="o">|=</span> <span class="n">TF_ZEROPAD</span>
<span class="n">case</span> <span class="s1">'-'</span> <span class="o">=></span> <span class="n">info</span> <span class="n">flags</span> <span class="o">|=</span> <span class="n">TF_LEFT</span>
<span class="n">case</span> <span class="s1">' '</span> <span class="o">=></span> <span class="n">info</span> <span class="n">flags</span> <span class="o">|=</span> <span class="n">TF_SPACE</span>
<span class="n">case</span> <span class="s1">'+'</span> <span class="o">=></span> <span class="n">info</span> <span class="n">flags</span> <span class="o">|=</span> <span class="n">TF_EXP_SIGN</span>
<span class="n">case</span> <span class="o">=></span> <span class="k">break</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="o">/*</span> <span class="n">Find</span> <span class="n">the</span> <span class="n">field</span> <span class="n">width</span><span class="o">.</span> <span class="o">*/</span>
<span class="n">info</span> <span class="n">fieldwidth</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">while</span><span class="p">(</span><span class="n">p</span><span class="err">@</span> <span class="n">digit</span><span class="err">?</span><span class="p">())</span> <span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">info</span> <span class="n">fieldwidth</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span>
<span class="n">info</span> <span class="n">fieldwidth</span> <span class="o">*=</span> <span class="mi">10</span>
<span class="n">info</span> <span class="n">fieldwidth</span> <span class="o">+=</span> <span class="p">(</span><span class="n">p</span><span class="err">@</span> <span class="k">as</span> <span class="n">Int</span> <span class="o">-</span> <span class="mh">0x30</span><span class="p">)</span>
<span class="n">checkedInc</span><span class="p">()</span>
<span class="p">}</span>
<span class="o">/*</span> <span class="n">Find</span> <span class="n">the</span> <span class="n">precision</span><span class="o">.</span> <span class="o">*/</span>
<span class="n">info</span> <span class="n">precision</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="k">if</span><span class="p">(</span><span class="n">p</span><span class="err">@</span> <span class="o">==</span> <span class="s1">'.'</span><span class="p">)</span> <span class="p">{</span>
<span class="n">checkedInc</span><span class="p">()</span>
<span class="n">info</span> <span class="n">precision</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">if</span><span class="p">(</span><span class="n">p</span><span class="err">@</span> <span class="o">==</span> <span class="s1">'*'</span><span class="p">)</span> <span class="p">{</span>
<span class="n">T</span> <span class="p">:</span><span class="o">=</span> <span class="n">va</span><span class="err">@</span> <span class="n">getNextType</span><span class="p">()</span>
<span class="n">info</span> <span class="n">precision</span> <span class="o">=</span> <span class="n">argNext</span><span class="p">(</span><span class="n">va</span><span class="p">,</span> <span class="n">T</span><span class="p">)</span> <span class="k">as</span> <span class="n">Int</span>
<span class="n">checkedInc</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">while</span><span class="p">(</span><span class="n">p</span><span class="err">@</span> <span class="n">digit</span><span class="err">?</span><span class="p">())</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">info</span> <span class="n">precision</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span>
<span class="n">info</span> <span class="n">precision</span> <span class="o">*=</span> <span class="mi">10</span>
<span class="n">info</span> <span class="n">precision</span> <span class="o">+=</span> <span class="p">(</span><span class="n">p</span><span class="err">@</span> <span class="k">as</span> <span class="n">Int</span> <span class="o">-</span> <span class="mh">0x30</span><span class="p">)</span>
<span class="n">checkedInc</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="o">/*</span> <span class="n">Find</span> <span class="n">the</span> <span class="n">length</span> <span class="n">modifier</span><span class="o">.</span> <span class="o">*/</span>
<span class="n">info</span> <span class="n">length</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">while</span> <span class="p">(</span><span class="n">p</span><span class="err">@</span> <span class="o">==</span> <span class="s1">'l'</span> <span class="o">||</span> <span class="n">p</span><span class="err">@</span> <span class="o">==</span> <span class="s1">'h'</span> <span class="o">||</span> <span class="n">p</span><span class="err">@</span> <span class="o">==</span> <span class="s1">'L'</span><span class="p">)</span> <span class="p">{</span>
<span class="n">info</span> <span class="n">length</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">checkedInc</span><span class="p">()</span>
<span class="p">}</span>
<span class="n">info</span> <span class="n">bytesProcessed</span> <span class="o">=</span> <span class="n">p</span> <span class="k">as</span> <span class="n">SizeT</span> <span class="o">-</span> <span class="n">start</span> <span class="k">as</span> <span class="n">SizeT</span>
<span class="p">}</span>
</code></pre></div>
<p>看起来有些长,不过没关系,你不需要理解这个函数,因为引起segmentatian fault的其实只有几行,让我们把它单独拿出来:</p>
<div class="highlight"><pre><span></span><code> <span class="n">checkedInc</span> <span class="p">:</span><span class="o">=</span> <span class="k">func</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">p</span> <span class="o"><</span> <span class="n">end</span><span class="p">)</span> <span class="n">p</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">else</span> <span class="n">InvalidFormatException</span> <span class="n">new</span><span class="p">(</span><span class="n">start</span><span class="p">)</span> <span class="n">throw</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div>
<p>这是什么? 对,闭包,那么我们可以想象到编译器做了什么,让我们来看看C代码:</p>
<div class="highlight"><pre><span></span><code><span class="n">__lang_Format_lang_Format_closure26_ctx</span><span class="o">*</span> <span class="n">__lang_Format_ctx27</span> <span class="o">=</span> <span class="n">lang_Memory__gc_malloc</span><span class="p">(((</span><span class="n">lang_types__Class</span><span class="o">*</span><span class="p">)</span><span class="n">__lang_Format_lang_Format_closure26_ctx_class</span><span class="p">())</span><span class="o">-></span><span class="n">size</span><span class="p">);</span>
<span class="p">(</span><span class="o">*</span><span class="p">(</span><span class="n">__lang_Format_ctx27</span><span class="p">))</span> <span class="o">=</span> <span class="p">(</span><span class="n">__lang_Format_lang_Format_closure26_ctx</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">end,</span>
<span class="n">start</span><span class="p">,</span>
<span class="o">&</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
<span class="p">};</span>
<span class="n">lang_types__Closure</span> <span class="n">__lang_Format_closure28</span> <span class="o">=</span> <span class="p">(</span><span class="n">lang_types__Closure</span><span class="p">)</span> <span class="p">{</span>
<span class="n">lang_Format____lang_Format_lang_Format_closure26_thunk</span><span class="p">,</span>
<span class="n">__lang_Format_ctx27</span>
<span class="p">};</span>
<span class="n">lang_types__Closure</span> <span class="n">checkedInc</span> <span class="o">=</span> <span class="n">__lang_Format_closure28</span><span class="p">;</span>
<span class="n">gc_free</span><span class="p">(</span><span class="n">__lang_Format_closure28</span><span class="p">);</span>
<span class="p">(</span><span class="o">*</span><span class="n">info</span><span class="p">).</span><span class="n">flags</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">while</span> <span class="p">(((</span><span class="n">lang_types__Pointer</span><span class="p">)</span> <span class="p">(</span><span class="n">p</span><span class="p">))</span> <span class="o"><</span> <span class="kd">end)</span> <span class="p">{</span>
</code></pre></div>
<p>没错,我们为checkedInc生成了一个很漂亮的运行环境,但问题来了,我们在当前Scope里,并且是当前Statement之后就释放了它,那么显然,如果任何后面的代码使用了这个闭包,那么我们得到的就是一个空指针。</p>
<p>解决方法似乎并不是很难,让我们仔细想想——我们到底什么时候才不要这个闭包,显然,如果这个闭包定义来自函数,那么就是这个函数返回时。让我们来实现它:</p>
<div class="highlight"><pre><span></span><code><span class="n">i</span><span class="w"> </span><span class="err">:</span><span class="o">=</span><span class="w"> </span><span class="n">getSize</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="k">while</span><span class="p">(</span><span class="n">i</span><span class="o">>=</span><span class="mi">0</span><span class="p">)</span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="n">trail</span><span class="w"> </span><span class="k">get</span><span class="p">(</span><span class="n">i</span><span class="p">)</span><span class="w"> </span><span class="n">instanceOf</span><span class="vm">?</span><span class="p">(</span><span class="n">FunctionDecl</span><span class="p">))</span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="n">fun</span><span class="w"> </span><span class="err">:</span><span class="o">=</span><span class="w"> </span><span class="n">trail</span><span class="w"> </span><span class="k">get</span><span class="p">(</span><span class="n">i</span><span class="p">)</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">FunctionDecl</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="p">(</span><span class="n">i</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">fun</span><span class="w"> </span><span class="n">body</span><span class="w"> </span><span class="n">indexOf</span><span class="p">(</span><span class="n">this</span><span class="p">)..</span><span class="n">fun</span><span class="w"> </span><span class="n">body</span><span class="w"> </span><span class="k">size</span><span class="p">)</span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="n">fun</span><span class="w"> </span><span class="n">body</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="w"> </span><span class="n">instanceOf</span><span class="vm">?</span><span class="p">(</span><span class="k">Return</span><span class="p">))</span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="n">fun</span><span class="w"> </span><span class="k">add</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">ctxFreeCall</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="err">}</span><span class="w"></span>
<span class="w"> </span><span class="err">}</span><span class="w"></span>
<span class="w"> </span><span class="err">}</span><span class="w"></span>
<span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">-=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="err">}</span><span class="w"></span>
</code></pre></div>
<p>我们在每个return里都追加了free——并且它们一定要在当前statement之后。不过相信你一定已经注意到了潜在的问题——Function的Body不见得都是Statement,它有可能是If,有可能是Scope,还有可能是别的闭包,这个问题突然就变得麻烦起来了。</p>
<p>不过我们当然有办法,我们针对每个类型判断,然后决定是不是该递归去寻找Return:</p>
<div class="highlight"><pre><span></span><code><span class="k">if</span><span class="p">(</span><span class="n">fun</span><span class="w"> </span><span class="n">body</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="w"> </span><span class="n">instanceOf</span><span class="vm">?</span><span class="p">(</span><span class="n">ControlStatement</span><span class="p">))</span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="p">(</span><span class="n">i</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">fun</span><span class="w"> </span><span class="n">body</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">ControlStatement</span><span class="w"> </span><span class="n">body</span><span class="p">)</span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="o">//</span><span class="w"> </span><span class="k">recursive</span><span class="w"> </span><span class="n">do</span><span class="w"></span>
<span class="w"> </span><span class="err">}</span><span class="w"></span>
<span class="err">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="n">fun</span><span class="w"> </span><span class="n">body</span><span class="o">[</span><span class="n">i</span><span class="o">]</span><span class="w"> </span><span class="n">instanceOf</span><span class="vm">?</span><span class="p">(</span><span class="k">Scope</span><span class="p">))</span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="o">//</span><span class="w"> </span><span class="p">....</span><span class="w"></span>
<span class="err">}</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="p">....</span><span class="w"></span>
</code></pre></div>
<p>虽然很麻烦,但不是特别困难,如果问题到此为止,那么它远远算不上是<code>complex</code>,让我们再来想想,闭包还能怎么用? 对,初始化一个函数指针。如果这个发生在Function里面,那么一切都很好,但实际情况是——我们可以在Class里初始化一个函数指针,也可以在Cover里这么做。现在问题一下复杂起来了,因为定义在class里面的闭包并不属于一个函数,我们除了在Destroy(或者finalization)里面添加释放函数之外别无它法。但更大的问题是——我们无法保证class已经被完全解析!也就是说,在某些情况下,我们不知道这个class的任何信息,当然更不要说去添加释放函数。</p>
<p>然后,测试结果给了我们更大的打击——单纯的在Statement前面添加释放函数有时会破坏函数的结构!当我用上面的补丁对编译器进行boostrap的时候,它毫不客气的扔了一个错误,并且压根没有Backtrace。</p>
<p>为什么? 我想你已经注意到了,因为闭包可以作为指针被返回!</p>
<h2>The Last, By Last</h2>
<p>不知道你怎么看这个问题,至少上面这些东西就已经消耗了我整整一天的时间。并且问题还远远没有解决。</p>
<p>现在是时候来总结下原因了,简单来说,闭包可以作为指针被传来传去。在有GC时,GC会追踪它并且释放它,但当GC被关闭时,用户没法获得闭包真正的内容,因此甚至无法手动释放。而当闭包作为指针被传来传去时,我们找不到合适的时机来释放它。</p>
<p>那么我们来做最后一次尝试:我们追踪这个变量,一旦这个变量不再被使用时,我们就立刻追加一个<code>gc_free</code>,当然,这仅限于当前Scope,当它出了这个scope,我们就要求外界必须获得一个拷贝。这个实现也不难,就像这样:</p>
<div class="highlight"><pre><span></span><code><span class="nl">addByLastUse</span><span class="p">:</span><span class="w"> </span><span class="n">func</span><span class="p">(</span><span class="nl">ctx</span><span class="p">:</span><span class="w"> </span><span class="n">VariableDecl</span><span class="p">,</span><span class="w"> </span><span class="nl">marker</span><span class="p">:</span><span class="w"> </span><span class="k">Statement</span><span class="p">,</span><span class="w"> </span><span class="nl">newcomer</span><span class="p">:</span><span class="w"> </span><span class="k">Statement</span><span class="p">)</span><span class="w"> </span><span class="o">-></span><span class="w"> </span><span class="n">Bool</span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="err">:</span><span class="o">=</span><span class="w"> </span><span class="n">getSize</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="k">while</span><span class="p">(</span><span class="n">i</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"> </span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="n">node</span><span class="w"> </span><span class="err">:</span><span class="o">=</span><span class="w"> </span><span class="k">data</span><span class="w"> </span><span class="k">get</span><span class="p">(</span><span class="n">i</span><span class="p">)</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="n">Node</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="n">node</span><span class="w"> </span><span class="n">instanceOf</span><span class="vm">?</span><span class="p">(</span><span class="k">Scope</span><span class="p">))</span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="n">sc</span><span class="w"> </span><span class="err">:</span><span class="o">=</span><span class="w"> </span><span class="n">node</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="k">Scope</span><span class="w"></span>
<span class="w"> </span><span class="k">last</span><span class="w"> </span><span class="err">:</span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="n">j</span><span class="w"> </span><span class="err">:</span><span class="o">=</span><span class="w"> </span><span class="n">sc</span><span class="w"> </span><span class="n">list</span><span class="w"> </span><span class="k">size</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="k">while</span><span class="p">(</span><span class="n">j</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="n">sc</span><span class="w"> </span><span class="n">list</span><span class="w"> </span><span class="n">indexOf</span><span class="p">(</span><span class="n">marker</span><span class="p">))</span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="n">accessed</span><span class="p">(</span><span class="n">sc</span><span class="w"> </span><span class="n">list</span><span class="o">[</span><span class="n">j</span><span class="o">]</span><span class="p">,</span><span class="w"> </span><span class="n">ctx</span><span class="p">))</span><span class="err">{</span><span class="w"></span>
<span class="w"> </span><span class="n">sc</span><span class="w"> </span><span class="n">list</span><span class="w"> </span><span class="k">add</span><span class="p">(</span><span class="n">j</span><span class="o">+</span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="n">newcomer</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="k">true</span><span class="w"></span>
<span class="w"> </span><span class="err">}</span><span class="w"></span>
<span class="w"> </span><span class="n">j</span><span class="w"> </span><span class="o">-=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="err">}</span><span class="w"></span>
<span class="w"> </span><span class="o">//</span><span class="w"> </span><span class="k">no</span><span class="w"> </span><span class="n">access</span><span class="p">,</span><span class="w"> </span><span class="n">we</span><span class="w"> </span><span class="n">can</span><span class="w"> </span><span class="k">free</span><span class="w"> </span><span class="n">it</span><span class="w"> </span><span class="k">after</span><span class="w"> </span><span class="k">scope</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">addAfterInScope</span><span class="p">(</span><span class="n">marker</span><span class="p">,</span><span class="w"> </span><span class="n">newcomer</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="err">}</span><span class="w"></span>
<span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">-=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="err">}</span><span class="w"></span>
<span class="w"> </span><span class="k">false</span><span class="w"></span>
<span class="err">}</span><span class="w"></span>
</code></pre></div>
<p>access是一个判断当前变量有没有被使用过的函数,我们需要处理很多情况,就像这样:</p>
<div class="highlight"><pre><span></span><code> <span class="k">if</span><span class="ss">(</span><span class="nv">node</span> <span class="nv">instanceOf</span>?<span class="ss">(</span><span class="nv">FunctionCall</span><span class="ss">))</span>{
<span class="k">for</span><span class="ss">(</span><span class="nv">i</span> <span class="nv">in</span> <span class="nv">node</span> <span class="nv">as</span> <span class="nv">FunctionCall</span> <span class="nv">args</span><span class="ss">)</span>{
<span class="k">if</span><span class="ss">(</span><span class="nv">accessed</span><span class="ss">(</span><span class="nv">i</span>, <span class="nv">v</span><span class="ss">))</span> <span class="k">return</span> <span class="nv">true</span>
}
} <span class="k">else</span> <span class="k">if</span><span class="ss">(</span><span class="nv">node</span> <span class="nv">instanceOf</span>?<span class="ss">(</span><span class="nv">VariableAccess</span><span class="ss">))</span>{
<span class="k">if</span><span class="ss">(</span><span class="nv">node</span> <span class="nv">as</span> <span class="nv">VariableAccess</span> <span class="nv">getRef</span><span class="ss">()</span> <span class="o">&&</span> <span class="nv">node</span> <span class="nv">as</span> <span class="nv">VariableAccess</span> <span class="nv">getRef</span><span class="ss">()</span> <span class="nv">instanceOf</span>?<span class="ss">(</span><span class="nv">VariableDecl</span><span class="ss">)</span> <span class="o">&&</span> <span class="nv">node</span> <span class="nv">as</span> <span class="nv">VariableAccess</span> <span class="nv">getRef</span><span class="ss">()</span> <span class="nv">as</span> <span class="nv">VariableDecl</span> <span class="nv">name</span> <span class="o">==</span> <span class="nv">v</span> <span class="nv">name</span><span class="ss">)</span>{
<span class="k">return</span> <span class="nv">true</span>
}
} <span class="k">else</span> <span class="k">if</span><span class="ss">(</span><span class="nv">node</span> <span class="nv">instanceOf</span>?<span class="ss">(</span><span class="nv">Scope</span><span class="ss">))</span>{
<span class="k">for</span><span class="ss">(</span><span class="nv">i</span> <span class="nv">in</span> <span class="nv">node</span> <span class="nv">as</span> <span class="nv">Scope</span> <span class="nv">list</span><span class="ss">)</span>{
<span class="k">if</span><span class="ss">(</span><span class="nv">accessed</span><span class="ss">(</span><span class="nv">i</span>, <span class="nv">v</span><span class="ss">))</span> <span class="k">return</span> <span class="nv">true</span>
}
} <span class="k">else</span> <span class="k">if</span><span class="ss">(</span><span class="nv">node</span> <span class="nv">instanceOf</span>?<span class="ss">(</span><span class="nv">ControlStatement</span><span class="ss">))</span>{
<span class="k">for</span><span class="ss">(</span><span class="nv">i</span> <span class="nv">in</span> <span class="nv">node</span> <span class="nv">as</span> <span class="nv">ControlStatement</span> <span class="nv">body</span><span class="ss">)</span>{
<span class="k">if</span><span class="ss">(</span><span class="nv">accessed</span><span class="ss">(</span><span class="nv">i</span>, <span class="nv">v</span><span class="ss">))</span> <span class="k">return</span> <span class="nv">true</span>
}
}
</code></pre></div>
<p>最终,这个功能完成了。按道理,这个问题会被解决。当我们试着用修正后的编译器时,问题来了:我们得到的依然是Segmentation Fault。</p>
<p>为什么?</p>
<p>问题来自与<strong>指针</strong>,因为一个闭包可以<code>间接</code>的走出当前的Scope。比如</p>
<div class="highlight"><pre><span></span><code><span class="n">foo</span> <span class="p">:</span><span class="o">=</span> <span class="k">func</span><span class="p">(){</span> <span class="o">//</span> <span class="n">closure</span> <span class="n">A</span>
<span class="p">}</span>
<span class="n">mystructB</span> <span class="p">:</span><span class="o">=</span> <span class="n">struct</span> <span class="n">new</span><span class="p">(</span><span class="n">foo</span><span class="p">,</span> <span class="n">varia</span><span class="p">)</span>
<span class="o">//</span><span class="n">mystructB</span> <span class="n">go</span> <span class="n">out</span> <span class="n">from</span> <span class="n">scope</span>
</code></pre></div>
<p>这里,mystruct是个结构体,我们不能阻止它走出scope,但它携带了闭包,也就是说我们要阻止他直接拷贝,这是很难的事情。另外一个办法是阻止它走出scope,当然,有一个看起来很合理的规则:</p>
<ul>
<li>如果一个指针从A赋值到了B,那么它的内容的所有权要从A转移到B。</li>
</ul>
<p>看起来有点眼熟? 对,这就是Rust。</p>
<p>好吧,让我们来想想如何实现Rust的推断系统——分析其里每一个Node都要有Owner,然后我们要在每一次resolve里都谨慎的判断owner是不是在改变,然后判断它有没有被返回,有没有被拷贝,有没有被当作initializer里的参数…… 这可能要重写编译器50%的代码。</p>
<p>好吧,我放弃了。</p>Introduction to The OOC Programming Language2015-01-27T00:00:00+09:002015-01-27T00:00:00+09:00Horsaltag:www.horsal.dev,2015-01-27:/introduction-to-the-ooc-programming-language.html<p>我想试一门新语言……但:</p>
<ul>
<li>我希望这门语言能简洁易懂 —— 排除了Perl/Rust...</li>
<li>我不想自己管理内存 —— 排除了C/C++/Object Pascal...</li>
<li>最好它能跟C差不多快 —— 排除了Python/Ruby...</li>
<li>并且最好能在任何地方编译&运行 —— 排除了D</li>
<li>我不想带着一个数百兆的运行库 —— 排除了Java</li>
<li><strong>我不怕Bug</strong> —— 欢迎来到OOC的世界</li>
</ul>
<h2>Why OOC?</h2>
<ul>
<li>
<p>Compile to C,所有的代码都会首先编译成C,然后由clang或者gcc编译成二进制代码
这意味着,只要你有一台能运行ooc编译器的电脑,那么你的代码就可以在几乎任何有C编译器的平台上运行。</p>
</li>
<li>
<p>Class,Function overloading,Extend,Operator Overloading.... OOC拥有绝大部分你所期待的高级语言的功能。
ooc借鉴了很多语言,尝试这把这些语言里优秀(并且有趣)的元素融合到一起。</p>
</li>
<li>
<p>Easy interface to c. OOC可以直接使用C的头文件,也可以在C里简单的使用ooc的函数。</p>
</li>
</ul>
<p>OOC的<a href="http://ooc-lang.org/">官方网站</a>里有更多介绍和参考资料 …</p><p>我想试一门新语言……但:</p>
<ul>
<li>我希望这门语言能简洁易懂 —— 排除了Perl/Rust...</li>
<li>我不想自己管理内存 —— 排除了C/C++/Object Pascal...</li>
<li>最好它能跟C差不多快 —— 排除了Python/Ruby...</li>
<li>并且最好能在任何地方编译&运行 —— 排除了D</li>
<li>我不想带着一个数百兆的运行库 —— 排除了Java</li>
<li><strong>我不怕Bug</strong> —— 欢迎来到OOC的世界</li>
</ul>
<h2>Why OOC?</h2>
<ul>
<li>
<p>Compile to C,所有的代码都会首先编译成C,然后由clang或者gcc编译成二进制代码
这意味着,只要你有一台能运行ooc编译器的电脑,那么你的代码就可以在几乎任何有C编译器的平台上运行。</p>
</li>
<li>
<p>Class,Function overloading,Extend,Operator Overloading.... OOC拥有绝大部分你所期待的高级语言的功能。
ooc借鉴了很多语言,尝试这把这些语言里优秀(并且有趣)的元素融合到一起。</p>
</li>
<li>
<p>Easy interface to c. OOC可以直接使用C的头文件,也可以在C里简单的使用ooc的函数。</p>
</li>
</ul>
<p>OOC的<a href="http://ooc-lang.org/">官方网站</a>里有更多介绍和参考资料。</p>
<h2>First Impression</h2>
<p>一个求Fibonacci数的程序看起来是这样:</p>
<div class="highlight"><pre><span></span><code><span class="n">fibonacci</span><span class="o">:</span> <span class="n">func</span><span class="o">(</span><span class="n">n</span><span class="o">:</span> <span class="n">Int</span><span class="o">)</span> <span class="o">-></span> <span class="n">Int</span><span class="o">{</span>
<span class="n">n</span> <span class="o"><</span> <span class="mi">2</span> <span class="o">?</span> <span class="n">n</span> <span class="o">:</span> <span class="n">fibonacci</span><span class="o">(</span><span class="n">n</span><span class="o">-</span><span class="mi">1</span><span class="o">)</span> <span class="o">+</span> <span class="n">fibonacci</span><span class="o">(</span><span class="n">n</span><span class="o">-</span><span class="mi">2</span><span class="o">)</span>
<span class="o">}</span>
<span class="k">for</span><span class="o">(</span><span class="n">i</span> <span class="k">in</span> <span class="mi">0</span><span class="o">..</span><span class="mi">30</span><span class="o">)</span> <span class="s2">"f(#{i})=#{fibonacci(i)}"</span> <span class="n">println</span><span class="o">()</span>
</code></pre></div>
<p>这段程序输出了前30个Fibonacci数。看起来是不是很容易懂?
函数通过<code>函数名: func(参数)->返回类型</code>来定义, 而它的内容则与C一模一样。同时,它不需要分号,不需要return,因为最后一个表达式的值会作为函数的返回值(当然也可以显式的使用return)。</p>
<h2>Getting Started</h2>
<p>如果在看了上面的代码之后你有兴趣继续,那么是时候准备一下编译环境了。<a href="https://github.com/fasterthanlime/rock">OOC的一个实现可以在github上简单的获得</a>。好的,让我们一步一步来:</p>
<ul>
<li>
<p>首先,clone编译器(rock)的源代码:</p>
<p>git clone https://github.com/fasterthanlime/rock</p>
</li>
<li>
<p>随后,至于要运行make即可</p>
<p>cd rock && make rescue</p>
</li>
</ul>
<p>rock是一个bootstrap的编译器,也就是说它本身是由OOC写成的。首先makefile会下载一套预编译的C源代码,用C编译得到的编译器来进一步编译现行代码。不出意外,make之后就会得到一个可以运行的ooc编译器了,它默认是<code>./bin/rock</code>,你可以用soft link把它放到任何地方。</p>
<p>使用rock编译ooc的程序非常简单,只需要简单的执行</p>
<div class="highlight"><pre><span></span><code>rock yourfile.ooc
</code></pre></div>
<p>就会得到可执行文件<code>yourfile</code>。</p>
<h2>Hello World</h2>
<p>为了确认编译器是不是正常工作,首先来编译一个简单的<code>Hello World</code></p>
<div class="highlight"><pre><span></span><code>"Hello world!" println()
</code></pre></div>
<p>把上面的代码保存在<code>hello.ooc</code>里,然后执行</p>
<div class="highlight"><pre><span></span><code>rock hello.ooc
</code></pre></div>
<p>你就会得到<code>hello</code>,执行它,看看结果吧。</p>
<p>当然,你也可以加一些佐料:</p>
<div class="highlight"><pre><span></span><code>rock --cc=clang --O3 --pr hello.ooc
</code></pre></div>
<p><code>--cc</code>用来指定使用的编译器,<code>--O3</code>与gcc下的意思是一样的,代表了最高优化,而<code>--pr</code>则表示是发行版。同样,你可以指定<code>--pg</code>来获得调试版。</p>
<h2>Basic Elements</h2>
<p>OOC来自与C,编译成C。因此绝大部分内容跟C十分类似,在这里仅仅介绍一些不同的元素。</p>
<h3>变量定义</h3>
<div class="highlight"><pre><span></span><code><span class="n">foo</span> <span class="o">:</span> <span class="n">Int</span>
<span class="n">bar</span> <span class="o">:</span> <span class="n">String</span>
<span class="n">cstyle</span> <span class="o">:</span> <span class="n">Float</span><span class="o">*</span>
<span class="n">p</span> <span class="o">:</span> <span class="n">Pointer</span>
</code></pre></div>
<p>ooc的变量定义类似Pascal,用<code>变量名:类型</code>的格式,同时需要注意的是所有的类型首字母都是大写,并且c里面的<code>void*</code>在ooc里是<code>Pointer</code>类型。当然,在变量定义时也可以赋值:</p>
<div class="highlight"><pre><span></span><code><span class="n">error</span><span class="o">:</span> <span class="n">String</span> <span class="o">=</span> <span class="s2">"System Error!"</span>
</code></pre></div>
<p>不仅如此,在变量定义有初始值时,ooc还允许省略类型,这个特性跟Go或者IO语言里的特性是类似的:</p>
<div class="highlight"><pre><span></span><code>error := "System Error!"
</code></pre></div>
<p>OOC里,变量的类型跟C一一对应,并且有非常简单的特征,比如:</p>
<table>
<thead>
<tr>
<th>ooc</th>
<th>c</th>
</tr>
</thead>
<tbody>
<tr>
<td>Char</td>
<td>char</td>
</tr>
<tr>
<td>UChar</td>
<td>unsigned char</td>
</tr>
<tr>
<td>Int</td>
<td>int</td>
</tr>
<tr>
<td>UInt</td>
<td>unsigned int</td>
</tr>
<tr>
<td>Float</td>
<td>float</td>
</tr>
</tbody>
</table>
<h3>循环与判断</h3>
<p>if/while语句与c完全一样:</p>
<div class="highlight"><pre><span></span><code><span class="k">if</span><span class="ss">(</span><span class="nv">c</span><span class="ss">)</span> { <span class="k">return</span> <span class="nv">true</span> }
<span class="k">else</span> { <span class="nv">renurn</span> <span class="nv">false</span> }
<span class="k">while</span><span class="ss">(</span><span class="nv">c</span><span class="ss">)</span>{ <span class="nv">c</span> <span class="o">-=</span> <span class="mi">1</span> }
</code></pre></div>
<p>稍微有些不同的是ooc的for:</p>
<div class="highlight"><pre><span></span><code><span class="k">for</span><span class="ss">(</span><span class="nv">i</span> <span class="nv">in</span> <span class="mi">0</span>..<span class="mi">100</span><span class="ss">)</span> { <span class="nv">a</span> <span class="o">+=</span> <span class="nv">i</span> }
</code></pre></div>
<p>ooc的for语句不支持c格式,for仅仅能够遍历一个范围,但这在普通情况下已经够用了。同时,对于c里的switch语句,ooc里使用的是match:</p>
<div class="highlight"><pre><span></span><code><span class="nv">match</span> <span class="ss">(</span><span class="nv">token</span><span class="ss">)</span> {
<span class="nv">case</span> <span class="s2">"</span><span class="s">if</span><span class="s2">"</span> <span class="o">=></span> <span class="mi">1</span>
<span class="nv">case</span> <span class="s2">"</span><span class="s">case</span><span class="s2">"</span> <span class="o">=></span> <span class="mi">2</span>
<span class="nv">case</span> <span class="s2">"</span><span class="s">while</span><span class="s2">"</span> <span class="o">=></span> <span class="mi">3</span>
<span class="nv">case</span> <span class="o">=></span> <span class="o">-</span><span class="mi">1</span>
}
</code></pre></div>
<p>这里每一个case执行完之后就会自动中断,并不会继续执行下一个case。同时,ooc的match可以用来配对几乎任何东西,即使对于对象(后述),只要它有<code>matches?</code>成员,就能够用来match-case里。</p>
<h3>函数</h3>
<p>就像在最初所展示的一样,ooc的函数定义类似Pascal(或者Ada):</p>
<div class="highlight"><pre><span></span><code><span class="n">myfunc</span><span class="o">:</span> <span class="n">func</span><span class="o">(</span><span class="n">argument</span><span class="o">:</span> <span class="n">Int</span><span class="o">,</span> <span class="n">argument2</span><span class="o">:</span> <span class="n">String</span><span class="o">)</span> <span class="o">-></span> <span class="n">Int</span><span class="o">{</span>
<span class="c1">// do what you want</span>
<span class="o">}</span>
</code></pre></div>
<p>函数的返回值不仅仅可以是单个的值,也可以是tuple:</p>
<div class="highlight"><pre><span></span><code><span class="n">swap</span><span class="o">:</span> <span class="n">func</span><span class="o">(</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">:</span> <span class="n">Int</span><span class="o">)</span> <span class="o">-></span> <span class="o">(</span><span class="n">Int</span><span class="o">,</span> <span class="n">Int</span><span class="o">){</span> <span class="o">(</span><span class="n">b</span><span class="o">,</span> <span class="n">a</span><span class="o">)</span> <span class="o">}</span>
<span class="o">(</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">)</span> <span class="o">:=</span> <span class="n">swap</span><span class="o">(</span><span class="mi">10</span><span class="o">,</span> <span class="mi">5</span><span class="o">)</span> <span class="c1">// result is a=5, b=10</span>
</code></pre></div>
<p>当然,跟其他语言一样,ooc也支持First-class Function:</p>
<div class="highlight"><pre><span></span><code><span class="n">foo</span><span class="o">:</span> <span class="n">func</span><span class="o">(</span><span class="n">add</span><span class="o">:</span> <span class="n">Func</span><span class="o">(</span><span class="n">Int</span><span class="o">,</span><span class="n">Int</span><span class="o">)-></span><span class="n">Int</span><span class="o">)</span> <span class="o">-></span> <span class="n">Int</span><span class="o">{</span> <span class="n">add</span><span class="o">(</span><span class="mi">10</span><span class="o">,</span> <span class="mi">20</span><span class="o">)</span> <span class="o">}</span>
<span class="n">bar</span><span class="o">:</span> <span class="n">func</span><span class="o">(</span><span class="n">a</span><span class="o">,</span><span class="n">b</span><span class="o">:</span> <span class="n">Int</span><span class="o">)</span> <span class="o">-></span> <span class="n">Int</span><span class="o">{</span> <span class="n">a</span> <span class="o">+</span> <span class="n">b</span> <span class="o">}</span>
<span class="n">foo</span><span class="o">(</span><span class="n">bar</span><span class="o">)</span>
<span class="n">foo</span><span class="o">(|</span><span class="n">x</span><span class="o">,</span> <span class="n">y</span><span class="o">|</span> <span class="n">x</span> <span class="o">+</span> <span class="n">y</span><span class="o">)</span>
</code></pre></div>
<p>这里foo函数的参数add是一个函数(或者也可以说是“函数指针”),需要注意这里的Func是大写——因为在ooc里,所有的类型的首字母都是大写。bar可以直接作为参数使用。同时,<code>|x, y| x + y</code>是lambda表达式,它定义了一个以参数x,y为输入的函数。当然,函数也可以作为变量(闭包):</p>
<div class="highlight"><pre><span></span><code><span class="n">fileFilter</span><span class="o">:</span> <span class="n">func</span><span class="o">(</span><span class="n">name</span><span class="o">:</span> <span class="n">String</span><span class="o">)</span> <span class="o">-></span> <span class="n">Bool</span><span class="o">{</span>
<span class="n">getName</span> <span class="o">:=</span> <span class="n">func</span><span class="o">(</span><span class="n">s</span><span class="o">:</span> <span class="n">String</span><span class="o">)</span> <span class="o">-></span> <span class="n">String</span><span class="o">{</span>
<span class="c1">// get name ...</span>
<span class="o">}</span>
<span class="c1">// use getName as function</span>
<span class="o">}</span>
</code></pre></div>
<p>对于指针,ooc里几乎与c是一样的:</p>
<div class="highlight"><pre><span></span><code><span class="n">mul2</span><span class="o">:</span> <span class="n">func</span><span class="o">(</span><span class="n">v</span> <span class="o">:</span> <span class="n">Int</span><span class="o">*){</span> <span class="n">mul2</span><span class="err">@</span> <span class="o">*=</span> <span class="mi">2</span> <span class="o">}</span>
<span class="n">mul2ref</span><span class="o">:</span> <span class="n">func</span><span class="o">(</span><span class="n">v</span> <span class="o">:</span> <span class="n">Int</span><span class="err">@</span><span class="o">){</span> <span class="n">mul2</span> <span class="o">*=</span> <span class="mi">2</span> <span class="o">}</span>
<span class="n">myvar</span> <span class="o">:=</span> <span class="mi">3</span>
<span class="n">mul2</span><span class="o">(</span><span class="n">myvar</span><span class="o">&)</span>
<span class="n">mul2ref</span><span class="o">(</span><span class="n">myvar</span><span class="o">&)</span>
</code></pre></div>
<p>这里定义了一个叫mul2的函数,它的它的定义与使用跟c并没有太大差别,唯一需要注意的就是在ooc里,访问指针不再是<code>*var</code>而是<code>var@</code>,类似的取地址也不是<code>&var</code>而是<code>var&</code>。在另外一个函数mul2ref里,我们用了Int@,它代表了变量v是一个参照——也就是说虽然它通过指针传递,但在使用时会被自动取值。</p>
<p>最后,函数是可以overloading的,比如下面的例子:</p>
<div class="highlight"><pre><span></span><code><span class="n">foo</span><span class="p">:</span> <span class="k">func</span> <span class="p">(</span><span class="n">i</span><span class="p">:</span> <span class="n">Int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Int</span><span class="p">{</span> <span class="n">i</span> <span class="o">*</span> <span class="mi">3</span> <span class="p">}</span>
<span class="n">foo</span><span class="p">:</span> <span class="k">func</span> <span class="o">~</span><span class="n">withj</span> <span class="p">(</span><span class="n">i</span><span class="p">:</span> <span class="n">Int</span><span class="p">,</span> <span class="n">j</span><span class="p">:</span> <span class="n">Int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Int</span><span class="p">{</span> <span class="n">i</span> <span class="o">*</span> <span class="n">j</span> <span class="p">}</span>
</code></pre></div>
<p>ooc里,函数的特征(signature)并不仅仅是函数名和参数列表,你还可以给任何一个函数添加”后缀“,也就是这里<code>~withj</code>的地方,通过后缀,可以实现函数的overloading。后缀跟函数名不同的地方在于,在执行时,即使不加后缀,编译器会自动寻找最合适的函数去执行,而通过制定后缀,可以硬性的制定一个函数, 比如:</p>
<div class="highlight"><pre><span></span><code>foo(1) // 执行第一个
foo(1,2) //执行第二个
foo ~withj (1,2) //执行第二个
</code></pre></div>
<h3>类与覆盖</h3>
<p>虽然最终编译成C,与其他高级语言一样,ooc里有类的概念,类的定义十分简单:</p>
<div class="highlight"><pre><span></span><code><span class="n">myclass</span><span class="p">:</span> <span class="k">class</span><span class="p">{</span> <span class="n">init</span><span class="p">:</span> <span class="k">func</span> <span class="p">}</span>
</code></pre></div>
<p>其中init是类的构造器,但它并不需要是static(静态)的,因为编译器会自动生成真正的构造器<code>new</code>,也就是说,在使用myclass,应该像下面这样:</p>
<div class="highlight"><pre><span></span><code>v := myclass new() // new is auto-generated static function
</code></pre></div>
<p>注意,每一个类必须有至少一个init,因为如果没有init,编译器就不会为它生成new,因此也就无法初始化。当然,你也可以自己定义<code>new</code>:</p>
<div class="highlight"><pre><span></span><code><span class="n">myclass</span><span class="p">:</span> <span class="k">class</span><span class="p">{</span>
<span class="n">new</span><span class="p">:</span> <span class="k">static</span> <span class="k">func</span> <span class="o">-></span> <span class="n">This</span><span class="p">{</span>
<span class="o">//</span> <span class="n">do</span> <span class="n">initialization</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>我们之前已经说了很多次,ooc里所有的类型都是首字母大写,因此代表着当前对象的this首字母大写之后代表这当前类。不过需要注意,自己定义new并不是见好事情,因为class最终还是由c里的struct实现的,因此你需要自己分配内存,管理初始化……等等。静态的new只在包装c函数时有用处,比如包装SDL-ttf时:</p>
<div class="highlight"><pre><span></span><code><span class="n">TTFFont</span><span class="p">:</span> <span class="n">cover</span> <span class="n">from</span> <span class="n">TTF_Font</span><span class="o">*</span><span class="p">{</span>
<span class="n">new</span><span class="p">:</span> <span class="k">static</span> <span class="k">func</span><span class="p">(</span><span class="n">filename</span><span class="p">:</span> <span class="nb nb-Type">String</span><span class="p">,</span> <span class="n">ptSize</span><span class="p">:</span> <span class="n">Int</span><span class="p">)</span> <span class="o">-></span> <span class="n">This</span><span class="p">{</span>
<span class="n">TTF</span> <span class="n">open</span><span class="p">(</span><span class="n">filename</span> <span class="n">toCString</span><span class="p">(),</span> <span class="n">ptSize</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">new</span><span class="p">:</span> <span class="k">static</span> <span class="k">func</span> <span class="o">~</span><span class="n">rw</span> <span class="p">(</span><span class="n">data</span><span class="p">:</span> <span class="n">Pointer</span><span class="p">,</span> <span class="n">freedata</span><span class="p">:</span> <span class="n">Int</span><span class="p">,</span> <span class="n">ptSize</span><span class="p">:</span> <span class="n">Int</span><span class="p">)</span> <span class="o">-></span> <span class="n">This</span><span class="p">{</span>
<span class="n">TTF</span> <span class="n">openRW</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">freedata</span><span class="p">,</span> <span class="n">ptSize</span><span class="p">)</span>
<span class="p">}</span>
<span class="o">....</span>
<span class="p">}</span>
</code></pre></div>
<p>这样就可以非常自然的将c函数转换成了类。当然,在这里代码使用了cover(覆盖),它在地位上等同与c的struct,但可以拥有函数,也可以被扩展,你可以认为cover是一个仅仅在使用c代码时才会用到的特殊类(class)。对于普通的类,只能继承(extends)其他的类,但对于覆盖,它既可以来自其他覆盖,也可以来自c的struct。比如:</p>
<div class="highlight"><pre><span></span><code><span class="n">Array</span><span class="o">:</span> <span class="n">cover</span> <span class="n">from</span> <span class="n">_lang_array__Array</span> <span class="o">{</span>
<span class="n">length</span><span class="o">:</span> <span class="n">extern</span> <span class="n">SizeT</span>
<span class="n">data</span><span class="o">:</span> <span class="n">extern</span> <span class="n">Pointer</span>
<span class="n">free</span><span class="o">:</span> <span class="n">extern</span><span class="o">(</span><span class="n">_lang_array__Array_free</span><span class="o">)</span> <span class="n">func</span>
<span class="o">}</span>
</code></pre></div>
<p>这段代码里<code>_lang_array__Array</code>是定义在c的头文件里的struct,而length和data都是它的成员,运用cover,可以很简单的将c中struct转换成ooc里可用的类型。</p>
<p>一个类的函数成员可以是static,可以是final。当它是final时,你是不能继承它的,比如:</p>
<div class="highlight"><pre><span></span><code><span class="n">foo</span><span class="p">:</span> <span class="k">class</span><span class="p">{</span>
<span class="n">init</span><span class="p">:</span> <span class="k">func</span>
<span class="n">a</span><span class="p">:</span> <span class="n">final</span> <span class="k">func</span>
<span class="p">}</span>
<span class="n">bar</span><span class="p">:</span> <span class="k">class</span> <span class="k">extends</span> <span class="n">foo</span><span class="p">{</span>
<span class="n">init</span><span class="p">:</span> <span class="k">func</span>
<span class="n">a</span><span class="p">:</span> <span class="k">func</span>
<span class="p">}</span>
</code></pre></div>
<p>这段代码会出现编译错误:</p>
<div class="highlight"><pre><span></span><code><span class="err">$</span><span class="w"> </span><span class="n">rock</span><span class="w"> </span><span class="n">ff</span><span class="p">.</span><span class="n">ooc</span><span class="w"></span>
<span class="n">test</span><span class="p">.</span><span class="nl">ooc</span><span class="p">:</span><span class="mi">8</span><span class="err">:</span><span class="mi">5</span><span class="w"> </span><span class="n">error</span><span class="w"> </span><span class="n">Can</span><span class="w"> </span><span class="ow">not</span><span class="w"> </span><span class="n">inherit</span><span class="w"> </span><span class="k">from</span><span class="w"> </span><span class="n">final</span><span class="w"> </span><span class="k">function</span><span class="w"> </span><span class="s1">'a'</span><span class="w"></span>
<span class="w"> </span><span class="nl">a</span><span class="p">:</span><span class="w"> </span><span class="n">func</span><span class="w"></span>
<span class="w"> </span><span class="o">~</span><span class="w"></span>
<span class="n">test</span><span class="p">.</span><span class="nl">ooc</span><span class="p">:</span><span class="mi">3</span><span class="err">:</span><span class="mi">5</span><span class="w"> </span><span class="n">info</span><span class="w"> </span><span class="p">...</span><span class="k">first</span><span class="w"> </span><span class="n">definition</span><span class="w"> </span><span class="n">was</span><span class="w"> </span><span class="nl">here</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nl">a</span><span class="p">:</span><span class="w"> </span><span class="n">final</span><span class="w"> </span><span class="n">func</span><span class="w"></span>
<span class="w"> </span><span class="o">~</span><span class="w"></span>
<span class="o">[</span><span class="n">FAIL</span><span class="o">]</span><span class="w"></span>
</code></pre></div>
<p>最后,类与覆盖都是可以被扩展(extend)的,它类似与ruby的extend,允许你向已经存在的类或覆盖里追加新的成员函数:</p>
<div class="highlight"><pre><span></span><code><span class="n">extend</span> <span class="n">Int</span><span class="p">{</span>
<span class="n">isOdd</span><span class="err">?</span><span class="p">:</span> <span class="k">func</span> <span class="o">-></span> <span class="n">Bool</span><span class="p">{</span>
<span class="n">this</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">1</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>这里我们给Int类型添加了一个<code>isOdd?</code>函数,这里问号没有特殊意义,仅仅是函数名的一部分,ooc允许函数名的最后一个字符是问号,用来表示这是一个“查询”函数,在编译成c是,问号会被翻译成字符串“query”。</p>
<p>好的现在我们可以是一下新定义的函数了,成员函数的访问与其他语言不一样,不是通过点(.)来实现, 而是简单的空格:</p>
<div class="highlight"><pre><span></span><code><span class="mf">1</span> <span class="n">isOdd</span><span class="err">?</span><span class="p">()</span> <span class="kr">to</span><span class="n">String</span><span class="p">()</span> <span class="kr">print</span><span class="n">ln</span><span class="p">()</span>
<span class="mf">2</span> <span class="n">isOdd</span><span class="err">?</span><span class="p">()</span> <span class="kr">to</span><span class="n">String</span><span class="p">()</span> <span class="kr">print</span><span class="n">ln</span><span class="p">()</span>
</code></pre></div>
<p>可以看到它们已经能够正常执行了。这里,isOdd?是Int的成员函数,而toString是Bool的成员函数,println()又是String(toString的返回类型)的成员函数,这点跟ruby非常接近。
同时,你可能已经注意到了,所有的函数调用都必须加上括号,否则编译器会抛出错误,那么有没有办法让不加括号呢? 答案是有的,至于要定义属性即可:</p>
<div class="highlight"><pre><span></span><code>extend Int{
isOdd : Bool {
get { this % 2 == 1 }
}
}
</code></pre></div>
<p>然后就可以像普通成员变量一样使用它了:</p>
<div class="highlight"><pre><span></span><code><span class="mf">1</span> <span class="n">isOdd</span> <span class="kr">to</span><span class="n">String</span><span class="p">()</span> <span class="kr">print</span><span class="n">ln</span><span class="p">()</span>
<span class="mf">2</span> <span class="n">isOdd</span> <span class="kr">to</span><span class="n">String</span><span class="p">()</span> <span class="kr">print</span><span class="n">ln</span><span class="p">()</span>
</code></pre></div>
<p>在这里,我们定义了一个只读的属性,对于这种属性,ooc提供了一个更简单的定义方式:(pdfe)</p>
<div class="highlight"><pre><span></span><code>extend Int{
isOdd ::= (this % 2 == 1)
}
</code></pre></div>
<p>这段代码的效果跟上面是完全一样的。</p>
<p>最后,类还支持运算符重载, 比如:</p>
<div class="highlight"><pre><span></span><code><span class="n">vector</span><span class="o">:</span> <span class="kd">class</span><span class="o">{</span>
<span class="n">operator</span> <span class="o">+</span> <span class="o">(</span><span class="n">v</span><span class="o">:</span> <span class="n">This</span><span class="o">)</span> <span class="o">-></span> <span class="n">This</span><span class="o">{</span>
<span class="c1">// add this and v</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div>
<h3>泛型</h3>
<p>ooc里存在泛型,但它完全不同与其他语言里的泛型,在ooc里,它可以“接受任何类型的变量”,而在其他语言里,泛型意味这“自动生成对应类型的实现”。这种差别决定了ooc的泛型远没有其他语言里强力。
你可以认为,ooc里的泛型是为集合(Collection)引入的,比如ArrayList和HashMap,对于普通函数,大量使用泛型不会有任何优势。</p>
<p>这里仅仅举一个简单的例子:</p>
<div class="highlight"><pre><span></span><code><span class="n">foo</span><span class="o">:</span> <span class="kd">class</span> <span class="o"><</span><span class="n">T</span><span class="o">>{</span>
<span class="n">data</span><span class="o">:</span> <span class="n">T</span><span class="o">*</span>
<span class="n">length</span><span class="o">:</span> <span class="n">Int</span>
<span class="n">init</span><span class="o">:</span> <span class="n">func</span><span class="o">(=</span><span class="n">length</span><span class="o">){</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">gc_malloc</span><span class="o">(</span><span class="n">T</span> <span class="n">size</span> <span class="o">*</span> <span class="n">length</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">myfoo</span> <span class="o">:=</span> <span class="n">foo</span><span class="o"><</span><span class="n">Int</span><span class="o">></span> <span class="k">new</span><span class="o">(</span><span class="mi">10</span><span class="o">)</span>
</code></pre></div>
<p>在这里,<code>=length</code>代表了参数直接赋值给成员,随后我们会根据参数大小分配一块内存给数据。这里的T可以是任何类型。</p>
<p>在ooc里,泛型是一个非常容易出错的部分,具体的设计思想可以参见我翻译的另一篇文章。在这里不多描述。</p>
<h3>库与头文件</h3>
<p>ooc里使用库是非常简单的,所需要的就是简单的import而已:</p>
<div class="highlight"><pre><span></span><code><span class="kn">import</span> <span class="nn">structs</span><span class="o">/</span><span class="n">ArrayList</span> <span class="o">//</span> <span class="n">使用ArrayList</span>
</code></pre></div>
<p>编译器会在<code>../sdk</code>或者<code>$OOC_LIBS</code>里寻找所有的库,然后自动使用和编译它们。对于C语言的头文件,可以通过include来使用:</p>
<div class="highlight"><pre><span></span><code><span class="k">include</span> <span class="nv">stdbool</span>
</code></pre></div>
<p>这样就可以使用stdbool里面定义的内容了。</p>
<h2>一个小例子</h2>
<p>这个小例子是Computer Langugae Benchamark Game里BinaryTree的一个实现,可以拿它跟C语言的版本来做比较。</p>
<div class="highlight"><pre><span></span><code><span class="n">Node</span><span class="o">:</span> <span class="kd">class</span><span class="o">{</span>
<span class="n">item</span><span class="o">:</span> <span class="n">Int</span>
<span class="n">left</span><span class="o">,</span><span class="n">right</span><span class="o">:</span> <span class="n">Node</span>
<span class="n">init</span><span class="o">:</span> <span class="n">func</span><span class="o">(</span><span class="n">depth</span><span class="o">:</span> <span class="n">Int</span><span class="o">,</span> <span class="o">=</span><span class="n">item</span><span class="o">){</span>
<span class="k">if</span><span class="o">(</span><span class="n">depth</span><span class="o"><=</span><span class="mi">0</span><span class="o">)</span> <span class="k">return</span>
<span class="n">left</span> <span class="o">=</span> <span class="n">Node</span> <span class="k">new</span><span class="o">(</span><span class="n">depth</span><span class="o">-</span><span class="mi">1</span><span class="o">,</span> <span class="mi">2</span><span class="o">*</span><span class="n">item</span><span class="o">-</span><span class="mi">1</span><span class="o">);</span>
<span class="n">right</span> <span class="o">=</span> <span class="n">Node</span> <span class="k">new</span><span class="o">(</span><span class="n">depth</span><span class="o">-</span><span class="mi">1</span><span class="o">,</span> <span class="mi">2</span><span class="o">*</span><span class="n">item</span><span class="o">);</span>
<span class="o">}</span>
<span class="n">itemCheck</span><span class="o">:</span> <span class="n">func</span><span class="o">()</span> <span class="o">-></span> <span class="n">Int</span><span class="o">{</span>
<span class="k">if</span><span class="o">(!</span><span class="n">left</span><span class="o">)</span> <span class="k">return</span> <span class="n">item</span>
<span class="k">return</span> <span class="n">item</span><span class="o">+</span><span class="n">left</span> <span class="n">itemCheck</span><span class="o">()-</span><span class="n">right</span> <span class="n">itemCheck</span><span class="o">()</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">mindep</span> <span class="o">:=</span> <span class="mi">4</span>
<span class="n">main</span><span class="o">:</span> <span class="n">func</span><span class="o">(</span><span class="n">argv</span><span class="o">:</span> <span class="n">Int</span><span class="o">,</span> <span class="n">argc</span><span class="o">:</span> <span class="n">CString</span><span class="o">*)</span> <span class="o">-></span> <span class="n">Int</span><span class="o">{</span>
<span class="n">depth</span><span class="o">:</span> <span class="n">Int</span>
<span class="k">if</span><span class="o">(</span><span class="n">argv</span><span class="o">></span><span class="mi">1</span><span class="o">)</span> <span class="n">depth</span> <span class="o">=</span> <span class="n">argc</span><span class="o">[</span><span class="mi">1</span><span class="o">]</span> <span class="n">toString</span><span class="o">()</span> <span class="n">toInt</span><span class="o">()</span>
<span class="k">else</span> <span class="k">return</span> <span class="mi">1</span>
<span class="n">stretch</span> <span class="o">:=</span> <span class="n">depth</span><span class="o">+</span><span class="mi">1</span>
<span class="n">check</span> <span class="o">:=</span> <span class="n">Node</span> <span class="k">new</span><span class="o">(</span><span class="n">stretch</span><span class="o">,</span><span class="mi">0</span><span class="o">)</span> <span class="n">itemCheck</span><span class="o">()</span>
<span class="s2">"stretch tree of depth %d\t check: %d"</span> <span class="n">printfln</span><span class="o">(</span><span class="n">stretch</span><span class="o">,</span> <span class="n">check</span><span class="o">)</span>
<span class="n">longlived</span> <span class="o">:=</span> <span class="n">Node</span> <span class="k">new</span><span class="o">(</span><span class="n">depth</span><span class="o">,</span><span class="mi">0</span><span class="o">)</span>
<span class="n">i</span> <span class="o">:=</span> <span class="n">mindep</span>
<span class="k">while</span><span class="o">(</span><span class="n">i</span><span class="o"><=</span><span class="n">depth</span><span class="o">){</span>
<span class="n">iterations</span> <span class="o">:=</span> <span class="mi">1</span><span class="o"><<(</span><span class="n">depth</span><span class="o">-</span><span class="n">i</span><span class="o">+</span><span class="n">mindep</span><span class="o">)</span>
<span class="n">check</span><span class="o">:</span> <span class="n">Int</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">for</span><span class="o">(</span><span class="n">j</span> <span class="k">in</span> <span class="mi">1</span><span class="o">..</span><span class="n">iterations</span><span class="o">+</span><span class="mi">1</span><span class="o">){</span>
<span class="n">check</span> <span class="o">+=</span> <span class="n">Node</span> <span class="k">new</span><span class="o">(</span><span class="n">i</span><span class="o">,</span><span class="n">j</span><span class="o">)</span> <span class="n">itemCheck</span><span class="o">()</span>
<span class="n">check</span> <span class="o">+=</span> <span class="n">Node</span> <span class="k">new</span><span class="o">(</span><span class="n">i</span><span class="o">,-</span><span class="n">j</span><span class="o">)</span> <span class="n">itemCheck</span><span class="o">()</span>
<span class="o">}</span>
<span class="s2">"%d\ttrees of depth %d\t check: %d"</span> <span class="n">printfln</span><span class="o">(</span><span class="n">iterations</span><span class="o">*</span><span class="mi">2</span><span class="o">,</span> <span class="n">i</span><span class="o">,</span> <span class="n">check</span><span class="o">);</span>
<span class="n">i</span><span class="o">+=</span><span class="mi">2</span>
<span class="o">}</span>
<span class="s2">"long lived tree fo depth %d\t check %d"</span> <span class="n">printfln</span><span class="o">(</span><span class="n">depth</span><span class="o">,</span> <span class="n">longlived</span><span class="o">.</span><span class="na">itemCheck</span><span class="o">());</span>
<span class="k">return</span> <span class="mi">0</span>
<span class="o">}</span>
</code></pre></div>
<p>至于这个程序的执行结果:</p>
<table>
<thead>
<tr>
<th>ooc</th>
<th>c</th>
</tr>
</thead>
<tbody>
<tr>
<td>16.95</td>
<td>16.45</td>
</tr>
</tbody>
</table>
<p>可以看到,二者几乎没有差别。</p>
<h2>结语</h2>
<p>OOC是一个很不错的第二,或者第三语言。虽然有公司在用ooc做些事情,但我并不认为那很明智。的确,ooc兼具执行效率和开发效率,但目前它的编译器还远远没有完美。如果你有兴趣,那么不妨<a href="https://github.com/fasterthanlime/rock">fork一下</a>,让ooc的编译器更加完善。</p>OOC, Generics and Flawed Design2015-01-24T00:00:00+09:002015-01-24T00:00:00+09:00Horsaltag:www.horsal.dev,2015-01-24:/ooc-generics-and-flawed-design.html<p>昨天在github上讨论OOC的一些特性,随后这个语言的设计者写了这篇文章。
虽然大部分都在谈ooc的编译器设计,但更多的内容在于程序设计的思想,复杂度,维护上面。我希望这篇文章能对读者有哪怕一丁点的帮助。</p>
<p>原文地址:http://fasterthanlime.com/blog/2015/ooc-generics-and-flawed-designs</p>
<p>ooc可能是我(注:原作者)最得意的成果,但同时,对我来说它也是让我头疼不已的“刺头”。</p>
<p>一个重要的原因就是它的设计(构架)并不让人满意,并且在当下,很多东西没法简单的解决。但不要误解: 在某种意义上,所有的设计都是“糟糕”的,不论是由一个人掌管,或者有社区驱动。没有任何一个设计能被称为完美。尽管我们并没有关于“完美”的定义和度量。</p>
<p>现在回到ooc上来,一些话题反复的被提起,比如interface(接口)和cover(覆盖)。这个问题的原因一部分来自与人们会简单的认为这些概念应该跟其他语言里的一样,因为它们的名字完全相同。但事实并不是这样,纵使拥有相同的名字,它们的意义也不尽相同。如果这些概念跟其他语言里的概念完全一样,那么ooc会是一个简单的C …</p><p>昨天在github上讨论OOC的一些特性,随后这个语言的设计者写了这篇文章。
虽然大部分都在谈ooc的编译器设计,但更多的内容在于程序设计的思想,复杂度,维护上面。我希望这篇文章能对读者有哪怕一丁点的帮助。</p>
<p>原文地址:http://fasterthanlime.com/blog/2015/ooc-generics-and-flawed-designs</p>
<p>ooc可能是我(注:原作者)最得意的成果,但同时,对我来说它也是让我头疼不已的“刺头”。</p>
<p>一个重要的原因就是它的设计(构架)并不让人满意,并且在当下,很多东西没法简单的解决。但不要误解: 在某种意义上,所有的设计都是“糟糕”的,不论是由一个人掌管,或者有社区驱动。没有任何一个设计能被称为完美。尽管我们并没有关于“完美”的定义和度量。</p>
<p>现在回到ooc上来,一些话题反复的被提起,比如interface(接口)和cover(覆盖)。这个问题的原因一部分来自与人们会简单的认为这些概念应该跟其他语言里的一样,因为它们的名字完全相同。但事实并不是这样,纵使拥有相同的名字,它们的意义也不尽相同。如果这些概念跟其他语言里的概念完全一样,那么ooc会是一个简单的C++/Java/Ruby的底层实现。但ooc不是,ooc与上述每一种语言都有重合,而我(注:原作者)和过去六年里一起帮过我了人们,则尝试着把这些特性融合到一起。</p>
<h2>关于泛型</h2>
<p>这是ooc里对于泛型的定义: 它(范型)是一个指向未知类型的指针,同时保存着对应的值(with value semantics)。</p>
<p>首先让我们来看一个简单的泛型函数(generic function):</p>
<div class="highlight"><pre><span></span><code><span class="n">add</span><span class="p">:</span> <span class="k">func</span> <span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="n">T</span><span class="p">)</span> <span class="o">-></span> <span class="n">T</span><span class="p">{</span>
<span class="n">a</span> <span class="o">+</span> <span class="n">b</span>
<span class="p">}</span>
</code></pre></div>
<p>译者注: 对不熟悉ooc的人,解释一下上面的代码: 上面的代码定义了一个叫做add的函数,参数是a,b(泛型),返回值是二者之和。</p>
<p>现在,如果ooc的generic(泛型)是template(模板),那么这段代码没有任何问题——因为它会根据类型来生成“add”的实现。如果<code>a</code>和<code>b</code>的类型不一样,抛出错误,如果T没有“+”的运算符重载,抛出错误,等等等等。</p>
<p>但这并不是ooc里泛型的工作原理。对于一个ooc的泛型,在__“运行时”__,你所知道是只有:</p>
<ul>
<li>它在内存的哪一部分(指向它的指针)</li>
<li>它是什么类型(名字+大小+指针大小)</li>
</ul>
<p>但在__“编译时”__,你什么都得不到。对,什么都得不到。在编译期,你能对泛型所做的就是“复制(移动)它到不同地方”。对于Collections(集合)来说,这些足够了,因为在绝大多数情况下,<code>ArrayList<T></code>, <code>HashMap<K,V></code>等等,并不需要知道关于里面元素类型的太多信息。</p>
<p>好吧,我撒了个谎,他们仍然需要知道一些内容。但大多数情况下是针对比较和哈希。比如,<code>ArrayList<T> indexOf(value: T) -> Int</code>,我们需要比较两个T类型的值,否则我们没法从数组里面搜索到它。</p>
<p>对于<code>HashMap<K,V></code>,我们则需要对“K”做哈希,否则我们没法将值保存起来,也没法在搜索的时候得到键的哈希值。同样,为了避免冲突,我们还需要比较K的值。</p>
<p>在ooc的标准库(注:在ooc里,标准库被称作sdk),我们是怎么做的呢?我们用了一些小技俩(cheating)。你应该记得我说过,在运行时,我们只知道泛型的地址和类型。好的,我们可以这么写:</p>
<div class="highlight"><pre><span></span><code><span class="n">add</span><span class="p">:</span> <span class="k">func</span> <span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="n">T</span><span class="p">)</span> <span class="o">-></span> <span class="n">T</span> <span class="p">{</span>
<span class="k">match</span> <span class="n">T</span> <span class="p">{</span>
<span class="n">case</span> <span class="n">Int</span> <span class="o">=></span>
<span class="n">a</span> <span class="k">as</span> <span class="n">Int</span> <span class="o">+</span> <span class="n">b</span> <span class="k">as</span> <span class="n">Int</span>
<span class="n">case</span> <span class="o">=></span>
<span class="n">Exception</span> <span class="n">new</span><span class="p">(</span><span class="s2">"Don't know how to add type #{T name}"</span><span class="p">)</span> <span class="n">throw</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>现在,很多人会开始认为这是一个十分糟糕的设计,我(注:原作者)甚至不应该称这个为“泛型”,等等。但这就是它的工作原理。纵使向6年前的我抱怨为什么选了这么一种设计,对我来说也没有任何意义。我仅仅是在试着解释它的工作原理,从而消除未来可能发生的误解。</p>
<p>好的,让我们回到HashMap(注:标准库里的一个实现),当你建立一个HashMap时,它会跟上面类似,根据K的类型选择最好的哈希函数,这个取决与K是不是字符串,是不是数字,或者是其他的什么东西。</p>
<h2>类型值(Type as values)</h2>
<p>在Github的一个问题报告里(注:就是我报告的……),有人说对于下面这个函数</p>
<div class="highlight"><pre><span></span><code><span class="n">identity</span><span class="p">:</span> <span class="k">func</span> <span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">(</span><span class="n">t</span><span class="p">:</span> <span class="n">T</span><span class="p">)</span> <span class="o">-></span> <span class="n">T</span> <span class="p">{</span> <span class="n">t</span> <span class="p">}</span>
</code></pre></div>
<p>这段代码可以运行:</p>
<div class="highlight"><pre><span></span><code>identity(42)
</code></pre></div>
<p>但是这个不能:</p>
<div class="highlight"><pre><span></span><code>identity<Int>(42)
</code></pre></div>
<p>但如果你定义了一个范型类:</p>
<div class="highlight"><pre><span></span><code><span class="n">Cell</span><span class="o">:</span> <span class="kd">class</span> <span class="o"><</span><span class="n">T</span><span class="o">>{</span>
<span class="n">t</span><span class="o">:</span> <span class="n">T</span>
<span class="n">init</span><span class="o">:</span> <span class="n">func</span>
<span class="o">}</span>
</code></pre></div>
<p>那么你就可以用任意一种:</p>
<div class="highlight"><pre><span></span><code>cell := Cell<Int> new()
</code></pre></div>
<p>通常,这是一个糟糕的,但同时多少保持了内部一致性的设计。对于这种东西,有一个原因。</p>
<p>对于Cell(泛型类),构造器(constructor)没有接受任何关于T的实例——也就是说在构造器里,我们没法推断T的类型,因此我们需要在尖括号里指定一个类型。泛型可以作为类型参数——这是它的工作原理。</p>
<p>但函数不同。在相同的情况下,如果推断得来的类型不是你想要的类型(在下面的例子里是SSizeT),那么你总是可以在调用函数是强制转换成你想要的:</p>
<div class="highlight"><pre><span></span><code><span class="n">identity</span><span class="p">(</span><span class="mi">42</span><span class="p">)</span> <span class="c1">// -> SSizeT</span>
<span class="n">identity</span><span class="p">(</span><span class="mi">42</span> <span class="kr">as</span> <span class="n">Int</span><span class="p">)</span> <span class="c1">// -> Int</span>
</code></pre></div>
<p>有些时候,有可能根本没法在编译期根据参数推断类型,比如:</p>
<div class="highlight"><pre><span></span><code><span class="n">getSome</span><span class="p">:</span> <span class="k">func</span> <span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="o">-></span> <span class="n">T</span><span class="p">{</span>
<span class="k">match</span> <span class="n">T</span> <span class="p">{</span>
<span class="n">case</span> <span class="n">Int</span> <span class="o">=></span>
<span class="mi">4</span> <span class="o">//</span> <span class="n">guaranteed</span> <span class="n">by</span> <span class="n">fair</span> <span class="n">dice</span> <span class="n">roll</span>
<span class="n">case</span> <span class="n">Float</span> <span class="o">=></span>
<span class="mf">0.8</span>
<span class="n">case</span> <span class="o">=></span>
<span class="n">raise</span><span class="p">(</span><span class="s2">"Can't get some of #{T name}"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>这个函数永远不可能被正常调用。在别的语言里,你可以简单的用<code>getSome<Int>()</code>或者<code>getSome<Float>()</code>里指定类型,但在ooc里,你不能。取而代之,你需要把类型T放进参数列表里,比如:</p>
<div class="highlight"><pre><span></span><code><span class="n">getSome</span><span class="p">:</span> <span class="k">func</span> <span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">(</span><span class="n">T</span><span class="p">:</span> <span class="n">Class</span><span class="p">)</span> <span class="o">-></span> <span class="n">T</span><span class="p">{</span>
<span class="k">match</span> <span class="n">T</span><span class="p">{</span>
<span class="o">//</span> <span class="n">etc</span><span class="o">...</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>现在当你可以通过<code>getSome(Int)</code>或者<code>getSome(Float)</code>来调用这个函数了。再次回来,在这个问题上,我们总是可以无休止的去争论到底那一种才是更好的方法。对于其他的人来说,“下面这样是可以的:传递类型时用见括号,传递参数时用园括号”,而六年前的我,则对于标记有着独特的信念:类型仅仅是一个值,跟其他的值一样,我们不必在它们的周围加上各种各样的限制(注:意指括号)。</p>
<p>六年前的我是这么认为的:如果我们有一个“partial”的原语,那么我们就能简单的把<code>getSome(Int)</code>转换为<code>getSomeInt</code>,然后在其他地方用这个新函数,并认为它返回Int型:</p>
<div class="highlight"><pre><span></span><code><span class="o">//</span> <span class="n">fictional</span> <span class="n">code</span> <span class="p">(</span><span class="mi">6</span><span class="o">-</span><span class="n">years</span><span class="o">-</span><span class="n">ago</span><span class="o">-</span><span class="n">me</span> <span class="n">way</span><span class="p">)</span>
<span class="o">//</span> <span class="err">伪代码(</span><span class="mi">6</span><span class="err">年前的我的做法)</span>
<span class="n">getSome</span><span class="p">:</span> <span class="k">func</span> <span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">(</span><span class="n">T</span><span class="p">:</span> <span class="n">Class</span><span class="p">)</span> <span class="o">-></span> <span class="n">T</span> <span class="p">{</span> <span class="o">/*</span> <span class="n">see</span> <span class="n">above</span> <span class="o">*/</span> <span class="p">}</span>
<span class="n">getSomeInt</span><span class="p">:</span> <span class="n">partial</span><span class="p">(</span><span class="n">getSome</span><span class="p">,</span> <span class="n">Int</span><span class="p">)</span>
<span class="n">eng</span> <span class="p">:</span><span class="o">=</span> <span class="n">GameEngine</span> <span class="n">new</span><span class="p">()</span>
<span class="n">eng</span> <span class="n">setRandomNumberGenerator</span><span class="p">(</span><span class="n">getSomeInt</span><span class="p">)</span>
</code></pre></div>
<p>反过来看,在尖括号风格下面,你能怎么做? 好吧,跟你自己定义一个新的函数并没有什么太大区别,或者使用闭包:</p>
<div class="highlight"><pre><span></span><code><span class="o">//</span> <span class="n">fictional</span> <span class="n">code</span> <span class="p">(</span><span class="mi">6</span><span class="o">-</span><span class="n">years</span><span class="o">-</span><span class="n">ago</span><span class="o">-</span><span class="n">me</span> <span class="n">way</span><span class="p">)</span>
<span class="o">//</span> <span class="err">伪代码(</span><span class="mi">6</span><span class="err">年前的我的做法)</span>
<span class="n">getSome</span><span class="p">:</span> <span class="k">func</span> <span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">(</span><span class="n">T</span><span class="p">:</span> <span class="n">Class</span><span class="p">)</span> <span class="o">-></span> <span class="n">T</span> <span class="p">{</span> <span class="o">/*</span> <span class="n">see</span> <span class="n">above</span> <span class="o">*/</span> <span class="p">}</span>
<span class="n">eng</span> <span class="p">:</span><span class="o">=</span> <span class="n">GameEngine</span> <span class="n">new</span><span class="p">()</span>
<span class="n">eng</span> <span class="n">setRandomNumberGenerator</span><span class="p">(</span><span class="o">||</span> <span class="n">getSome</span><span class="o"><</span><span class="n">Int</span><span class="o">></span><span class="p">())</span>
</code></pre></div>
<p>这样你失去了一些高阶(high-order)函数工具。再次重申,这在目前的ooc里并不是一个大问题,因为标准库里根本没有partial这么一个东西。</p>
<h2>回到值上来</h2>
<p>我前面已经提到,相对范型值做一些有用的动作是很困难的。除非你把它们强制转换回“实”类型:</p>
<div class="highlight"><pre><span></span><code><span class="n">someFunction</span><span class="o">:</span> <span class="n">func</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">(</span><span class="n">t</span><span class="o">:</span> <span class="n">T</span><span class="p">)</span><span class="err">{</span>
<span class="o">//</span> <span class="n n-Quoted">`t`</span> <span class="k">is</span> <span class="n">kind</span> <span class="k">of</span> <span class="n">useless</span>
<span class="o">//</span> <span class="n n-Quoted">`t`</span><span class="n">没有任何用处</span>
<span class="n">u</span> <span class="o">:=</span> <span class="n">t</span> <span class="k">as</span> <span class="kt">Int</span>
<span class="o">//</span> <span class="n">Now</span><span class="p">,</span> <span class="k">with</span> <span class="n n-Quoted">`u`</span> <span class="n">we</span> <span class="n">can</span> <span class="k">do</span> <span class="n">anything</span> <span class="n">we</span> <span class="n">want</span><span class="p">.</span>
<span class="o">//</span> <span class="n">现在,我们可以对</span><span class="n n-Quoted">`u`</span><span class="n">做我们想做的事情了</span>
<span class="err">}</span>
</code></pre></div>
<p>把泛型转换成实类型是一个很危险的操作。因为一旦类型错误,我们就只能得到一些乱七八糟的乱码——编译器并不会做任何检查,因为它相信你知道自己在做什么(当然,事后来看,有可能是错误的操作)。但六年前的我反驳,有一种相当不错的机制可以让我们相当安全的把泛型转换会实类型:</p>
<div class="highlight"><pre><span></span><code><span class="n">someFunction</span><span class="o">:</span> <span class="n">func</span> <span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">(</span><span class="n">t</span><span class="o">:</span> <span class="n">T</span><span class="p">)</span> <span class="err">{</span>
<span class="k">match</span> <span class="n">t</span> <span class="err">{</span>
<span class="k">case</span> <span class="n">u</span><span class="o">:</span> <span class="kt">Int</span> <span class="o">=></span>
<span class="o">//</span> <span class="n">we</span><span class="s1">'re sure `t` was an Int and now we can use it as u</span>
<span class="s1"> // 我们确定`t`是Int类型,因此可以用u</span>
<span class="s1"> case =></span>
<span class="s1"> // error handling goes here.</span>
<span class="s1"> // 这里将会抛出错误</span>
<span class="s1"> }</span>
<span class="s1">}</span>
</code></pre></div>
<p>这些就是泛型的工作原理的一切。在上面的设计之上,(类型)推断器应该相当的聪明(仅仅是设想……),这样你可以省略尽可能多的类型判断。</p>
<p>举个例子,下面这段代码在前面叙述的系统里是可以正常执行的:</p>
<div class="highlight"><pre><span></span><code><span class="n">Sorter</span><span class="p">:</span> <span class="k">class</span> <span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span>
<span class="n">compare</span><span class="p">:</span> <span class="n">Func</span> <span class="p">(</span><span class="n">T</span><span class="p">,</span> <span class="n">T</span><span class="p">)</span> <span class="o">-></span> <span class="n">Int</span>
<span class="n">init</span><span class="p">:</span> <span class="k">func</span> <span class="p">(</span><span class="o">=</span><span class="n">compare</span><span class="p">)</span> <span class="p">{}</span>
<span class="n">sort</span><span class="p">:</span> <span class="k">func</span> <span class="p">(</span><span class="n">l</span><span class="p">:</span> <span class="n">List</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">)</span> <span class="o">-></span> <span class="n">List</span><span class="o"><</span><span class="n">T</span><span class="o">></span> <span class="p">{</span> <span class="o">/*</span> <span class="o">...</span> <span class="o">*/</span> <span class="p">}</span>
<span class="p">}</span>
<span class="n">s</span> <span class="p">:</span><span class="o">=</span> <span class="n">Sorter</span><span class="o"><</span><span class="n">Int</span><span class="o">></span><span class="p">(</span><span class="o">|</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="o">|</span> <span class="p">(</span><span class="n">a</span> <span class="o"><</span> <span class="n">b</span><span class="p">)</span> <span class="err">?</span> <span class="o">-</span><span class="mi">1</span> <span class="p">:</span> <span class="p">(</span><span class="n">a</span> <span class="o">></span> <span class="n">b</span> <span class="err">?</span> <span class="mi">1</span> <span class="p">:</span> <span class="mi">0</span><span class="p">))</span>
<span class="n">s</span> <span class="n">sort</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">]</span> <span class="k">as</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Int</span><span class="o">></span><span class="p">)</span>
</code></pre></div>
<h2>“维护”是一个艰难的工作</h2>
<p>现在,每当有人误解泛型,覆盖,接口,并且强调他们应该怎么工作,因为“在语言X或Y里有同样名字的特性”,我都不知道该如何应对。</p>
<p>有些时候,它们是正常的bug,我们可以修复它。对于这种情况,我很乐意用我的空闲时间来帮一些忙。但有些时候,这是不可能的。这经常是一个二选一问题:</p>
<ul>
<li>对应1%的情况,“撞大运”,给出12种人们应该知道的状况(直到下一个人踩到另一个地雷,然后抱怨这个问题,这看起来好公平!)。在已经补丁累累的编译器上不停的打新补丁。</li>
<li>选择大路。讨论这个特性应该怎么运转,然后重写10%,20%,或者50%的编译器代码(6年前的我还不懂的怎么去写一个真正的模块化编译器,却依然向里面塞了一堆东西——之后看起来很棒,现在确麻烦不断),然后80%的现存代码都无法编译(并不是很多,但依然有成百上千行,并且人们每天都在用,包括我)</li>
</ul>
<p>第一个想法,它经常被用到。但它并不解决问题。当你有一个好的,坚实的设计之时,补丁可以让你想要的东西变得更好。但当你的设计很糟糕是,就像6年前的我,继续添加补丁仅仅是九牛一毛,并且在技术上也是站不住脚的,并且会有更多用户会在这条路上满怀失望。</p>
<p>这就像在你的游戏里,你有一个问题: “玩家不能爬不平的面”。然后会有人站出来说,“看,在第三关(World3)里,湖后面的小山里有个梯子,但似乎没法爬,这是补丁”。然后这个补丁是只要是来自与IP“174.”的玩家,站在梯子下面,然后按住Ctrl+Shift+Alt向上看,就会被立即传送到顶端。当然,“你没法下梯子,这会在下个版本里解决!”</p>
<p>试想如果没有下个版本,因为开发者注意到这个问题远比自己最初的想象(爬湖后面小山的梯子)要复杂的多。只有光滑平面才能工作。因为他们并没有时间去重写整个游戏引擎,使得不光滑平面也能在考虑返回之内。因此他们开始看下一个游戏。纵使这个补丁被采用,也仅仅是烂在那里——因为没有人会想到去按Ctrl+Shift+Alt的同时去看某个特定的位置。</p>
<p>对你,游戏开发者,来说,你应该把“游戏只支持在光滑平面”写进文档。但你可能并不想这么做,因为你可能会羞与记录它,因为它象征着你没有时间和精力(别人会认为是技术)去修复这么一个问题。总之,作为一个专家,你应该做的更好。</p>
<p>于是你没有把这个问题放到游戏主页上,因为那对宣传是莫大的负作用。这是一个很好的选择,它平衡了谎言和对自己的伤害。并且,你开始另一个游戏工程,但你依然会偶尔回到之前那个游戏里,去玩一玩,然后加点因东西。因此它并没有完全的被抛弃,但已经不再是你的重点。对于长期玩家来说,这非常棒,因为他们知道这些事情,并且他们依然能从光滑平面里获得很多快乐。</p>
<h2>结论</h2>
<p>在这个比喻变得冗长之前,我该收笔了。但希望你能明白我再说什么。当你在处理一个设计糟糕的系统时(某种概念上,是程序设计里的所有东西),有些时候它值得你去尝试,并且改变一些东西,从而让它变得更好。但大多数时候,最好的方法是在这个糟糕设计的系统的限制之下,试一下它到底能不能完成你想要的目标。如果不能,那么最简单的方法就是换到别的(设计糟糕的)系统里(然后你会后悔,因为这个系统没有之前那个系统里你想要的特性……,然后重复)。</p>
<p>下面这些曾经在我的睡梦中挥之不去:怎么才能让它“完美”?它到底是不是符合每个人的用途用法?代码是不是漂亮?速度呢?浪费光阴的年轻人,到底能不能在生命终场之前也继续写代码?</p>
<p>但现在不会了(至少不像以前那样),因为现在我很满足与组合这些设计糟糕的系统,然后去做些我想做的事情。我也会留下点美丽的片段来让我的享受更具有艺术性。有些时候是雪人(Unicode Snowman,unicode的字符之一),有些时候是奇怪的提示信息。这很幼稚,这篇文章看起来像是在逃避责任,在看到有人希望能够改变ooc的大半,从而让他变得更好时,我依然会叹气,但我认为这不会发生。那是C++或者Scala的角色,我不想去承担那种责任。</p>Build Compiler with LALR(1) Parser Generator(III)2014-06-18T00:00:00+09:002014-06-18T00:00:00+09:00Horsaltag:www.horsal.dev,2014-06-18:/build-compiler-with-lalr1-parser-generatoriii.html<p><a href="https://www.horsal.dev/build-compiler-with-lalr1-parser-generatorii.html">在经过前两章的奋战</a>之后, 我们对于Parser终于多少有了点理解——虽然还有一些遗留问题。至少现在已经可以把一门简单的语言编译成bytecode然后解析执行了。不过就像上一章里所说的,有些时候我们还希望能够把程序编译成可执行文件,在这一篇文章里我们就来做这件事情,不用担心,并不是所有的工作都是由compiler来做的。在这里,基于前一章编译出来的ByteCode,我们首先生成汇编代码,然后通过NASM编译成elf object,最后通过ld链接成可执行文件。</p>
<p>首先来熟悉下最简单的汇编程序,比如一个Hello,World可以写成下面的样子:</p>
<div class="highlight"><pre><span></span><code><span class="k">global </span><span class="nv">_start</span>
<span class="k">section </span><span class="nv">.text</span>
<span class="nl">_print:</span>
<span class="nf">push</span> <span class="nb">rdx</span>
<span class="nf">push</span> <span class="nb">rax</span>
<span class="nf">push</span> <span class="nb">rbx</span>
<span class="nf">mov</span> <span class="nb">edx</span><span class="p">,</span> <span class="mi">1</span>
<span class="nf">mov</span> <span class="nb">eax</span><span class="p">,</span> <span class="mi">4</span>
<span class="nf">mov</span> <span class="nb">ebx</span><span class="p">,</span> <span class="mi">1</span>
<span class="nf">int</span> <span class="mh">0x80</span>
<span class="nf">pop</span> <span class="nb">rbx</span>
<span class="nf">pop</span> <span class="nb">rax</span>
<span class="nf">pop</span> <span class="nb">rdx</span>
<span class="nf">ret</span>
<span class="nl">_start:</span>
<span class="nf">mov …</span></code></pre></div><p><a href="https://www.horsal.dev/build-compiler-with-lalr1-parser-generatorii.html">在经过前两章的奋战</a>之后, 我们对于Parser终于多少有了点理解——虽然还有一些遗留问题。至少现在已经可以把一门简单的语言编译成bytecode然后解析执行了。不过就像上一章里所说的,有些时候我们还希望能够把程序编译成可执行文件,在这一篇文章里我们就来做这件事情,不用担心,并不是所有的工作都是由compiler来做的。在这里,基于前一章编译出来的ByteCode,我们首先生成汇编代码,然后通过NASM编译成elf object,最后通过ld链接成可执行文件。</p>
<p>首先来熟悉下最简单的汇编程序,比如一个Hello,World可以写成下面的样子:</p>
<div class="highlight"><pre><span></span><code><span class="k">global </span><span class="nv">_start</span>
<span class="k">section </span><span class="nv">.text</span>
<span class="nl">_print:</span>
<span class="nf">push</span> <span class="nb">rdx</span>
<span class="nf">push</span> <span class="nb">rax</span>
<span class="nf">push</span> <span class="nb">rbx</span>
<span class="nf">mov</span> <span class="nb">edx</span><span class="p">,</span> <span class="mi">1</span>
<span class="nf">mov</span> <span class="nb">eax</span><span class="p">,</span> <span class="mi">4</span>
<span class="nf">mov</span> <span class="nb">ebx</span><span class="p">,</span> <span class="mi">1</span>
<span class="nf">int</span> <span class="mh">0x80</span>
<span class="nf">pop</span> <span class="nb">rbx</span>
<span class="nf">pop</span> <span class="nb">rax</span>
<span class="nf">pop</span> <span class="nb">rdx</span>
<span class="nf">ret</span>
<span class="nl">_start:</span>
<span class="nf">mov</span> <span class="nb">ecx</span><span class="p">,</span> <span class="nv">str1</span>
<span class="nf">mov</span> <span class="nb">eax</span><span class="p">,</span> <span class="p">[</span><span class="nb">si</span><span class="nv">ze</span><span class="p">]</span>
<span class="nl">_loop1:</span>
<span class="nf">call</span> <span class="nv">_print</span>
<span class="nf">inc</span> <span class="nb">ecx</span>
<span class="nf">dec</span> <span class="nb">eax</span>
<span class="nf">test</span> <span class="nb">eax</span><span class="p">,</span> <span class="nb">eax</span>
<span class="nf">jne</span> <span class="nv">_loop1</span>
<span class="nf">mov</span> <span class="nb">eax</span><span class="p">,</span> <span class="mi">1</span>
<span class="nf">mov</span> <span class="nb">ebx</span><span class="p">,</span> <span class="mi">0</span>
<span class="nf">int</span> <span class="mh">0x80</span>
<span class="k">section </span><span class="nv">.data</span>
<span class="nf">str1</span> <span class="nv">db</span> <span class="s">'hello,world'</span><span class="p">,</span><span class="mh">0x0A</span><span class="p">,</span><span class="mh">0x00</span>
<span class="nf">size</span> <span class="nv">dd</span> <span class="mi">12</span>
</code></pre></div>
<p>当然这个例子并不是最好的写法,毕竟sys_write提供了一次性输出,不过考虑到跟接下来Brainfuck的衔接,还是吧print单独写成一个函数更加简单。另外需要注意的是虽然上面的代码是64位的,如果需要编译成elf32,那么需要把_print里面的push rxx改成push exx。</p>
<p>仔细观察上面的小例子,结合上一张的brainfuck语言,会发现其实实现一个ByteCode to assembly并不是很困难,只要从前向后解析Bytecode,然后printf(writeln)对应的汇编代码而已, 剩下的工作全部是Assembler和linker的了。作为第一步,首先来定义几个必要的函数:</p>
<div class="highlight"><pre><span></span><code><span class="nl">_read:</span>
<span class="nf">push</span> <span class="nb">rdx</span>
<span class="nf">push</span> <span class="nb">rax</span>
<span class="nf">push</span> <span class="nb">rbx</span>
<span class="nf">push</span> <span class="nb">rbp</span>
<span class="nf">mov</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">ecx</span><span class="p">],</span> <span class="mh">0x30</span>
<span class="nf">xor</span> <span class="nb">rbp</span><span class="p">,</span> <span class="nb">rbp</span>
<span class="nl">_start_read:</span>
<span class="nf">sub</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">ecx</span><span class="p">],</span> <span class="mh">0x30</span>
<span class="nf">imul</span> <span class="nb">ebp</span><span class="p">,</span> <span class="mi">10</span>
<span class="nf">add</span> <span class="nb">ebp</span><span class="p">,</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">ecx</span><span class="p">]</span>
<span class="nf">mov</span> <span class="nb">edx</span><span class="p">,</span> <span class="mi">1</span>
<span class="nf">mov</span> <span class="nb">eax</span><span class="p">,</span> <span class="mi">3</span>
<span class="nf">mov</span> <span class="nb">ebx</span><span class="p">,</span> <span class="mi">0</span>
<span class="nf">int</span> <span class="mh">0x80</span>
<span class="nf">cmp</span> <span class="kt">byte</span> <span class="p">[</span><span class="nb">ecx</span><span class="p">],</span> <span class="mh">0x30</span>
<span class="nf">jl</span> <span class="nv">_end_read</span>
<span class="nf">cmp</span> <span class="kt">byte</span> <span class="p">[</span><span class="nb">ecx</span><span class="p">],</span> <span class="mh">0x39</span>
<span class="nf">jg</span> <span class="nv">_end_read</span>
<span class="nf">cmp</span> <span class="kt">byte</span> <span class="p">[</span><span class="nb">ecx</span><span class="p">],</span><span class="mh">0x0A</span>
<span class="nf">jne</span> <span class="nv">_start_read</span>
<span class="nl">_end_read:</span>
<span class="nf">mov</span> <span class="p">[</span><span class="nb">ecx</span><span class="p">],</span> <span class="nb">ebp</span>
<span class="nf">pop</span> <span class="nb">rbp</span>
<span class="nf">pop</span> <span class="nb">rbx</span>
<span class="nf">pop</span> <span class="nb">rax</span>
<span class="nf">pop</span> <span class="nb">rdx</span>
<span class="nf">ret</span>
<span class="nl">_error:</span>
<span class="nf">mov</span> <span class="nb">rcx</span><span class="p">,</span> <span class="nv">memover</span>
<span class="nf">xor</span> <span class="nb">rbp</span><span class="p">,</span> <span class="nb">rbp</span>
<span class="nf">mov</span> <span class="nb">ebp</span><span class="p">,</span> <span class="kt">dword</span> <span class="p">[</span><span class="nv">memsize</span><span class="p">]</span>
<span class="nl">_loop_error:</span>
<span class="nf">call</span> <span class="nv">_print</span>
<span class="nf">dec</span> <span class="nb">rbp</span>
<span class="nf">inc</span> <span class="nb">rcx</span>
<span class="nf">test</span> <span class="nb">rbp</span><span class="p">,</span> <span class="nb">rbp</span>
<span class="nf">jg</span> <span class="nv">_loop_error</span>
<span class="nf">ret</span>
</code></pre></div>
<p>首先,除了最初的print之外,唯一需要的函数就是read了,它负责将stdin的内容读进目前磁头所指向的磁带。跟原始的brainfuck语言的spec稍有不一样的是:原语言的规格里认为磁带里每一个section都是byte,而为了能够进行更大规模的输入,这里每个section都是一个dword。另外,虽然理想的图灵机考虑有无限长的纸片,显然实际上是不可能的,所以当磁头走出预先定义的范围时,需要报一个内存溢出的错误。 在这里memover可以是任何有意义的错误提示,而memsize则是对应字符串的长度(dd类型)。这些函数都是完全固定的,因此并不需要特别的处理,只需要在compiler执行的时候输出它们就足够了。 剩下的就是简单的用汇编实现每个brainfuck operator的功能:</p>
<div class="highlight"><pre><span></span><code><span class="n">const</span> <span class="n">memSize</span> <span class="p">=</span> <span class="mi">5096</span><span class="p">;</span>
<span class="n">const</span> <span class="n">loopPrefix</span> <span class="p">=</span> <span class="s">"_loop"</span><span class="p">;</span>
<span class="n">const</span> <span class="n">loopSuffix</span> <span class="p">=</span> <span class="s">"_loopend"</span><span class="p">;</span>
<span class="n">string</span> <span class="n">assemble</span><span class="p">(</span><span class="n">ulong</span><span class="p">[]</span> <span class="n">op</span><span class="p">){</span>
<span class="n">string</span> <span class="n">ret</span><span class="p">;</span>
<span class="n">int</span><span class="p">[]</span> <span class="n">labelList</span><span class="p">;</span>
<span class="n">int</span> <span class="n">uniqCount</span> <span class="p">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">ret</span> <span class="p">=</span>
<span class="err">`</span><span class="k">global</span> <span class="n">_start</span>
<span class="n">section</span> <span class="p">.</span><span class="n">text</span>
<span class="n">_print</span><span class="p">:</span>
<span class="n">push</span> <span class="n">rdx</span>
<span class="n">push</span> <span class="n">rax</span>
<span class="n">push</span> <span class="n">rbx</span>
<span class="n">mov</span> <span class="n">edx</span><span class="p">,</span> <span class="mi">1</span>
<span class="n">mov</span> <span class="n">eax</span><span class="p">,</span> <span class="mi">4</span>
<span class="n">mov</span> <span class="n">ebx</span><span class="p">,</span> <span class="mi">1</span>
<span class="n">int</span> <span class="mh">0x80</span>
<span class="n">pop</span> <span class="n">rbx</span>
<span class="n">pop</span> <span class="n">rax</span>
<span class="n">pop</span> <span class="n">rdx</span>
<span class="n">ret</span>
<span class="n">_read</span><span class="p">:</span>
<span class="n">push</span> <span class="n">rdx</span>
<span class="n">push</span> <span class="n">rax</span>
<span class="n">push</span> <span class="n">rbx</span>
<span class="n">push</span> <span class="n">rbp</span>
<span class="n">mov</span> <span class="n">dword</span> <span class="p">[</span><span class="n">ecx</span><span class="p">],</span> <span class="mh">0x30</span>
<span class="n">xor</span> <span class="n">rbp</span><span class="p">,</span> <span class="n">rbp</span>
<span class="n">_start_read</span><span class="p">:</span>
<span class="n">sub</span> <span class="n">dword</span> <span class="p">[</span><span class="n">ecx</span><span class="p">],</span> <span class="mh">0x30</span>
<span class="n">imul</span> <span class="n">ebp</span><span class="p">,</span> <span class="mi">10</span>
<span class="n">add</span> <span class="n">ebp</span><span class="p">,</span> <span class="n">dword</span> <span class="p">[</span><span class="n">ecx</span><span class="p">]</span>
<span class="n">mov</span> <span class="n">edx</span><span class="p">,</span> <span class="mi">1</span>
<span class="n">mov</span> <span class="n">eax</span><span class="p">,</span> <span class="mi">3</span>
<span class="n">mov</span> <span class="n">ebx</span><span class="p">,</span> <span class="mi">0</span>
<span class="n">int</span> <span class="mh">0x80</span>
<span class="n">cmp</span> <span class="n">byte</span> <span class="p">[</span><span class="n">ecx</span><span class="p">],</span> <span class="mh">0x30</span>
<span class="n">jl</span> <span class="n">_end_read</span>
<span class="n">cmp</span> <span class="n">byte</span> <span class="p">[</span><span class="n">ecx</span><span class="p">],</span> <span class="mh">0x39</span>
<span class="n">jg</span> <span class="n">_end_read</span>
<span class="n">cmp</span> <span class="n">byte</span> <span class="p">[</span><span class="n">ecx</span><span class="p">],</span><span class="mh">0x0A</span>
<span class="n">jne</span> <span class="n">_start_read</span>
<span class="n">_end_read</span><span class="p">:</span>
<span class="n">mov</span> <span class="p">[</span><span class="n">ecx</span><span class="p">],</span> <span class="n">ebp</span>
<span class="n">pop</span> <span class="n">rbp</span>
<span class="n">pop</span> <span class="n">rbx</span>
<span class="n">pop</span> <span class="n">rax</span>
<span class="n">pop</span> <span class="n">rdx</span>
<span class="n">ret</span>
<span class="n">_error</span><span class="p">:</span>
<span class="n">mov</span> <span class="n">rcx</span><span class="p">,</span> <span class="n">memover</span>
<span class="n">xor</span> <span class="n">rbp</span><span class="p">,</span> <span class="n">rbp</span>
<span class="n">mov</span> <span class="n">ebp</span><span class="p">,</span> <span class="n">dword</span> <span class="p">[</span><span class="n">memsize</span><span class="p">]</span>
<span class="n">_loop_error</span><span class="p">:</span>
<span class="n">call</span> <span class="n">_print</span>
<span class="n">dec</span> <span class="n">rbp</span>
<span class="n">inc</span> <span class="n">rcx</span>
<span class="n">test</span> <span class="n">rbp</span><span class="p">,</span> <span class="n">rbp</span>
<span class="n">jg</span> <span class="n">_loop_error</span>
<span class="n">ret</span>
<span class="n">_start</span><span class="p">:</span>
<span class="n">xor</span> <span class="n">rax</span><span class="p">,</span> <span class="n">rax</span>
<span class="n">xor</span> <span class="n">rbx</span><span class="p">,</span> <span class="n">rbx</span>
<span class="n">xor</span> <span class="n">rcx</span><span class="p">,</span> <span class="n">rcx</span>
<span class="n">xor</span> <span class="n">rdx</span><span class="p">,</span> <span class="n">rdx</span>
<span class="n">mov</span> <span class="n">rax</span><span class="p">,</span> <span class="n">arr</span>
<span class="n">mov</span> <span class="n">rdx</span><span class="p">,</span> <span class="n">arr</span>
<span class="n">add</span> <span class="n">edx</span><span class="p">,</span> <span class="p">[</span><span class="n">size</span><span class="p">]</span>
<span class="err">`</span><span class="p">;</span>
<span class="n">size_t</span> <span class="n">i</span><span class="p">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">while</span><span class="p">(</span><span class="n">i</span><span class="p"><</span><span class="n">op</span><span class="p">.</span><span class="n">length</span><span class="p">){</span>
<span class="k">switch</span><span class="p">(</span><span class="n">op</span><span class="p">[</span><span class="n">i</span><span class="p">]){</span>
<span class="k">case</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="s">"\tadd rax,4\n\tcmp eax, edx\n\tjge _exit_error\n"</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="s">"\tsub eax,4\n"</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">2</span><span class="p">:</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="s">"\tinc dword [eax]\n"</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">3</span><span class="p">:</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="s">"\tdec dword [eax]\n"</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">4</span><span class="p">:</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="s">"\tmov ecx, eax\n\tcall _print\n"</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">5</span><span class="p">:</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="s">"\tmov ecx, eax\n\tcall _read\n"</span><span class="p">;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">6</span><span class="p">:</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="s">"\tmov ebx, [eax]\n\ttest ebx,ebx\n\tje "</span><span class="p">~</span><span class="n">loopSuffix</span><span class="p">~(</span><span class="n">uniqCount</span><span class="p">).</span><span class="n">to</span><span class="p">!</span><span class="n">string</span><span class="p">~</span><span class="s">"\n"</span><span class="p">;</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="n">loopPrefix</span><span class="p">~</span><span class="n">uniqCount</span><span class="p">.</span><span class="n">to</span><span class="p">!</span><span class="n">string</span><span class="p">~</span><span class="s">":\n"</span><span class="p">;</span>
<span class="n">labelList</span> <span class="p">~=</span> <span class="n">uniqCount</span><span class="p">;</span>
<span class="n">i</span><span class="p">++;</span> <span class="n">uniqCount</span> <span class="p">++;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">7</span><span class="p">:</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="s">"\tmov ebx, [eax]\n\ttest ebx, ebx\n\tjne "</span><span class="p">~</span><span class="n">loopPrefix</span><span class="p">~</span><span class="n">labelList</span><span class="p">[$-</span><span class="mi">1</span><span class="p">].</span><span class="n">to</span><span class="p">!</span><span class="n">string</span><span class="p">~</span><span class="s">"\n"</span><span class="p">;</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="n">loopSuffix</span><span class="p">~</span><span class="n">labelList</span><span class="p">[$-</span><span class="mi">1</span><span class="p">].</span><span class="n">to</span><span class="p">!</span><span class="n">string</span><span class="p">~</span><span class="s">":\n"</span><span class="p">;</span> <span class="n">i</span><span class="p">++;</span>
<span class="n">labelList</span> <span class="p">=</span> <span class="n">labelList</span><span class="p">[</span><span class="mf">0</span><span class="p">..$-</span><span class="mi">1</span><span class="p">];</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">default</span><span class="p">:</span>
<span class="k">throw</span> <span class="n">new</span> <span class="n">Exception</span><span class="p">(</span><span class="s">"Unknown Operator"</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">i</span><span class="p">++;</span>
<span class="p">}</span>
<span class="n">ret</span> <span class="p">~=</span>
<span class="s">"_exit_main:</span>
<span class="s"> mov eax, 1</span>
<span class="s"> mov ebx, 0</span>
<span class="s"> int 0x80</span>
<span class="s">_exit_error:</span>
<span class="s"> call _error</span>
<span class="s"> mov eax, 1</span>
<span class="s"> mov ebx, 1</span>
<span class="s"> int 0x80</span>
<span class="s">"</span><span class="p">;</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="s">"section .data\n arr\ttimes "</span><span class="p">~</span><span class="n">memSize</span><span class="p">.</span><span class="n">to</span><span class="p">!</span><span class="n">string</span><span class="p">~</span><span class="s">" dd 0\n\tsize dd "</span><span class="p">~(</span><span class="n">memSize</span><span class="p">*</span><span class="mi">4</span><span class="p">).</span><span class="n">to</span><span class="p">!</span><span class="n">string</span><span class="p">;</span>
<span class="n">ret</span> <span class="p">~=</span> <span class="s">"\n\tmemover db 'mem overflow',0x0A,0x00\nmemsize dd 13"</span><span class="p">;</span>
<span class="k">return</span> <span class="n">ret</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>这个工作或许很枯燥,不过没有一点难度。 唯一需要注意的就是在处理循环时,会遇到跟SourceCode to ByteCode Compiler同样的问题——当循环嵌套时,如何寻找对应的label? 这里我们用了一个类似堆栈的结构: 每当循环开始,这个循环的地址(label)都会被push进堆栈, 而遇到循环结束时, 只需要简单的从堆栈里pop出来就可以了。最后,在程序结束时,不要忘记使用sys_exit来退出程序,用ebx可以指定exit code。最后,用这个程序来编译下hello world,可以得到:</p>
<div class="highlight"><pre><span></span><code><span class="k">global </span><span class="nv">_start</span>
<span class="k">section </span><span class="nv">.text</span>
<span class="nf">....</span> <span class="nv">PRE</span> <span class="nv">DEFINED</span> <span class="nv">FUNC</span> <span class="nv">....</span>
<span class="nl">_start:</span>
<span class="nf">xor</span> <span class="nb">rax</span><span class="p">,</span> <span class="nb">rax</span>
<span class="nf">xor</span> <span class="nb">rbx</span><span class="p">,</span> <span class="nb">rbx</span>
<span class="nf">xor</span> <span class="nb">rcx</span><span class="p">,</span> <span class="nb">rcx</span>
<span class="nf">xor</span> <span class="nb">rdx</span><span class="p">,</span> <span class="nb">rdx</span>
<span class="nf">mov</span> <span class="nb">rax</span><span class="p">,</span> <span class="nv">arr</span>
<span class="nf">mov</span> <span class="nb">rdx</span><span class="p">,</span> <span class="nv">arr</span>
<span class="nf">add</span> <span class="nb">edx</span><span class="p">,</span> <span class="p">[</span><span class="nb">si</span><span class="nv">ze</span><span class="p">]</span>
<span class="nf">inc</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">inc</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">inc</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">inc</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">inc</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">inc</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">inc</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">inc</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">inc</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">inc</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">mov</span> <span class="nb">ebx</span><span class="p">,</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">test</span> <span class="nb">ebx</span><span class="p">,</span><span class="nb">ebx</span>
<span class="nf">je</span> <span class="nv">_loopend0</span>
<span class="nl">_loop0:</span>
<span class="nf">add</span> <span class="nb">rax</span><span class="p">,</span><span class="mi">4</span>
<span class="nf">....</span> <span class="nv">LOOP</span> <span class="nv">BODY</span> <span class="nv">...</span>
<span class="nf">test</span> <span class="nb">ebx</span><span class="p">,</span> <span class="nb">ebx</span>
<span class="nf">jne</span> <span class="nv">_loop0</span>
<span class="nl">_loopend0:</span>
<span class="nf">add</span> <span class="nb">rax</span><span class="p">,</span><span class="mi">4</span>
<span class="nf">cmp</span> <span class="nb">eax</span><span class="p">,</span> <span class="nb">edx</span>
<span class="nf">jge</span> <span class="nv">_exit_error</span>
<span class="nf">....</span> <span class="nv">PRINT</span> <span class="nv">....</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">mov</span> <span class="nb">ecx</span><span class="p">,</span> <span class="nb">eax</span>
<span class="nf">call</span> <span class="nv">_print</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">dec</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">mov</span> <span class="nb">ecx</span><span class="p">,</span> <span class="nb">eax</span>
<span class="nf">call</span> <span class="nv">_print</span>
<span class="nf">add</span> <span class="nb">rax</span><span class="p">,</span><span class="mi">4</span>
<span class="nf">cmp</span> <span class="nb">eax</span><span class="p">,</span> <span class="nb">edx</span>
<span class="nf">jge</span> <span class="nv">_exit_error</span>
<span class="nf">inc</span> <span class="kt">dword</span> <span class="p">[</span><span class="nb">eax</span><span class="p">]</span>
<span class="nf">mov</span> <span class="nb">ecx</span><span class="p">,</span> <span class="nb">eax</span>
<span class="nf">call</span> <span class="nv">_print</span>
<span class="nf">add</span> <span class="nb">rax</span><span class="p">,</span><span class="mi">4</span>
<span class="nf">cmp</span> <span class="nb">eax</span><span class="p">,</span> <span class="nb">edx</span>
<span class="nf">jge</span> <span class="nv">_exit_error</span>
<span class="nf">mov</span> <span class="nb">ecx</span><span class="p">,</span> <span class="nb">eax</span>
<span class="nf">call</span> <span class="nv">_print</span>
<span class="nl">_exit_main:</span>
<span class="nf">mov</span> <span class="nb">eax</span><span class="p">,</span> <span class="mi">1</span>
<span class="nf">mov</span> <span class="nb">ebx</span><span class="p">,</span> <span class="mi">0</span>
<span class="nf">int</span> <span class="mh">0x80</span>
<span class="nl">_exit_error:</span>
<span class="nf">call</span> <span class="nv">_error</span>
<span class="nf">mov</span> <span class="nb">eax</span><span class="p">,</span> <span class="mi">1</span>
<span class="nf">mov</span> <span class="nb">ebx</span><span class="p">,</span> <span class="mi">1</span>
<span class="nf">int</span> <span class="mh">0x80</span>
<span class="k">section </span><span class="nv">.data</span>
<span class="nf">arr</span> <span class="nv">times</span> <span class="mi">5096</span> <span class="nv">dd</span> <span class="mi">0</span>
<span class="nf">size</span> <span class="nv">dd</span> <span class="mi">20384</span>
<span class="nf">memover</span> <span class="nv">db</span> <span class="s">'mem overflow'</span><span class="p">,</span><span class="mh">0x0A</span><span class="p">,</span><span class="mh">0x00</span>
<span class="nf">memsize</span> <span class="nv">dd</span> <span class="mi">13</span>
</code></pre></div>
<p>因为全部的源代码很长并且没有什么意义, 大部分内容都被省略掉了。不过可惜的是这么直接生成的代码一点也不漂亮,更不高效,充满着让人忍不住去优化的部分,那些内容是Optimizer的工作,在这里我们不会触及。总之,这是一个可运行的代码,剩下的就是编译一下即可:</p>
<div class="highlight"><pre><span></span><code>./compiler test.bf <span class="nb">test</span>
./assem <span class="nb">test</span> test.asm
nasm -f elf64 test.asm
ld test.o -o <span class="nb">test</span>
</code></pre></div>
<p>运行下最终生成的test,屏幕上将会显示出Hello,world。</p>
<p>好了, 到这里为止我们完成了Brainfuck的一个Lexer,Parser,ByteCodeCompiler,VM,以及最简单的到汇编的转换器。虽然省略了相当多的细节,毕竟是第一个可以正常运行的编译器。下面的工作就是回归最初的问题——如何给最初的计算器添加循环/判断和函数?</p>
<p>如果在刚刚完成最初版本的计算器时提出这个问题, 或许没有答案,但放到现在,一切就显得很自然了——跟Brainfuck里的括号运算符一样,只需要把源程序编译成ByteCode即可,甚至可以不编译,只要能够把它Parse成某种可以运行的形式,剩下的循环不过是简单的添加一个指令而已。既然有了明确的答案,那么先来回想下在Brainfuck里的做法: 我们给每一个operator都定义了一个数字,然后按照顺序把它们放进一个数组,当遇到循环时会插入硬编码过的地址…… 对于简单的程序语言似乎没什么问题,但当语言变得复杂,可以想象这个过程会多么容易出错,所有的东西都要直接写在Parser的执行部,并且所有的内容都或多或少有些不一样。因此我们需要一个容易描述程序的结构,它叫做Abstract Syntax Tree,简写为AST,中文称它为抽象语法树。看名字似乎很高端,但实际上所做的事情跟普通的树没什么大差别,下面这幅图来自与AST的<a href="http://en.wikipedia.org/wiki/Abstract_syntax_tree">维基百科</a>.</p>
<p><img alt="AST" src="../images/ast.png"></p>
<p>让我们来手动模拟一下它的执行过程: 首先是Statement节点,它本身没有什么意义,然后深度搜索这棵树,左侧是while节点,于是我们知道这是一个循环结构,而它的循环条件放在树的左侧,向下探索就会遇到compare,这个节点需要比较左右值的内容,继续向下探索,最终我们遇到了的左端的b。按照深度探索的规则,继续向右,会找到第二个值(右值),这两个节点执行之后会返回他们的内容,随后返回到compare节点,有了这左右值之后,我们自然就可以比较结果了,之后的循环体也如发炮制即可。</p>
<p>可以看到AST的想法很简单,将所有程序的elements都化为结点,用树来表示它们的结构,进一步想象一下,如果每一个节点都有一个Execute的function来执行自己对应的命令,那么我们需要做的就是Stat.Execute()而已。并且最大的优势在于,编写节点的Execute时,因为所有内容都是递归执行的,因此不必考虑当前指令的执行者来自与哪里,只需要执行玩自己的命令,然后返回相应的值即可。下面来实现一个简单的AST,因为当前的AST还不是特别复杂,我们可以按照自己的想法来编码。</p>
<p>首先来定义一个基本Class,所有的节点都要从继承这个class。</p>
<div class="highlight"><pre><span></span><code><span class="k">class</span> <span class="n">ASTNode</span><span class="p">{</span>
<span class="n">public</span><span class="p">{</span>
<span class="n">string</span> <span class="n">id</span><span class="p">;</span>
<span class="n">double</span> <span class="n">value</span><span class="p">;</span>
<span class="n">int</span> <span class="n">time</span><span class="p">;</span>
<span class="n">ASTNode</span><span class="p">[]</span> <span class="n">node</span><span class="p">;</span>
<span class="n">void</span> <span class="n">add</span><span class="p">(</span><span class="n">ASTNode</span> <span class="n">n</span><span class="p">){</span> <span class="n">node</span> <span class="p">~=</span> <span class="n">n</span><span class="p">;</span> <span class="p">}</span>
<span class="n">abstract</span> <span class="n">double</span> <span class="n">evaluate</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">ASTNode</span><span class="p">[</span><span class="n">string</span><span class="p">]</span> <span class="n">tab</span><span class="p">;</span>
</code></pre></div>
<p>node用来储存字节点,evaluate用来取得当前节点的返回值,id和value用来储存值和ident,下面会用到。另外考虑到变量的问题,这里把之前定义在parser里的table也移了过来,鉴于目前的程序并没有scope一说,可以直接把所有变量都当成全局变量。下面先来实现常量和变量:</p>
<div class="highlight"><pre><span></span><code><span class="n">final</span> <span class="k">class</span> <span class="n">nul</span><span class="p">:</span> <span class="n">ASTNode</span><span class="p">{</span>
<span class="n">override</span> <span class="n">double</span> <span class="n">evaluate</span><span class="p">(){</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">final</span> <span class="k">class</span> <span class="n">num</span><span class="p">:</span> <span class="n">ASTNode</span><span class="p">{</span>
<span class="n">override</span> <span class="n">double</span> <span class="n">evaluate</span><span class="p">(){</span>
<span class="k">return</span> <span class="n">value</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">final</span> <span class="k">class</span> <span class="n">var</span><span class="p">:</span> <span class="n">ASTNode</span><span class="p">{</span>
<span class="n">override</span> <span class="n">double</span> <span class="n">evaluate</span><span class="p">(){</span>
<span class="k">if</span><span class="p">((</span><span class="n">id</span> <span class="k">in</span> <span class="n">tab</span><span class="p">)</span> <span class="p">!</span><span class="k">is</span> <span class="kc">null</span><span class="p">)</span>
<span class="k">return</span> <span class="n">tab</span><span class="p">[</span><span class="n">id</span><span class="p">].</span><span class="n">evaluate</span><span class="p">;</span>
<span class="k">else</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>常量直接储存在了节点的value里面,但这并不是普通的做法,以后我们会渐渐改掉,而变量则储存在了table里面。还有一个nul用来表示空值,比如什么都没有的statement。其他的节点也都类似:</p>
<div class="highlight"><pre><span></span><code><span class="n">final</span> <span class="k">class</span> <span class="n">plus</span><span class="p">:</span> <span class="n">ASTNode</span><span class="p">{</span>
<span class="n">override</span> <span class="n">double</span> <span class="n">evaluate</span><span class="p">(){</span>
<span class="k">return</span> <span class="n">node</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">evaluate</span><span class="p">+</span><span class="n">node</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="n">evaluate</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">final</span> <span class="k">class</span> <span class="n">pr</span><span class="p">:</span> <span class="n">ASTNode</span><span class="p">{</span>
<span class="n">override</span> <span class="n">double</span> <span class="n">evaluate</span><span class="p">(){</span>
<span class="n">double</span> <span class="n">res</span><span class="p">;</span>
<span class="k">for</span><span class="p">(</span><span class="n">size_t</span> <span class="n">i</span><span class="p">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="p"><</span><span class="n">node</span><span class="p">.</span><span class="n">length</span><span class="p">;</span> <span class="n">i</span><span class="p">++){</span>
<span class="n">res</span> <span class="p">=</span> <span class="n">node</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">evaluate</span><span class="p">;</span>
<span class="n">writefln</span><span class="p">(</span><span class="s">"%.20f"</span><span class="p">,</span> <span class="n">res</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">res</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">final</span> <span class="k">class</span> <span class="n">eq</span><span class="p">:</span> <span class="n">ASTNode</span><span class="p">{</span>
<span class="n">override</span> <span class="n">double</span> <span class="n">evaluate</span><span class="p">(){</span>
<span class="n">auto</span> <span class="n">n</span> <span class="p">=</span> <span class="n">new</span> <span class="n">num</span><span class="p">;</span>
<span class="n">n</span><span class="p">.</span><span class="n">value</span> <span class="p">=</span> <span class="n">node</span><span class="p">[</span><span class="mi">1</span><span class="p">].</span><span class="n">evaluate</span><span class="p">;</span>
<span class="n">tab</span><span class="p">[</span><span class="n">node</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">id</span><span class="p">]</span> <span class="p">=</span> <span class="n">n</span><span class="p">;</span>
<span class="k">return</span> <span class="n">n</span><span class="p">.</span><span class="n">value</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>上面三个分别是加法,输出(print)和变量赋值(定义),这里的实现稍微有些绕弯子。其他的已有的元素都可以仿照上面来逐个定义。对于循环结构,一切也没有什么变化:</p>
<div class="highlight"><pre><span></span><code><span class="n">final</span> <span class="k">class</span> <span class="n">loop</span><span class="p">:</span> <span class="n">ASTNode</span><span class="p">{</span>
<span class="n">override</span> <span class="n">double</span> <span class="n">evaluate</span><span class="p">(){</span>
<span class="n">double</span> <span class="n">res</span><span class="p">;</span>
<span class="k">while</span><span class="p">(</span><span class="n">node</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">evaluate</span> <span class="p">></span> <span class="mi">0</span><span class="p">){</span>
<span class="k">for</span><span class="p">(</span><span class="n">size_t</span> <span class="n">j</span><span class="p">=</span><span class="mi">1</span><span class="p">;</span> <span class="n">j</span><span class="p"><</span><span class="n">node</span><span class="p">.</span><span class="n">length</span><span class="p">;</span> <span class="n">j</span><span class="p">++){</span>
<span class="n">res</span> <span class="p">=</span> <span class="n">node</span><span class="p">[</span><span class="n">j</span><span class="p">].</span><span class="n">evaluate</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">res</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>这个循环结构无非就是不停的执行循环体,然后返回最后一次的值而已(因为之前定义了所有的block都有自己的返回值),而真正的循环是又D语言的循环结构实现的,再向下想象,D语言的结构是又汇编的test,jnz来实现的,汇编的循环最终是由硬件来实现的——也就是说我们并没有“创造”循环,而只是把最初的循环换成了更容易使用的形式而已。</p>
<p>好了,说了一堆,甚至把AST都写好了,还没有给出循环的parser,首先来看一个想象的例子:</p>
<div class="highlight"><pre><span></span><code>c = 50
loop (c=c-1)(
b = 1
a = c
loop (a=a-1)(
b = b*a
)
print b
)
</code></pre></div>
<p>这是一个计算<span class="math">\(n!\)</span>的例子,等于好代表变量赋值,loop代表这循环结构,后边紧跟一个block表示循环条件,随后是循环体。再次提醒,因为计算器只有float类型,所以循环只能用当前值是否大于0来判断,而之前规定了所有block都有返回值,因此在循环条件的部分不需要特殊处理,只需要一个code block即可。如果把实数全部改成自然数的话,相信每个人都能简单的把这个程序跟<span class="math">\(\lambda\)</span>演算等价起来。下面是这个程序的Lexer和Parser的定义:</p>
<div class="highlight"><pre><span></span><code><span class="o">:</span><span class="n">text</span>
<span class="nf">%token</span> <span class="n">LOOP</span> <span class="s">"loop"</span>
<span class="nf">%token</span> <span class="n">SO</span> <span class="s">":="</span>
<span class="p">.................</span>
<span class="nl">program</span><span class="p">:</span> <span class="n">block</span> <span class="o">!</span><span class="p">{</span> <span class="n">result</span> <span class="o">=</span> <span class="n">$1</span><span class="p">.</span><span class="n">n</span><span class="p">;</span> <span class="o">!</span><span class="p">}.</span>
<span class="nl">block</span><span class="p">:</span> <span class="n">equation</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">order</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$1</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">block</span> <span class="n">equation</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">order</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$1</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$2</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="p">.</span>
<span class="nl">prblock</span><span class="p">:</span> <span class="n">LPR</span> <span class="n">block</span> <span class="n">RPR</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="n">$2</span><span class="p">.</span><span class="n">n</span><span class="p">;</span> <span class="o">!</span><span class="p">}</span>
<span class="p">.</span>
<span class="nl">lo</span><span class="p">:</span> <span class="n">LOOP</span> <span class="n">expr</span> <span class="s">"("</span> <span class="n">block</span> <span class="s">")"</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">loop</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$2</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$4</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">LOOP</span> <span class="n">expr</span> <span class="n">TERM</span> <span class="s">"("</span> <span class="n">block</span> <span class="s">")"</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">loop</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$2</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$5</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="p">.</span>
<span class="nl">equation</span><span class="p">:</span><span class="n">TERM</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">nul</span><span class="p">;</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">expr</span> <span class="n">TERM</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="n">$1</span><span class="p">.</span><span class="n">n</span><span class="p">;</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">equation</span> <span class="n">TERM</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="n">$1</span><span class="p">.</span><span class="n">n</span><span class="p">;</span> <span class="o">!</span><span class="p">}</span>
<span class="p">.</span>
<span class="nl">expr</span><span class="p">:</span> <span class="n">lo</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="n">$1</span><span class="p">.</span><span class="n">n</span><span class="p">;</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">ID</span> <span class="n">SO</span> <span class="n">prblock</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">eqf</span><span class="p">;</span> <span class="k">auto</span> <span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">var</span><span class="p">;</span> <span class="n">n</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">$1</span><span class="p">.</span><span class="n">ident</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">n</span><span class="p">);</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$3</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">ID</span> <span class="n">SO</span> <span class="n">expr</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">eqf</span><span class="p">;</span> <span class="k">auto</span> <span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">var</span><span class="p">;</span> <span class="n">n</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">$1</span><span class="p">.</span><span class="n">ident</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">n</span><span class="p">);</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$3</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">ID</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">var</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">$1</span><span class="p">.</span><span class="n">ident</span><span class="p">;</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">ID</span> <span class="s">"="</span> <span class="n">expr</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">eq</span><span class="p">;</span> <span class="k">auto</span> <span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">var</span><span class="p">;</span> <span class="n">n</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">$1</span><span class="p">.</span><span class="n">ident</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">n</span><span class="p">);</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$3</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">ID</span> <span class="s">"="</span> <span class="n">prblock</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">eq</span><span class="p">;</span> <span class="k">auto</span> <span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">var</span><span class="p">;</span> <span class="n">n</span><span class="p">.</span><span class="n">id</span> <span class="o">=</span> <span class="n">$1</span><span class="p">.</span><span class="n">ident</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">n</span><span class="p">);</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$3</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">expr</span> <span class="s">"+"</span> <span class="n">expr</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">plus</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$1</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$3</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">expr</span> <span class="s">"-"</span> <span class="n">expr</span> <span class="o">!</span><span class="p">{</span><span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">min</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$1</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$3</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">expr</span> <span class="s">"*"</span> <span class="n">expr</span> <span class="o">!</span><span class="p">{</span><span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">time</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$1</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$3</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">expr</span> <span class="s">"/"</span> <span class="n">expr</span> <span class="o">!</span><span class="p">{</span><span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">div</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$1</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$3</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="s">"("</span> <span class="n">expr</span> <span class="s">")"</span> <span class="o">!</span><span class="p">{</span><span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="n">$2</span><span class="p">.</span><span class="n">n</span><span class="p">;</span><span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="s">"-"</span> <span class="n">expr</span> <span class="nf">%prec</span> <span class="n">UMINUS</span> <span class="o">!</span><span class="p">{</span><span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">um</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$2</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="s">"+"</span> <span class="n">expr</span> <span class="nf">%prec</span> <span class="n">UPLUS</span> <span class="o">!</span><span class="p">{</span><span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="n">$2</span><span class="p">.</span><span class="n">n</span><span class="p">;</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">NUMBER</span> <span class="o">!</span><span class="p">{</span><span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">num</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">$1</span><span class="p">.</span><span class="n">value</span><span class="p">;</span> <span class="o">!</span><span class="p">}</span>
<span class="o">|</span> <span class="n">PR</span> <span class="n">expr</span> <span class="o">!</span><span class="p">{</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span> <span class="o">=</span> <span class="k">new</span> <span class="n">pr</span><span class="p">;</span> <span class="n">$$</span><span class="p">.</span><span class="n">n</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">$2</span><span class="p">.</span><span class="n">n</span><span class="p">);</span> <span class="o">!</span><span class="p">}</span>
<span class="p">.</span>
</code></pre></div>
<p>一切看起来都很简单,我们所做的就似乎把之前的硬编码换成树操作,剩下最大的挑战就是添加循环之后避免Shift/Reduce Conflict而已。在整个定义没有歧义之后,这个“语言”就算完成了。需要注意的是这里我们返回的是ASTNode型的树根,而不再是之前的float型的执行结果了。这样在Evaluator里,我们可以执行parse(text).evaluate来获得结果。</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span> <span class="n">std</span><span class="p">.</span><span class="n">stdio</span><span class="p">,</span> <span class="n">std</span><span class="p">.</span><span class="n">file</span><span class="p">,</span> <span class="n">std</span><span class="p">.</span><span class="n">string</span><span class="p">;</span>
<span class="k">import</span> <span class="n">parser2</span><span class="p">,</span> <span class="n">ast</span><span class="p">;</span>
<span class="n">int</span> <span class="n">main</span><span class="p">(</span><span class="n">string</span><span class="p">[]</span> <span class="n">args</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">args</span><span class="p">.</span><span class="n">length</span> <span class="p">==</span> <span class="mi">2</span><span class="p">){</span>
<span class="n">writefln</span><span class="p">(</span><span class="s">"Program Exited: %f"</span><span class="p">,</span> <span class="n">evaluate</span><span class="p">(</span><span class="n">readText</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">])).</span><span class="n">evaluate</span><span class="p">);</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>试着执行之前我们想象的那个程序,可以得到:</p>
<div class="highlight"><pre><span></span><code>INSERT$ ./evaluator2 fact
12413915592536068246584490201464214099653701243174007003414528.00000000000000000000
258623241511168266876471800775489592695198290419683860414464.00000000000000000000
5502622159812089153567237889924947737578015493424280502272.00000000000000000000
119622220865480210353064206097531035159161141491744636928.00000000000000000000
2658271574788449549981313790911632279366972557834059776.00000000000000000000
60415263063373844708264795564905929160694072661245952.00000000000000000000
1405006117752880121086634743819753058130737680613376.00000000000000000000
33452526613163797571690755229534229392170908909568.00000000000000000000
815915283247898008314102179727920573516005507072.00000000000000000000
20397882081197446658430873854155690147131162624.00000000000000000000
523022617466601037913697377988137380787257344.00000000000000000000
13763753091226345578872114833606270345281536.00000000000000000000
371993326789901332234925207831002568720384.00000000000000000000
10333147966386144222209170348167175077888.00000000000000000000
295232799039604081776217808048838672384.00000000000000000000
8683317618811889480490536047552233472.00000000000000000000
263130836933693554659840465146216448.00000000000000000000
8222838654177923583120014535819264.00000000000000000000
265252859812191104246398737973248.00000000000000000000
8841761993739700772720181510144.00000000000000000000
304888344611713801550158626816.00000000000000000000
10888869450418351940239884288.00000000000000000000
403291461126605719042260992.00000000000000000000
15511210043330983907819520.00000000000000000000
620448401733239409999872.00000000000000000000
25852016738884978212864.00000000000000000000
1124000727777607680000.00000000000000000000
51090942171709440000.00000000000000000000
2432902008176640000.00000000000000000000
121645100408832000.00000000000000000000
6402373705728000.00000000000000000000
355687428096000.00000000000000000000
20922789888000.00000000000000000000
1307674368000.00000000000000000000
87178291200.00000000000000000000
6227020800.00000000000000000000
479001600.00000000000000000000
39916800.00000000000000000000
3628800.00000000000000000000
362880.00000000000000000000
40320.00000000000000000000
5040.00000000000000000000
720.00000000000000000000
120.00000000000000000000
24.00000000000000000000
6.00000000000000000000
2.00000000000000000000
1.00000000000000000000
1.00000000000000000000
Program Exited: 1.000000
</code></pre></div>
<p>可以看到这个程序已经可以正确执行了。虽然这个程序的功能依然很弱,对于简单的运算来说已经足够了,比如计算<span class="math">\(\pi\)</span>:</p>
<div class="highlight"><pre><span></span><code>len = 20000000
pi = 1
a = 1
c = 1
s = -1
loop len-(a=a+2) (
pi = pi + s*1/a
s = -1 * s
)
print pi*4
</code></pre></div>
<p>执行之后可以获得如下结果:</p>
<div class="highlight"><pre><span></span><code>INSERT$ ./evaluator2 pi
3.14159255358979150330
Program Exited: 3.141593
</code></pre></div>
<p>好了,到此为止,最初开始是作为练习提出来的问题终于大部分得到了解决,但并不完美,实际执行之后发现,完全实时的evaluate非常慢,上面计算<span class="math">\(\pi\)</span>的例子就足足花了15秒。 因此在下一章,我们会先把进一步扩充语言放到一边,转而引入LLVM(Low Level Virtual Machine),让目前的计算器获得巨大的性能提升。</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Build Compiler with LALR(1) Parser Generator(II)2014-06-03T00:00:00+09:002014-06-03T00:00:00+09:00Horsaltag:www.horsal.dev,2014-06-03:/build-compiler-with-lalr1-parser-generatorii.html<p><a href="https://www.horsal.dev/build-compiler-with-lalr1-parser-generatori.html">在上一篇文章里</a>, 我们通过最简单的dunnart自带例子构建了一个计算器,并且对Parser Generator的结构做了说明,不过就像结尾时所说的一样,那个计算器只能进行简单的四则运算,无法保存输出,没有变量,并不怎么让人兴奋。因此,在这篇文章的前半段,我们会对之前的代码进行小小的修改,让它支持变量,然后会尝试让他支持循环——但这个事情并不是那么容易,因此在这次的后半段,会有另外一个小例子来了解下循环的构造。</p>
<p>首先,从上次已经完成的计算器程序开始,我们希望添加变量之后能够实现下面的功能:</p>
<div class="highlight"><pre><span></span><code>a = 1
b = 2
a+b
(Output -> 3)
</code></pre></div>
<p>简单来说就是任何第一次使用并且以字母或者下划线开头的字符串都是变量, 并且只能通过赋值的方式来定义。 现阶段为了例子简单,我们只考虑变量是浮点数的情况。在动手折腾lexer和parser之前,先来定义一个最简单的储存器,它的主要功能是指定一个String之后,能够储存或者读取相应的值。这并不是一个很难的事情,你可以用喜欢的任何Container来实现它,比如HashedList,LinkedList,甚至是RandomAccess的Stack。在这里我选用了最简单的Associative Array(内部实现也是HashedList):</p>
<div class="highlight"><pre><span></span><code><span class="n">double</span><span class="p">[</span><span class="n">string</span><span class="p">]</span> <span class="n">list</span><span class="p">;</span>
</code></pre></div>
<p>在有了这个储存设备之后,就可以真正的着手来添加功能了,首先我们需要定义Identifier的Token …</p><p><a href="https://www.horsal.dev/build-compiler-with-lalr1-parser-generatori.html">在上一篇文章里</a>, 我们通过最简单的dunnart自带例子构建了一个计算器,并且对Parser Generator的结构做了说明,不过就像结尾时所说的一样,那个计算器只能进行简单的四则运算,无法保存输出,没有变量,并不怎么让人兴奋。因此,在这篇文章的前半段,我们会对之前的代码进行小小的修改,让它支持变量,然后会尝试让他支持循环——但这个事情并不是那么容易,因此在这次的后半段,会有另外一个小例子来了解下循环的构造。</p>
<p>首先,从上次已经完成的计算器程序开始,我们希望添加变量之后能够实现下面的功能:</p>
<div class="highlight"><pre><span></span><code>a = 1
b = 2
a+b
(Output -> 3)
</code></pre></div>
<p>简单来说就是任何第一次使用并且以字母或者下划线开头的字符串都是变量, 并且只能通过赋值的方式来定义。 现阶段为了例子简单,我们只考虑变量是浮点数的情况。在动手折腾lexer和parser之前,先来定义一个最简单的储存器,它的主要功能是指定一个String之后,能够储存或者读取相应的值。这并不是一个很难的事情,你可以用喜欢的任何Container来实现它,比如HashedList,LinkedList,甚至是RandomAccess的Stack。在这里我选用了最简单的Associative Array(内部实现也是HashedList):</p>
<div class="highlight"><pre><span></span><code><span class="n">double</span><span class="p">[</span><span class="n">string</span><span class="p">]</span> <span class="n">list</span><span class="p">;</span>
</code></pre></div>
<p>在有了这个储存设备之后,就可以真正的着手来添加功能了,首先我们需要定义Identifier的Token:</p>
<div class="highlight"><pre><span></span><code>%field double value
%field string ident
...
%token <ident> ID ([A-Za-z_][A-Za-z_0-9]*)
%token TERM ([;\n])
...
%left "+" "-"
%right "="
%nonassoc TERM
</code></pre></div>
<p>可以看到这里的定义是这样的: 第一个字符只能是大小写字母或者下划线, 后面可以包含数字。这个定义基本上没有跟我们熟悉的变量定义冲突。这里还顺便定义了一个终止符,它可以是分号或者一行的结尾,后面它会用来区分不同的表达式,从而让parser可以一次性执行多个语句。</p>
<p>在定义了Token之后,下一步就是完善Parser,仔细考虑下就会发现一共只有两种情况: 赋值(定义)和使用。 在赋值时,左边只可能是变量,而当变量出现在右边时,他应该跟普通的表达式有着完全相同的作用。把这上面的文字用BNF写出来:</p>
<div class="highlight"><pre><span></span><code>expr: ID !{ $$.value = (($1.ident in list) is null)?0:list[$1.ident]; !}
| ID "=" expr !{ list[$1.ident] = $3.value; $$.value = $3.value; $1.value = $3.value; !}
| expr "+" expr !{$$.value = $1.value + $3.value;!}
....
</code></pre></div>
<p>其他部分则没有任何变化。 让我们再稍微仔细的看下上面两行的定义: 首先如果identifier出现在右边,那么它应当是一个Expression,此时它的值是Variable Table里储存的值, 如果不存在,那么默认它为0。另外,如果ID出现在了左侧,那么只可能是对它赋值,这就是第二行里所定义的东西, 不过需要注意的是,在这里赋值语句也是一个表达式,它会返回等号右边的值,这种定义能够允许诸如“(a=3)+b*4”这种写法,另外在这里虽然有对$1.vaule赋值的语句,但它没有任何意义,因为即使遇到的Identifier,也并不会直接使用ID的value,Reduce时总是会从表里查询实际的值的。</p>
<p>到这里对变量的处理就完成了,不过为了能够一次性执行多个语句, 还需要让lexer里定义的TERM能够发挥作用。在这里然我们用写简单粗暴的方法来实现它:</p>
<div class="highlight"><pre><span></span><code>program: equation.
equation:TERM !{ result = 0; !}
| expr TERM !{ result = $1.value; !}
| equation expr TERM !{ result = $2.value; !}
| equation TERM !{ result = $2.value; !}
.
</code></pre></div>
<p>可以看到在这里定义的program不再是一个expr, 而是一个equation(实际上更“准确”的名字应该是block),而equation则是分号,expr与分号的组合。在实现了这些之后,至少这个计算器就能用不同的语句来操作变量和进行简单的计算了。不过是不是觉得还少些什么? 对,这个计算器没有输出,直到现在为止都是result变量在做这个工作——但有时候仅仅输出一个变量是不够的,比如我们执行如下语句:</p>
<div class="highlight"><pre><span></span><code>a = 3
b = c = 4
d = a*3+c=b*4+c*5
</code></pre></div>
<p>这时候parser将默认返回d的值,但如果同时希望取得c的值(这将有助于调试operator的优先级),那么就显得不够用了,因此我们还需要一个输出语句,这里定义为“print”。在lexer定义的最后追加这个Token:</p>
<div class="highlight"><pre><span></span><code>%token PR "print"
%skip ([ \t\r]+)
%skip (--[^\n]*\n)
.....
</code></pre></div>
<p>这里我们顺便给计算器追加了注释语句, 任何以"--"开头的语句是一个整行注释,直接将符合条件的正则表达式放进skip语句里即可。而实现输出是非常简单的,只需要在expr的定义最末尾追加一句输出:</p>
<div class="highlight"><pre><span></span><code>expr: ....
| PR expr !{ $$.value = $2.value; writeln($2.value); !}
.
</code></pre></div>
<p>这样当程序遇到类似"print a"的语句时,就会输出它的值了。</p>
<p>到这里,给计算器追加变量的任务就完成了,试着执行一个简单的脚本:</p>
<div class="highlight"><pre><span></span><code>INSERT$ cat test_input
--toaehustohae test file
a = 10
print 10+a
b = 20
print b+a
print (c=10)+b+a
print (d=10)+d*d
(f=e=d+1)
print f
print e
0
INSERT$ ./evaluator test_input
20
30
40
110
11
11
0
</code></pre></div>
<p>可以看到变量已经可以正常使用了。 不过先不要急着高兴,这些元素依然不足以让这个计算器变成一门程序语言,因为至少还需要一个循环才能让这个计算器变得有意义。不过遗憾的是到目前为止,我们接触到的内容还不足以实现一个loop循环——想想看,现在所有的语句都是在触发规则时就已经被执行了,因此每一个节点里只需要储存执行的结果,如果用现在的思想来实现loop,它看起来就像是样:</p>
<div class="highlight"><pre><span></span><code>loop: "while" expr "then" "begin" equation "end" TERM !{ while($2.value) equation; $$.value = $5.value; !}.
</code></pre></div>
<p>我们并没有办法去实际执行expr和equation,而是只能获得他们的结果,自然循环的说法无从谈起。 这个问题的完整解决办法需要在下一章才能实现,在这里,作为导入,先把计算器问题放到一边,我们来实现一个图灵完备的小程序语言——<a href="http://en.wikipedia.org/wiki/Brainfuck">Brainfuck</a>。</p>
<p>Brainfuck是一个很精简的语言,整个语言也不过使用了8个Operator,并且既然是图灵完备那么它就一定在某种意义上实现了递归或者循环,仔细阅读它的Spec就会发现"["和"]"是一个循环结构。现在开始,目标不再是给计算器添加循环,而是实现一个Brainfuck的lexer,parser,并且把它编译成能够在虚拟机里运行的二进制代码。</p>
<p>首先让来人力模拟一下这个程序的执行过程,比如下面的一段小程序:</p>
<div class="highlight"><pre><span></span><code>+[,.]
</code></pre></div>
<p>那么该怎么执行呢? 显然的,首先把当前位置的内容加1, 然后执行循环里面的内容,对于任何输入都原封不动的输出它, 一直重复到输入值为0………… 看起来似乎很简单, 但是等等, 这里有一个被忽略了的问题: 在执行到"]"时,我们如何才能知道对应的"["在哪里? 对于手工执行, 这个答案看起来也不难——我们向前查找, 一直到第一次遇到"[",那么它的下一条指令就是对应的循环体。虽然不是怎么高效,但容易理解并且易于实现。在正式的开始实现parser时,不妨先来实现一个具有上述功能的解析器。 用块内存来表示磁带, 用一个指针(或者Array Index)来表示磁头的位置,把所有brainfuck语言翻译成我们使用的高级语言,可以得到:</p>
<div class="highlight"><pre><span></span><code><span class="p">%{</span>
<span class="k">import</span> <span class="n">std</span><span class="p">.</span><span class="n">stdio</span><span class="p">;</span>
<span class="k">import</span> <span class="n">std</span><span class="p">.</span><span class="n">conv</span><span class="p">:</span> <span class="n">to</span><span class="p">;</span>
<span class="n">int</span><span class="p">[]</span> <span class="n">mem</span> <span class="p">;</span>
<span class="n">int</span> <span class="n">result</span><span class="p">;</span>
<span class="n">void</span> <span class="n">execute</span><span class="p">(</span><span class="n">int</span><span class="p">[]</span> <span class="n">op</span><span class="p">){</span>
<span class="n">int</span> <span class="n">ap</span> <span class="p">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">mem</span><span class="p">.</span><span class="n">length</span> <span class="p">=</span> <span class="mi">512</span><span class="p">;</span> <span class="n">mem</span><span class="p">[]</span> <span class="p">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">while</span><span class="p">(</span><span class="n">ap</span><span class="p"><</span><span class="n">op</span><span class="p">.</span><span class="n">length</span><span class="p">){</span>
<span class="n">int</span> <span class="n">loopLayer</span> <span class="p">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">switch</span><span class="p">(</span><span class="n">op</span><span class="p">[</span><span class="n">ap</span><span class="p">]){</span>
<span class="k">case</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">result</span><span class="p">++;</span> <span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">result</span><span class="p">--;</span> <span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">2</span><span class="p">:</span>
<span class="n">mem</span><span class="p">[</span><span class="n">result</span><span class="p">]++;</span> <span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">3</span><span class="p">:</span>
<span class="k">if</span><span class="p">(</span><span class="n">mem</span><span class="p">[</span><span class="n">result</span><span class="p">]></span><span class="mi">0</span><span class="p">)</span>
<span class="n">mem</span><span class="p">[</span><span class="n">result</span><span class="p">]--;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">4</span><span class="p">:</span>
<span class="n">write</span><span class="p">(</span><span class="n">mem</span><span class="p">[</span><span class="n">result</span><span class="p">].</span><span class="n">to</span><span class="p">!</span><span class="n">dchar</span><span class="p">);</span> <span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">5</span><span class="p">:</span>
<span class="n">write</span><span class="p">(</span><span class="s">"Waiting for input: "</span><span class="p">);</span>
<span class="n">readf</span><span class="p">(</span><span class="s">"%d\n"</span><span class="p">,</span> <span class="p">&</span><span class="n">mem</span><span class="p">[</span><span class="n">result</span><span class="p">]);</span> <span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">6</span><span class="p">:</span>
<span class="k">if</span><span class="p">(</span><span class="n">mem</span><span class="p">[</span><span class="n">result</span><span class="p">]==</span><span class="mi">0</span><span class="p">)</span>
<span class="k">while</span><span class="p">(</span><span class="n">op</span><span class="p">[++</span><span class="n">ap</span><span class="p">]!=</span><span class="mi">7</span> <span class="p">||</span> <span class="n">loopLayer</span> <span class="p">></span> <span class="mi">0</span><span class="p">){</span>
<span class="k">if</span><span class="p">(</span><span class="n">op</span><span class="p">[</span><span class="n">ap</span><span class="p">]</span> <span class="p">==</span> <span class="mi">6</span><span class="p">)</span> <span class="n">loopLayer</span><span class="p">++;</span>
<span class="k">if</span><span class="p">(</span><span class="n">op</span><span class="p">[</span><span class="n">ap</span><span class="p">]</span> <span class="p">==</span> <span class="mi">7</span> <span class="p">&&</span> <span class="n">loopLayer</span> <span class="p">></span> <span class="mi">0</span><span class="p">)</span> <span class="n">loopLayer</span><span class="p">--;</span>
<span class="p">}</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">7</span><span class="p">:</span>
<span class="k">if</span><span class="p">(</span><span class="n">mem</span><span class="p">[</span><span class="n">result</span><span class="p">]!=</span><span class="mi">0</span><span class="p">){</span>
<span class="k">while</span><span class="p">(</span><span class="n">op</span><span class="p">[--</span><span class="n">ap</span><span class="p">]!=</span><span class="mi">6</span> <span class="p">||</span> <span class="n">loopLayer</span> <span class="p">></span> <span class="mi">0</span><span class="p">){</span>
<span class="k">if</span><span class="p">(</span><span class="n">op</span><span class="p">[</span><span class="n">ap</span><span class="p">]</span> <span class="p">==</span> <span class="mi">7</span><span class="p">)</span> <span class="n">loopLayer</span><span class="p">++;</span>
<span class="k">if</span><span class="p">(</span><span class="n">op</span><span class="p">[</span><span class="n">ap</span><span class="p">]</span> <span class="p">==</span> <span class="mi">6</span> <span class="p">&&</span> <span class="n">loopLayer</span> <span class="p">></span> <span class="mi">0</span><span class="p">)</span> <span class="n">loopLayer</span><span class="p">--;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="n">loopLayer</span><span class="p">--;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">default</span><span class="p">:</span>
<span class="k">throw</span> <span class="n">new</span> <span class="n">Exception</span><span class="p">(</span><span class="s">"Unknown Operator"</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">ap</span><span class="p">++;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">%}</span>
<span class="p">%{</span>
<span class="n">double</span> <span class="n">evaluate</span><span class="p">(</span><span class="n">string</span> <span class="n">text</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">int</span><span class="p">[]</span> <span class="n">op</span><span class="p">;</span>
<span class="k">if</span><span class="p">(</span><span class="n">text</span><span class="p">.</span><span class="n">length</span> <span class="p">==</span> <span class="mi">0</span><span class="p">)</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">%}</span>
<span class="p">%</span><span class="n">field</span> <span class="n">int</span> <span class="n">value</span>
<span class="p">%</span><span class="n">token</span> <span class="n">PP</span> <span class="s">">"</span>
<span class="p">%</span><span class="n">token</span> <span class="n">PM</span> <span class="s">"<"</span>
<span class="p">%</span><span class="n">token</span> <span class="n">MP</span> <span class="s">"+"</span>
<span class="p">%</span><span class="n">token</span> <span class="n">MM</span> <span class="s">"-"</span>
<span class="p">%</span><span class="n">token</span> <span class="n">PR</span> <span class="s">"."</span>
<span class="p">%</span><span class="n">token</span> <span class="n">RD</span> <span class="s">","</span>
<span class="p">%</span><span class="n">token</span> <span class="n">JF</span> <span class="s">"["</span>
<span class="p">%</span><span class="n">token</span> <span class="n">JB</span> <span class="s">"]"</span>
<span class="p">%</span><span class="n">skip</span> <span class="p">([</span> <span class="p">\</span><span class="n">n</span><span class="p">\</span><span class="n">t</span><span class="p">\</span><span class="n">r</span><span class="p">]+)</span>
<span class="p">%%</span>
<span class="n">program</span><span class="p">:</span> <span class="n">expr</span> <span class="p">!{</span> <span class="n">execute</span><span class="p">(</span><span class="n">op</span><span class="p">);</span> <span class="p">!}.</span>
<span class="n">expr</span><span class="p">:</span> <span class="n">atom</span>
<span class="p">|</span> <span class="n">expr</span> <span class="n">atom</span>
<span class="p">.</span>
<span class="n">atom</span><span class="p">:</span> <span class="n">PP</span> <span class="p">!{</span> <span class="n">op</span><span class="p">~=</span><span class="mi">0</span><span class="p">;</span> <span class="p">!}</span>
<span class="p">|</span> <span class="n">PM</span> <span class="p">!{</span> <span class="n">op</span><span class="p">~=</span><span class="mi">1</span><span class="p">;</span> <span class="p">!}</span>
<span class="p">|</span> <span class="n">MP</span> <span class="p">!{</span> <span class="n">op</span><span class="p">~=</span><span class="mi">2</span><span class="p">;</span> <span class="p">!}</span>
<span class="p">|</span> <span class="n">MM</span> <span class="p">!{</span> <span class="n">op</span><span class="p">~=</span><span class="mi">3</span><span class="p">;</span> <span class="p">!}</span>
<span class="p">|</span> <span class="n">PR</span> <span class="p">!{</span> <span class="n">op</span><span class="p">~=</span><span class="mi">4</span><span class="p">;</span> <span class="p">!}</span>
<span class="p">|</span> <span class="n">RD</span> <span class="p">!{</span> <span class="n">op</span><span class="p">~=</span><span class="mi">5</span><span class="p">;</span> <span class="p">!}</span>
<span class="p">|</span> <span class="n">JF</span> <span class="p">!{</span> <span class="n">op</span><span class="p">~=</span><span class="mi">6</span><span class="p">;</span> <span class="p">!}</span>
<span class="p">|</span> <span class="n">JB</span> <span class="p">!{</span> <span class="n">op</span><span class="p">~=</span><span class="mi">7</span><span class="p">;</span> <span class="p">!}</span>
<span class="p">.</span>
<span class="p">%{</span>
<span class="n">dd_parse_text</span><span class="p">(</span><span class="n">text</span><span class="p">);</span>
<span class="n">writeln</span><span class="p">();</span>
<span class="k">return</span> <span class="n">result</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">%}</span>
</code></pre></div>
<p>上面的代码描述了一个最简单的解析器, 为了让这个部分跟这个系列的主题——LALR(1) Parser Generator——有点关联,源代码先通过parser解析成了数字,但直接用字符作为输入进行相同的操作也并没有什么不同。注意到这里的Case6和case7,跟之前描述的一样, 每当需要跳转时,我们就会去搜索需要的位置。而对于其他Operator,需要做的就是简单的加减Tape上的记录和移动磁头,这根最初的图灵机是几乎一模一样的。</p>
<div class="highlight"><pre><span></span><code><span class="k">case</span> <span class="mi">6</span><span class="p">:</span>
<span class="k">if</span><span class="p">(</span><span class="n">mem</span><span class="p">[</span><span class="n">result</span><span class="p">]==</span><span class="mi">0</span><span class="p">)</span>
<span class="k">while</span><span class="p">(</span><span class="n">op</span><span class="p">[++</span><span class="n">ap</span><span class="p">]!=</span><span class="mi">7</span> <span class="p">||</span> <span class="n">loopLayer</span> <span class="p">></span> <span class="mi">0</span><span class="p">){</span>
<span class="k">if</span><span class="p">(</span><span class="n">op</span><span class="p">[</span><span class="n">ap</span><span class="p">]</span> <span class="p">==</span> <span class="mi">6</span><span class="p">)</span> <span class="n">loopLayer</span><span class="p">++;</span>
<span class="k">if</span><span class="p">(</span><span class="n">op</span><span class="p">[</span><span class="n">ap</span><span class="p">]</span> <span class="p">==</span> <span class="mi">7</span> <span class="p">&&</span> <span class="n">loopLayer</span> <span class="p">></span> <span class="mi">0</span><span class="p">)</span> <span class="n">loopLayer</span><span class="p">--;</span>
<span class="p">}</span>
</code></pre></div>
<p>不过还有一个小细节需要注意: 考虑循环嵌套的情形, 比如类似"[>[,.]]"的程序段,如果没有任何约束直接搜索最近的括号,那么对第二个"]"向前搜索时,实际上返回的结果是内层循环的开始——这显然是不正确的。为了解决这个问题,不能仅仅进行简单的搜索, 而是引入一个Loop Layer的概念,它标志这目前所在的循环是第几次嵌套,这样,只查找相同layer的循环,就能够实现括号的配对了。而Loop Layer的维护也非常简单——根据遇到的Operator进行加减即可。</p>
<p>不过在这个程序里, 你会发现我们并没有维护全局的Loop Layer,每次这个变量都会被初始化为0,这是因为这里我们只需要相对Layer就足够了——跟当前括号配对的括号总是0,除此之外别的信息在现阶段没有太大意义。</p>
<p>虽然这个程序看起来很简单,也很不优雅,但他确实具有了执行brainfuck程序的能力, 编译他, 然后执行HelloWorld程序的话:</p>
<div class="highlight"><pre><span></span><code>INSERT$ cat hello.bf
++++++++++[>+++++++>++++++++++>+++>+<<<<-]
>++.>+.+++++++..+++.>++.<<+++++++++++++++.
>.+++.------.--------.>+.>.
INSERT$ ./evaluator hello.bf
Hello World!
4
</code></pre></div>
<p>我们得到了最初的结果,当然,这只是第一步,因为没有人希望自己的程序在每次跳转时都要做一次搜索,那么我们有必要寻找更好的解决办法。一个简单的想法就是把跳转的目标地址放在Operator Code的后面,每当需要跳转,只需要跳转到后面一个Op所指向的位置就可以了。这看起来似乎是个好主意,免去了搜索,并且看起来很容易实现。在这里,为了避免二次编译的问题,我们不会用全局跳转地址,而是采用类似Intel 8088汇编的方式,所有的短跳转都用相对位置来表示。 同时,从现在开始在parser里导入循环的定义:</p>
<div class="highlight"><pre><span></span><code>%{
alias OPList = ulong[];
%}
%{
OPList evaluate(string text)
{
if(text.length == 0) return [];
OPList machineCode;
%}
%field OPList value
%token PP ">"
%token PM "<"
%token MP "+"
%token MM "-"
%token PR "."
%token RD ","
%token JF "["
%token JB "]"
%skip ([ \n\t\r]+)
%%
program: expr !{ machineCode = $1.value; !}.
expr: atom !{ $$.value = $1.value; !}
| loop !{ $$.value = $1.value; !}
| expr loop !{ $$.value = $1.value ~ $2.value; !}
| expr atom !{ $$.value = $1.value ~ $2.value; !}
.
loop: JF expr JB !{ $$.value = [cast(ulong)6]~[$2.value.length]~$2.value~[cast(ulong)7]~[$2.value.length]; !}
.
atom: PP !{ $$.value~=[0]; !}
| PM !{ $$.value~=[1]; !}
| MP !{ $$.value~=[2]; !}
| MM !{ $$.value~=[3]; !}
| PR !{ $$.value~=[4]; !}
| RD !{ $$.value~=[5]; !}
.
%{
dd_parse_text(text);
return machineCode;
}
%}
</code></pre></div>
<p>循环被正式的作为一种结构写入了Parser,而不是完全在执行器(VM)里实现了。虽然写出他的BNF定义不难,不过注意,在之前的小程序里,所有的Operator Code都是直接追加到结果里面的,而循环并不能从前向后的被解析,因为首先循环体需要被Reduce成一个expr才行。如果没一条指令都直接把结果写进返回值里,那么最后得到的将是一堆乱码。</p>
<p>为了解决这个问题,在这里,每一个Node都有一个数组,在自下而上的Parsing过程中,首先每个小元素都会存进数组中, 然后通过合并的方式产生更大的数组, 最后把所有数组片都集合起来, 就是我们的程序。这种做法大大简化了我们的工作量,因为在遇到循环时,"[]"内部的expression已经被解析完毕了,也就是说已经知道循环体的长度,剩下的就是简单的将跳转相对地址存到对应的位置即可。同时也不必担心多层循环——因为每一个"]"总是和最近的"["配对,这个工作会有Parser Generator帮我们完成。最后这个parser的返回值也从double改成了生成的Operator Code,只要把它保存到本地文件,就可以说“编译”成功了。</p>
<p>在有了这个二进制Operator Code之后, 我们的工作就简单多了,之前的解析器绝大多数地方都不需要改变, 需要改变的只有跳转部分:</p>
<div class="highlight"><pre><span></span><code> <span class="k">case</span> <span class="mi">6</span><span class="p">:</span>
<span class="k">if</span><span class="p">(</span><span class="n">mem</span><span class="p">[</span><span class="n">result</span><span class="p">]==</span><span class="mi">0</span><span class="p">)</span>
<span class="n">ap</span> <span class="p">+=</span> <span class="n">op</span><span class="p">[</span><span class="n">ap</span><span class="p">+</span><span class="mi">1</span><span class="p">]+</span><span class="mi">3</span><span class="p">;</span>
<span class="k">else</span> <span class="n">ap</span><span class="p">++;</span>
<span class="k">break</span><span class="p">;</span>
<span class="k">case</span> <span class="mi">7</span><span class="p">:</span>
<span class="k">if</span><span class="p">(</span><span class="n">mem</span><span class="p">[</span><span class="n">result</span><span class="p">]!=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">ap</span> <span class="p">-=</span> <span class="n">op</span><span class="p">[</span><span class="n">ap</span><span class="p">+</span><span class="mi">1</span><span class="p">]+</span><span class="mi">1</span><span class="p">;</span>
<span class="k">else</span> <span class="n">ap</span><span class="p">++;</span>
<span class="k">break</span><span class="p">;</span>
</code></pre></div>
<p>这里可以通过直接设定ap(Array Index)的值来实现循环了。不过相应的,因为记录了多余的内容(跳转地址),因此在执行的时候需要跳过它们。下面是一个求<span class="math">\(n!\)</span>的程序,让我们来测试一下:</p>
<div class="highlight"><pre><span></span><code> >++++++++++>>>+>+[>>>+[-[<<<<<[+<<<<<]>>[[-]>[<<+>+>-]<[>+<-]<[>+<-[>+<-[>
+<-[>+<-[>+<-[>+<-[>+<-[>+<-[>+<-[>[-]>>>>+>+<<<<<<-[>+<-]]]]]]]]]]]>[<+>-
]+>>>>>]<<<<<[<<<<<]>>>>>>>[>>>>>]++[-<<<<<]>>>>>>-]+>>>>>]<[>++<-]<<<<[<[
>+<-]<<<<]>>[->[-]++++++[<++++++++>-]>>>>]<<<<<[<[>+>+<<-]>.<<<<<]>.>>>>]
:::text
INSERT$ ./compiler fact.bf fact
INSERT$ ./executer fact
1
1
2
6
24
120
720
5040
40320
362880
.......
</code></pre></div>
<p>它会一直执行下去,直到内存溢出。</p>
<p>到此为止,我们实现了一个最简单的图灵完备的语言。虽然它很难看,也没有任何实用价值,但毕竟是第一步。而于此同时,相信肯定会有人想到,除了托管代码之外,有时候还会希望生成本地代码。因此在下一章中,我们会实现一个简单的brainfuck到汇编的转换器,并且用nasm和ld编译成实际可以执行的本地程序,同时,也会导入一些更加“高级”的结构,向着最初的目标——给计算器添加循环——走进一大步。</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>Build Compiler with LALR(1) Parser Generator(I)2014-06-02T00:00:00+09:002014-06-02T00:00:00+09:00Horsaltag:www.horsal.dev,2014-06-02:/build-compiler-with-lalr1-parser-generatori.html<p>其实从几年前开始就一直想着自己写个编译器, 不管是用LLVM编译成native code还是基于自己写的VM。起因不仅仅是因为写个编译器很“酷”, 它还是很多东西的基础——比如游戏引擎,Machine Learning时的配置文件(因为很多Parameters是需要动态改变的)。之前已经试过数次,但都因为各种各样的原因放弃了。几年过去,现在又萌生了这个想法,反正都要从头来, 干脆把这些都记录下来,就算再扔掉也不会从零开始。</p>
<p>之前曾经用过flex+yacc,也手写过LL(1)的Parser,但我并不喜欢C语言,也不打算折腾个底朝天,因为最近一直在用D语言,恰好DLang里有LALR(1)的Parser Generator - <a href="https://github.com/pwil3058/dunnart">dunnart</a>.于是以此为基础来试一试。</p>
<p>如果你不了解D语言,也不需要担心,它的语法很像C/C++,比如一个最简单的<em>Hello,World</em>:</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span> <span class="n">std</span><span class="p">.</span><span class="n">stdio</span><span class="p">;</span>
<span class="n">void</span> <span class="n">main</span><span class="p">(</span><span class="n">string</span><span class="p">[]</span> <span class="n">args</span><span class="p">){</span>
<span class="n">writeln</span><span class="p">(</span><span class="s">"Hello, World!"</span><span class="p">);</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>如果后面遇到特殊的语法 …</p><p>其实从几年前开始就一直想着自己写个编译器, 不管是用LLVM编译成native code还是基于自己写的VM。起因不仅仅是因为写个编译器很“酷”, 它还是很多东西的基础——比如游戏引擎,Machine Learning时的配置文件(因为很多Parameters是需要动态改变的)。之前已经试过数次,但都因为各种各样的原因放弃了。几年过去,现在又萌生了这个想法,反正都要从头来, 干脆把这些都记录下来,就算再扔掉也不会从零开始。</p>
<p>之前曾经用过flex+yacc,也手写过LL(1)的Parser,但我并不喜欢C语言,也不打算折腾个底朝天,因为最近一直在用D语言,恰好DLang里有LALR(1)的Parser Generator - <a href="https://github.com/pwil3058/dunnart">dunnart</a>.于是以此为基础来试一试。</p>
<p>如果你不了解D语言,也不需要担心,它的语法很像C/C++,比如一个最简单的<em>Hello,World</em>:</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span> <span class="n">std</span><span class="p">.</span><span class="n">stdio</span><span class="p">;</span>
<span class="n">void</span> <span class="n">main</span><span class="p">(</span><span class="n">string</span><span class="p">[]</span> <span class="n">args</span><span class="p">){</span>
<span class="n">writeln</span><span class="p">(</span><span class="s">"Hello, World!"</span><span class="p">);</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>如果后面遇到特殊的语法, 我也会尽量解说。</p>
<p>顺带说一句,现在几乎所有主流语言都是C-Based,而Pascal系已经几乎看不到了, 如果硬要说的话,Object Pascal/Delphi还能在每个月的TIOBE排名中看到,但却很少出现在实际应用里。唯一还能用到的似乎就是<a href="http://en.wikipedia.org/wiki/Ada_(programming_language)">Ada</a>了吧。</p>
<p>好吧,扯远了,回到正题上来。在这里我们会尝试用dunart构建一个能够运行的玩具语言,至少能够生成自己定义的VMCode,然后运行在一个最简单的虚拟机里。然后给它提供一个类似extern的keyword,让我们能够执行在host端注册过的函数,至少这样就可以拿它来写游戏脚本或者配置文件了。不过不要期待能很快实现,因为我压根不是这方面的行家,或者说,我压根不懂编译器和语言设计。因此一切从实际出发,只要最终的成品能够完成任务,我们不需要在意定义能不能满足Turning-complete,也不在意在我们的语言是否有Funarg Problem。</p>
<p>总之先从一个最简单的例子开始, 首先会构造一个计算器, 它能够计算加减乘除, 然后我们会渐渐的向里面添加功能, 包括注释,复数,Array Programming, 变量,函数, 以及类似Slot的东西来模拟object-oriented programming。</p>
<p>现在看代码,这是dunnart自带的例子, 你可以在<a href="https://github.com/pwil3058/dunnart/blob/master/examples/evaluater/parser.ddgs">dunnart的Github Repo</a>里找到它。</p>
<p><strong>parser.ddgs</strong> :</p>
<div class="highlight"><pre><span></span><code>%{
double result = 0;
double evaluate(string text){
%}
%field double value
%token PLUS "+"
%token MINUS "-"
%token TIMES "*"
%token DIVIDE "/"
%token <value> NUMBER ([0-9]+(\.[0-9]+){0,1})
%token LPR "("
%token RPR ")"
%skip ([ \n\t\r]+)
%right UMINUS
%right UPLUS
%left "*" "/"
%left "+" "-"
%%
program: expr !{ result = $1.value; !}.
expr: expr "+" expr !{$$.value = $1.value + $3.value;!}
| expr "-" expr !{$$.value = $1.value - $3.value;!}
| expr "*" expr !{$$.value = $1.value * $3.value;!}
| expr "/" expr !{$$.value = $1.value / $3.value;!}
| "(" expr ")" !{$$.value = $2.value;!}
| "-" expr %prec UMINUS !{$$.value = -$2.value;!}
| "+" expr %prec UPLUS !{$$.value = $2.value;!}
| NUMBER !{$$.value = $1.value;!}
.
%{
dd_parse_text(text);
return result;
}
%}
</code></pre></div>
<p>看起来是不是跟flex+yacc几乎一模一样? 整个文件主要有两部分, 它们之间用<strong>%%</strong>隔开,其中上半部分是lexer,下半部分是parser。而<em>%{ %}</em>所包围的区域会被原封不动的拷贝到目标文件中。让我们从头开始分析:</p>
<div class="highlight"><pre><span></span><code>%{
double result = 0;
double evaluate(string text){
%}
</code></pre></div>
<p>这一段里的东西会被直接拷贝到目标文件里, 在原例子里result是个local var, 这里我们把它移到外面,因为现在我们还没有变量,result暂时用来在运行期间保存结果。</p>
<div class="highlight"><pre><span></span><code>%field double value
%token PLUS "+"
%token MINUS "-"
%token TIMES "*"
%token DIVIDE "/"
%token <value> NUMBER ([0-9]+(\.[0-9]+){0,1})
%token LPR "("
%token RPR ")"
</code></pre></div>
<p>在这一段里定义了一个lexer,任何输入的内容都会首先通过这个lexer变成Token Stream。
比如对于输入:</p>
<div class="highlight"><pre><span></span><code>1 * 2 + 3
</code></pre></div>
<p>在通过lexer后会变成:</p>
<div class="highlight"><pre><span></span><code>NUMBER TIMES NUMBER PLUS NUMBER
</code></pre></div>
<p>显然, 在这里所有的字符串都变成了Token,但如何知道NUMBER的值呢? %field就是为此设置的。在scan的过程中,如果一个Token设置了field,那么获得的字符串会被自动转换为对应的类型。比如我们需要以字符串形式记录PLUS的值,那么只要做如下改变:</p>
<div class="highlight"><pre><span></span><code>%field double value
%field string ident
%token <ident> PLUS "+"
....
%token <value> NUMBER ....
</code></pre></div>
<p>这样当scanner遇到<strong>"+"</strong>时, 对应的值会被储存在ident里面。当然在这里没有任何意义, 但在后面我们需要添加labels,或者identifiers的时候,这会是个很重要的内容。 所有的Tokens都是通过正则表达式来定义的, 因此熟悉的人可以很轻松添加新的Tokens。</p>
<div class="highlight"><pre><span></span><code>%skip ([ \n\t\r]+)
%right UMINUS
%right UPLUS
%left "*" "/"
%left "+" "-"
</code></pre></div>
<p>至于这一段,相信意思很明确——skip正如字面意思,用来跳过空白字符, 而right和left则跟yacc中的用法一模一样,用来定义Operator associativity。简单的说就是这个operator出现在类似"A + B + C"的环境里时, 括号应该添加在哪一侧,而定义的顺序则表示了它们的优先级,定义在先的具有较高优先级,这跟yacc是不一样的, 在这里,因为乘除定义在先, 因此它们总是先于加减。如果一个operator不需要定义关联性,那么可以用<strong>%nonassoc</strong>来定义。最后,这里优先级最高的UMINUS和UPLUS定义了unary operator,我们在后面会看到。</p>
<p>总之,到这里为止,我们有了一个简单的lexer,它能够把所有可识别的内容变换成易于处理的Tokens。但显然不够,因为一个语言(计算器)不但需要这些最简单的blocks,还需要把这些blocks粘在一起的Rules。<em>yacc</em>以及派生程序都用Backus-Naur-Form(BNF)作为输入,对于BNF,拿golang的Function Types定义来做例子的话, 看起来就像这样:</p>
<div class="highlight"><pre><span></span><code>FunctionType = "func" Signature .
Signature = Parameters [ Result ] .
Result = Parameters | Type .
Parameters = "(" [ ParameterList [ "," ] ] ")" .
ParameterList = ParameterDecl { "," ParameterDecl } .
ParameterDecl = [ IdentifierList ] [ "..." ] Type .
</code></pre></div>
<p>这里Type以及IdentifierList在别的章节里已经定义过了,简单来说, 它指出Tokens(words)如果结合成一个程序(sentences),并且在定义中是允许递归的。让我们来看看这个计算器的定义:</p>
<div class="highlight"><pre><span></span><code>program: expr !{ result = $1.value; !}.
expr: expr "+" expr !{$$.value = $1.value + $3.value;!}
| expr "-" expr !{$$.value = $1.value - $3.value;!}
| expr "*" expr !{$$.value = $1.value * $3.value;!}
| expr "/" expr !{$$.value = $1.value / $3.value;!}
| "(" expr ")" !{$$.value = $2.value;!}
| "-" expr %prec UMINUS !{$$.value = -$2.value;!}
| "+" expr %prec UPLUS !{$$.value = $2.value;!}
| NUMBER !{$$.value = $1.value;!}
.
</code></pre></div>
<p>首先我们定义一个“程序”是由一个expression组成的, 并且返回值就是expression的值, 而expr则是又下面的东西组成的:</p>
<ul>
<li>两个expr和一个operator的组合 </li>
<li>括号里的expr</li>
<li>unary operator和expr的组合</li>
<li>单个数字</li>
</ul>
<p>在parsing的过程中, parser会维护一个Stack, 每遇到一个新的Token,都会将它Push进这个Stack,这个过程叫做Shift,然后检查是否满足定义里的任何一条, 如果满足,则执行预先定义的语句(<strong>!{!}</strong>里面的内容)。随后所有符合条件的Token都会被取出,然后用一个新的Node替代,这个过程叫做Reduce。在预设的命令当中,比标号从左开始设为1, 然后依次增加,比如对于 <em>expr "+" expr</em>, <span class="math">\(1.value代表了第一个expr的值,\)</span>2则是加号“+”,而$$则代表Reduce后新节点(表达式)。比如对于:</p>
<div class="highlight"><pre><span></span><code>1 + 2 * 3
</code></pre></div>
<p>经过Scanner,我们得到:</p>
<div class="highlight"><pre><span></span><code>NUMBER PLUS NUMBER TIMES NUMBER
</code></pre></div>
<p>我们用圆点"."来表示当前位置,那么在parsing时:</p>
<div class="highlight"><pre><span></span><code>. NUMBER PLUS NUMBER TIMES NUMBER
Shift
NUMBER . PLUS NUMBER TIMES NUMBER
Look Ahead: PLUS/Reduce
EXPR . PLUS NUMBER TIMES NUMBER
Shift
EXPR PLUS . NUMBER TIMES NUMBER
Shift
EXPR PLUS NUMBER . TIMES NUMBER
Look Ahead: TIMES/Reduce
EXPR PLUS EXPR . TIMES NUMBER
Look Ahead: TIMES(More precedence than +) /Shift
EXPR PLUS EXPR TIMES . NUMBER
Shift
EXPR PLUS EXPR TIMES NUMBER .
Reduce
EXPR PLUS EXPR TIMES EXPR .
Reduce
EXPR PLUS EXPR .
Reduce
EXPR .
</code></pre></div>
<p>既然名称叫做LALR(1),我们可以看到它总是在Look Ahead,然后根据这个来确定如何处理当前的Token。我们在scanner里定义的优先级也起了作用,尽管“1+2”已经可以进行Reduce,但因为“*”有更高的优先级,所以需要先处理后半部分的表达式。</p>
<p>最后,不要忘了补全最初定义的函数。</p>
<div class="highlight"><pre><span></span><code>%{
dd_parse_text(text);
return result;
}
%}
</code></pre></div>
<p>这样,当我们执行ddpg parser.ddpg时, 就会得到一个D语言的module,里面包含了我们定义的函数evaluate,调用它就可以parse输入的内容并且返回执行结果。当然,最后还要提供一个主函数:</p>
<div class="highlight"><pre><span></span><code><span class="k">import</span> <span class="n">std</span><span class="p">.</span><span class="n">stdio</span><span class="p">;</span>
<span class="k">import</span> <span class="n">std</span><span class="p">.</span><span class="n">file</span><span class="p">;</span>
<span class="k">import</span> <span class="n">std</span><span class="p">.</span><span class="n">string</span><span class="p">;</span>
<span class="k">import</span> <span class="n">parser</span><span class="p">;</span>
<span class="n">int</span> <span class="n">main</span><span class="p">(</span><span class="n">string</span><span class="p">[]</span> <span class="n">args</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="n">args</span><span class="p">.</span><span class="n">length</span> <span class="p">==</span> <span class="mi">2</span><span class="p">){</span>
<span class="n">writeln</span><span class="p">(</span><span class="n">evaluate</span><span class="p">(</span><span class="n">readText</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="mi">1</span><span class="p">])));</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">string</span> <span class="n">line</span><span class="p">;</span>
<span class="k">while</span><span class="p">((</span><span class="n">line</span> <span class="p">=</span> <span class="n">stdin</span><span class="p">.</span><span class="n">readln</span><span class="p">().</span><span class="n">strip</span><span class="p">)</span> <span class="p">!</span><span class="k">is</span> <span class="kc">null</span><span class="p">){</span>
<span class="n">writefln</span><span class="p">(</span><span class="s">"(%s) =>\n\t %s"</span><span class="p">,</span> <span class="n">line</span><span class="p">,</span> <span class="n">evaluate</span><span class="p">(</span><span class="n">line</span><span class="p">));</span>
<span class="p">}</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div>
<p>让我们来执行下简单的运算:</p>
<div class="highlight"><pre><span></span><code>./evaluator
60*60.0*24*365 ↩
(60*60.0*24*365) =>
3.1536e+07
</code></pre></div>
<p>但是这个程序依然很简陋——只能执行最简单的四则运算,没有循环, 没有变量,没有函数,甚至没法自由的获取和使用我们唯一的储存器<em>result</em>。因此在下一章,我们会给它添加变量,输出,并且以此为基础实现一个最简单的<strong>图灵完备</strong>的语言。</p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.3/latest.js?config=TeX-AMS-MML_HTMLorMML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" availableFonts: ['STIX', 'TeX']," +
" preferredFont: 'STIX'," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script>A New Blog2014-06-01T00:00:00+09:002014-06-01T00:00:00+09:00Horsaltag:www.horsal.dev,2014-06-01:/a-new-blog.html<p>之前已经好多次想过建个博客, 把自己的想法写下来。之前用过现有的博客服务, 包括国内的博客圆, CSDN,还有LiveDoor和Tumblr, 感觉并不符合我的需求, 大部分博客网站都不支持LaTex,并且没有好的Code Highlighting,最终决定用Static Site Generator, 一个是可以自己自由的修改模板, 另外还可以用Pygments的实现漂亮的高亮。 折腾了两三天,最终选择了<a href="http://blog.getpelican.com/">Pelican</a>, 有MathJax插件,并且配合已经有的Bootstrap3 Theme看起来很简洁,基本满足了要求。</p>
<p>Hosting在最初并没有过多去想,本来以为HTML无非就是提供一个小容量的httpd server,应该很容易找到服务商。实际找起来才发现并不多,能够考虑的似乎只有fc2, ninja和github。其中fc2已经取得了网络防火长城的认证,在国内不管怎么登录永远是链接被重置。最终用了ninja,看起来在国内也能正常访问,并且速度很快--比fc2稍微快一点:</p>
<ul>
<li>Ninja: 0.029s</li>
<li>FC2: 0.317s</li>
<li>Github: 1.249s</li>
</ul>
<p>唯一的遗憾是会在页面末尾追加广告, 并且很长,不过如果我能坚持把博客写下去的话,也并不会在意每年花一千多去掉广告--当然,能坚持下去的话。</p>
<p>毕竟1000块(日元 …</p><p>之前已经好多次想过建个博客, 把自己的想法写下来。之前用过现有的博客服务, 包括国内的博客圆, CSDN,还有LiveDoor和Tumblr, 感觉并不符合我的需求, 大部分博客网站都不支持LaTex,并且没有好的Code Highlighting,最终决定用Static Site Generator, 一个是可以自己自由的修改模板, 另外还可以用Pygments的实现漂亮的高亮。 折腾了两三天,最终选择了<a href="http://blog.getpelican.com/">Pelican</a>, 有MathJax插件,并且配合已经有的Bootstrap3 Theme看起来很简洁,基本满足了要求。</p>
<p>Hosting在最初并没有过多去想,本来以为HTML无非就是提供一个小容量的httpd server,应该很容易找到服务商。实际找起来才发现并不多,能够考虑的似乎只有fc2, ninja和github。其中fc2已经取得了网络防火长城的认证,在国内不管怎么登录永远是链接被重置。最终用了ninja,看起来在国内也能正常访问,并且速度很快--比fc2稍微快一点:</p>
<ul>
<li>Ninja: 0.029s</li>
<li>FC2: 0.317s</li>
<li>Github: 1.249s</li>
</ul>
<p>唯一的遗憾是会在页面末尾追加广告, 并且很长,不过如果我能坚持把博客写下去的话,也并不会在意每年花一千多去掉广告--当然,能坚持下去的话。</p>
<p>毕竟1000块(日元)跟我花在键盘上的钱相比没那么多。就是这款:</p>
<p><img alt="keyboard" src="https://www.horsal.dev/images/keyboard-box.jpg"></p>
<p>这是之前头脑发热买来的,之前一直用的是Filco的键盘,但因为某些特殊的习惯(比如Caps当ESC,不喜欢小指Enter),在每台新机子上都要折腾一番才能正常用,尤其是setxkbmap默认提供的功能并不全,每次都要修改rules,到最后干脆打算买个Programmable Keyboard算了,后来又想,既然打算买,那就干脆入个人体工学产品,反正已经是Dvorak用户,再复杂点也无所谓吧。</p>
<p>敲定目标后一通搜索, 最后选定了一些目标:</p>
<ul>
<li>Kinesis Contoured Keyboard</li>
<li>Truly Ergonomic Keyboard</li>
<li>μTRON Keyboard</li>
<li>TypeMatrix 2030 Keyboard</li>
</ul>
<p>其中Kinesis看起来很酷——虽然也很占空间。默认是Cheery Brown, 但不包括Function Keys, 也就是说最上面一排功能键不但小, 并且不是机械轴。不过这些还都在其次, 我最担心的是几乎所有的组合键都在大拇指区,这样会不会按起来需要技巧——比如Ctrl+Shift+V这类,在普通键盘上虽然左手会偏离HomeRow,但不至于按起来不舒服,而到了这个键盘上面是不是需要认真考虑左右手的组合才能按到,毕竟对于某些环境,比如emacs,是需要大量的组合键的,那样用起来会很痛苦。</p>
<p>然后是μTRON,这四款中唯一一个用了高大上RealForce的键盘, 当然价格也自然的高大上——高达5万日元并且不含税。 在设计上仅仅是简单的把普通IBM Model M分成两个部分,而价格确实最高端Realforce的近两倍,除非疯狂的爱好者,否则大概谁也不会简单出手吧。(顺带说一句,现在税率从5%涨到了8%, 也就意味着现在需要多花1500块来买这个键盘)</p>
<p>实际上TypeMatrix是我最初考虑的对象,一切看起来似乎不是很差, 除了两手没有倾斜角,以及这是一个薄膜键盘。能一键切换Dvorak和Qwert是个很吸引人的地方,四四方方的设计也有很好的视觉效果,不过就在我几乎要冒着用一年就仍的想法下单时,偶尔发现了Truly Erogonomic Keyboard。似乎本来人体工程学键盘的用户就不多,而能在网上找到的使用体验更少,一直没有注意到。跟比较容易买到的Kinesis相比,在亚马逊和雅虎都买不到,只有DiaTec(Filco的设计/销售商)在代理。考虑了很久, 我决定用3倍于TypeMatrix2030的价格买这个新玩具。</p>
<p>到手之后意外的很满意,明明我大部分情况下的冲动购买都会后悔。这款键盘几乎包含了我所需要的一切特性:</p>
<ul>
<li>多媒体键</li>
<li>可编程</li>
<li>拇指Enter</li>
<li>两手夹角, 可惜的是因为机械键盘的原因,不能想薄膜键盘做成一样中间突起的样子,没能够达到我理想中的最佳角度。</li>
</ul>
<p>唯一的问题是偶尔会遇到Double Letters的问题,并且在我的这个键盘上只有“P”键会出现Double Letters,把固件里的响应调高之后并没有解决问题,其他按键反而偶尔会没有响应。</p>
<p>对于这个问题,官方是这么解释的: </p>
<blockquote>
<p>It is not uncommon that some Mechanical keyswitches require to "break-in" before working properly; this means that the switch itself requires to be used several times before it performs properly. We therefore ask you to un-plug your keyboard from your computer, press the misbehave key(s) several times with normal pressure for a couple of minutes, then plug in your keyboard. This behavior usually disappears by itself after a few days of usage.</p>
</blockquote>
<p>不过实际问题是在我的Filco上一次也没有遇到过……</p>