博客
关于我
MIT6.824 lab1 提示思路
阅读量:242 次
发布时间:2019-03-01

本文共 3159 字,大约阅读时间需要 10 分钟。

  • The application Map and Reduce functions are loaded at run-time using the Go plugin package, from files whose names end in .so.
  • If you change anything in the mr/ directory, you will probably have to re-build any MapReduce plugins you use, with something like go build -buildmode=plugin ../mrapps/wc.go

 

这两条是对Map和Reduce函数的解释,通过.so在运行时加载。

 

  • This lab relies on the workers sharing a file system. That's straightforward when all workers run on the same machine, but would require a global filesystem like GFS if the workers run on different machines.

这些worker共享同一个文件系统,如果当worker跑在不同的机器上时,需要全局的文件系统。

 

  • A reasonable naming convention for intermediate files is mr-X-Y, where X is the Map task number, and Y is the reduce task number.

建议将中间文件名命名为mr-X-Y,X是Map任务数量,Y是reduce任务数

 

The worker's map task code will need a way to store intermediate key/value pairs in files in a way that can be correctly read back during reduce tasks. One possibility is to use Go's encoding/json package. To write key/value pairs to a JSON file:

 

worker的map任务需要存储时可以使用json

 

  • The map part of your worker can use the ihash(key) function (in worker.go) to pick the reduce task for a given key.

 

  • You can steal some code from mrsequential.go for reading Map input files, for sorting intermedate key/value pairs between the Map and Reduce, and for storing Reduce output in files.

可以从串行化的程序中参考到一些思路。

 

  • The master, as an RPC server, will be concurrent; don't forget to lock shared data.

对于RPC服务,可能会有并发问题。注意加锁。

 

 

  • Use Go's race detector, with go build -race and go run -race. test-mr.sh has a comment that shows you how to enable the race detector for the tests.

使用Go并发检测器

 

  • Workers will sometimes need to wait, e.g. reduces can't start until the last map has finished. One possibility is for workers to periodically ask the master for work, sleeping with time.Sleep() between each request. Another possibility is for the relevant RPC handler in the master to have a loop that waits, either with time.Sleep() or sync.Cond. Go runs the handler for each RPC in its own thread, so the fact that one handler is waiting won't prevent the master from processing other RPCs

 

  • The master can't reliably distinguish between crashed workers, workers that are alive but have stalled for some reason, and workers that are executing but too slowly to be useful. The best you can do is have the master wait for some amount of time, and then give up and re-issue the task to a different worker. For this lab, have the master wait for ten seconds; after that the master should assume the worker has died (of course, it might not have).

注意对worker就行心跳检测

 

  • To test crash recovery, you can use the mrapps/crash.go application plugin. It randomly exits in the Map and Reduce functions.

 

To ensure that nobody observes partially written files in the presence of crashes, the MapReduce paper mentions the trick of using a temporary file and atomically renaming it once it is completely written. You can use ioutil.TempFile to create a temporary file and os.Rename to atomically rename it.

 

  • test-mr.sh runs all the processes in the sub-directory mr-tmp, so if something goes wrong and you want to look at intermediate or output files, look there.

 

转载地址:http://xtqv.baihongyu.com/

你可能感兴趣的文章
Mysql学习总结(76)——MySQL执行计划(explain)结果含义总结
查看>>
Mysql学习总结(77)——温故Mysql数据库开发核心原则与规范
查看>>
Mysql学习总结(78)——MySQL各版本差异整理
查看>>
Mysql学习总结(79)——MySQL常用函数总结
查看>>
Mysql学习总结(7)——MySql索引原理与使用大全
查看>>
Mysql学习总结(80)——统计数据库的总记录数和库中各个表的数据量
查看>>
Mysql学习总结(81)——为什么MySQL不推荐使用uuid或者雪花id作为主键?
查看>>
Mysql学习总结(82)——MySQL逻辑删除与数据库唯一性约束如何解决?
查看>>
Mysql学习总结(83)——常用的几种分布式锁:ZK分布式锁、Redis分布式锁、数据库分布式锁、基于JDK的分布式锁方案对比总结
查看>>
Mysql学习总结(84)—— Mysql的主从复制延迟问题总结
查看>>
Mysql学习总结(85)——开发人员最应该明白的数据库设计原则
查看>>
Mysql学习总结(8)——MySql基本查询、连接查询、子查询、正则表达查询讲解
查看>>
Mysql学习总结(9)——MySql视图原理讲解与使用大全
查看>>
MySQL学习笔记十七:复制特性
查看>>
Mysql学习第一课-mysql的定义及sql语句
查看>>
mysql安全模式: sql_safe_updates
查看>>
mysql安装,卸载,连接
查看>>
MySQL安装之没有配置向导
查看>>
mysql安装出现 conflicts with mysql*的解决办法
查看>>
mysql安装卡在最后一步解决方案(附带万能安装方案)
查看>>