银河里的星星

落在人间

日志

关于我

星星

文章分类

Spanner: Google s next Massive Storage and Computation infrastructure(zz)

2010-12-18 12:22:41| 分类：搜索与分布式 | 标签： |举报 |字号大中小订阅

下载LOFTER 我的照片书 |

Spanner: Google s next Massive Storage and Computation infrastructure

MapReduce Bigtable and Pregel have their origins in Google and they all deal with large systems . But all of them may be dwarfed in size and complication by a new project Google is working on which was mentioned briefly (may be un-intentionally) at an event last year. Spanner: Google s next Massive Storage and Computation infrastructure(zz) - 星星 - 银河里的星星

Instead of caching data closer to user it looks love Google is trying to receive the data to the user. If you use GMail or a Google Doc service then with this framework Google could auto-magically move one of the master copies of your data to the nearest Google data center without truly having to cache anything locally. And since they are structure one single datastore cluster round the world instead of building hundreds of smaller ones for different applications it looks like they may not don t need dedicated clusters for specific projects anymore.

Below is the gist of Spanner from a talk by Jeff Dean at Symposium held at Cornell. Take a seem at the rest of the slides if you are interested in some impressive statistics on hardware performance and reliability.

Spanner: Storage & computation system that spans all our datacenters Single global namespaceNames are independent of location(s) of dataSimilarities to Bigtable: table families locality groups coprocessors Differences: hierarchical directories instead of rows fine-grained replicationFine-grained ACLs replication configuration at the per-directory levelsupport mingle of strong and weak consistency along datacentersStrong consistency implemented with Paxos across tablet replicasFull support for distributed transactions across directories/machinesmuch more automated operationSystem automatically moves and adds replicas of data and computation based on constraints and usage patternsAutomated allocation of resources across entire fleet of machines.

ReferencesGoogle: Designs Lessons and Advice from Building Large Distributed Systems2010: Google s Traffic is giant Which is why it should be your ISP 2009: Google Spanner instamatic redundancy for 10 million servers ? 2007: What is Google doing with all that dark fiber 2005: Google wants dark fiber

Related posts:

Pregel: Google s other data-processing infrastructureThe real concerns about Cloud infrastructure (as it is today)Is Yahoo launching a cloud storage solution : MObStorBrewers CAP Theorem on distributed systemsGoogle app locomotive review (Java edition)
datastore eventually consistent framework google mapreduce replication scalabilitydatastore eventually consistent google mapreduce replication scalability

You can follow any responses to this entry through the RSS 2.0 feed.You can leave a response or trackback from your own site.

http://sancc.com/archives/2293

谷歌曾提出一个名为 Google Spanner 的远景规划：该规划的中心是谷歌一旦流量激增、硬件负荷过重，数据便在百万级的数据中心中自动转移。

今年夏天，谷歌首次暗示将开发分布式技术增强服务器功能，当数据中过超载或者过热时，数据自动且即时进行再分配，该技术是一种后端技术。最近，谷歌工程师、杰出搜索专家杰夫·狄恩 (Jeff Dean) 证明了此消息，并提供了PDF介绍文档。

新平台名为 Google Spanner，在英文中，Spanner原意为“扳手”，又有“桥梁的交叉支撑”之意。狄恩在报告中描述说：“这是存储与计算系统，它横跨我们的数据中心，自动移动与复制数据”。它与带宽、数据包丢失、能源、资源相关，还包括失败模式等内容。他还说，平台是对整个数据中心资源的再定位。

目前，谷歌在全球有36个数据中心，少许还在建设之中。谷歌从未透露自己有多少服务器，但是从狄恩的报告中却透露谷歌的规划：未来将达到1000万台服务器。

在文档中，狡恩阐述了谷歌的远景规划 Google Spanner 将拥有100万－1000万台服务器，围绕10兆个虚拟目录，面向10的18次方字节存储空间，它们将分布在全球。

对此，谷歌拒评。谷歌的PR也未对 Google Spanner 多作说明。谷歌的高级工程师与架构师吉尔·沃克斯（Gill waxed）在旧金山的一个会议上却略有介绍。他将谷歌的分布式在线架构叫做“仓库式”机群，可以从危险与过热的数据中心中自动移动数据。他说：“我们正在建设的是仓库式的计算平台，当然，目前还是纸面意义多些。你必须与一切整合，从冷冻到CPU，都必须整合。”系统无须人工干预，即可完成大量工作。

当问及新技术是否已经投入使用时，沃克斯采用了外交式言辞作答：“我恐怕无法对此做出评论。”

评论这张

转发至微博

阅读(4609)| 评论(0)

历史上的今天

this.p={  m:2,
              b:2,
              loftPermalink:'',
              id:'fks_094065083087088069082080087095083086088068085081080068',
              blogTitle:'Spanner: Google s next Massive Storage and Computation infrastructure(zz)',
              blogAbstract:'Spanner: Google s next Massive Storage and Computation infrastructure<BR\>  <P style=\"TEXT-INDENT: 2em;\"  \>MapReduce Bigtable and Pregel have their origins in Google and they all deal with large systems . But all of them may be dwarfed in size and complication by a new project Google is working on which was mentioned briefly (may be un-intentionally) at an event last year. </P\>',
              blogTag:'spanner,谷歌,storage,infrastructure,google',
              blogUrl:'blog/static/7097176720101118112320',
              isPublished:1,
              istop:false,
              type:2,
              modifyTime:1316972026246,
              publishTime:1292646161504,
              permalink:'blog/static/7097176720101118112320',
              commentCount:0,
              mainCommentCount:0,
              recommendCount:0,
              bsrk:-100,
              publisherId:0,
              recomBlogHome:false,
              currentRecomBlog:false,
              attachmentsFileIds:[],
              vote:{},
              groupInfo:{},
              friendstatus:'none',
              followstatus:'unFollow',
              pubSucc:'',
              visitorProvince:'',
              visitorCity:'',
              visitorNewUser:false,
              postAddInfo:{},
              mset:'000',
              mcon:'',
              srk:-100,
              remindgoodnightblog:false,
              isBlackVisitor:false,
              isShowYodaoAd:false,
              hostIntro:'',
              hmcon:'1',
              selfRecomBlogCount:'0',
              lofter_single:'<iframe width="140" height="560" style="overflow:hidden;" src="http://www.lofter.com/mailEntry.do?blogad=1&blog" frameBorder="0"></iframe>'
            }

{list a as x}
    {if !!x}
    <div class="iblock nbw-fce nbw-f40">
      <a class="fc03 noul" target="_blank" hidefocus="true" href="http://blog.163.com/${x.visitorName}/">
      {if x.visitorName==visitor.userName}
      <img alt="${x.visitorNickname|escape}" onerror="this.src=location.f40" class="cwd bdwa bdc0" src="${fn1(x.visitorName)}&r=${visitor.imageUpdateTime}"/>
      {else}
      <img alt="${x.visitorNickname|escape}" onerror="this.src=location.f40" class="cwd bdwa bdc0" src="${fn1(x.visitorName)}"/>
      {/if}
      </a>
      <div class="cwd vname thide">
        {if x.moveFrom=='wap'}
          <a class="noul pnt" target="_blank" href="http://blog.163.com/services/wapblog.html?frompersonalbloghome"><span title="来自网易手机博客" class="iblock wapIcon"> </span></a>
        {elseif x.moveFrom=='iphone'}
          <a class="noul pnt" target="_blank"><span title="来自iPhone客户端" class="iblock iphoneIcon"> </span></a>
        {elseif x.moveFrom=='android'}
          <a class="noul pnt" target="_blank"><span title="来自Android客户端" class="iblock androidIcon"> </span></a>
        {elseif x.moveFrom=='mobile'}
          <a class="noul pnt" target="_blank" href="http://blog.163.com/services/emsblog.html?frompersonalbloghome"><span title="来自网易短信写博" class="iblock wapIcon"> </span></a>
        {/if}
        <a class="fc03 m2a"  target="_blank" hidefocus="true" href="http://blog.163.com/${x.visitorName}/">
          ${fn(x.visitorNickname,8)|escape}
        </a>
      </div>
    </div>
    {/if}
    {/list}

<#--最新日志，群博日志--> <#--推荐日志-->

<p class="fc06">推荐过这篇日志的人：</p>
    <div>
      {list a as x}
      {if !!x}
      <div class="iblock nbw-fce nbw-f40">
        <a class="fc03 noul" target="_blank" hidefocus="true" href="http://blog.163.com/${x.recommenderName}/">
        <img alt="${x.recommenderNickname|escape}" onerror="this.src=location.f40" class="cwd bdwa bdc0" src="${fn1(x.recommenderName)}"/>
        </a>
        <div class="cwd thide">
          <a class="fc03 m2a" target="_blank" hidefocus="true" href="http://blog.163.com/${x.recommenderName}/">
            ${fn(x.recommenderNickname,6)|escape}
          </a>
        </div>
      </div>
      {/if}
      {/list}
    </div>
    {if !!b&&b.length>0}
    <p  class="fc06">他们还推荐了：</p>
    <ul>
    {list b as y}
      {if !!y}
        <li class="rrb"><span class="iblock">·</span><a class="fc03 m2a" target="_blank" href="http://blog.163.com/${y.recommendBlogPermalink}/?from=blog/static/7097176720101118112320">${y.recommendBlogTitle|escape}</a></li>
      {/if}
    {/list}
    </ul>
    {/if}

<#--引用记录--> <#--博主推荐--> <#--随机阅读--> <#--首页推荐--> <#--历史上的今天--> <#--被推荐日志--> <#--上一篇，下一篇--> <#-- 热度 -->

{list a as x}
    {if !!x}
    <div class="hotItem iblock nbw-fce nbw-f40">
      <a class="fc03 noul" target="_blank" hidefocus="true" href="http://blog.163.com/${x.publisherUsername}/">
      {if x.publisherUsername==visitor.userName}
      <img alt="${x.publisherNickname|escape}" onerror="this.src=location.f40" class="cwd bdwa bdc0" src="${fn1(x.publisherUsername)}&r=${visitor.imageUpdateTime}"/>
      {else}
      <img alt="${x.publisherNickname|escape}" onerror="this.src=location.f40" class="cwd bdwa bdc0" src="${fn1(x.publisherUsername)}"/>
      {/if}
      </a>
      <div class="cwd vname thide">
        <a class="fc03 m2a"  target="_blank" hidefocus="true" href="http://blog.163.com/${x.publisherUsername}/">
          ${fn(x.publisherNickname,8)|escape}
        </a>
      </div>
      <a class="f-myLikeIcons hottype {if x.type==1} js-liketype{elseif x.type==2} js-reblogtype{elseif x.type==3} js-sharetype{else}{/if}" target="_blank" hidefocus="true" href="http://blog.163.com/${x.publisherUsername}/"> </a>
    </div>
    {/if}
    {/list}

<#-- 网易新闻广告 --> <#--右边模块结构--> <#--评论模块结构--> <#--引用模块结构--> <#--博主发起的投票-->

页脚

我的照片书 - 手机博客 - 下载LOFTER APP - 订阅此博客

银河里的星星

导航

日志

Spanner: Google s next Massive Storage and Computation infrastructure(zz)

历史上的今天

最近读者

热度

评论

页脚