《云计算的关键技术与挑战.pptx》由会员分享，可在线阅读，更多相关《云计算的关键技术与挑战.pptx（43页珍藏版）》请在课桌文档上搜索。
1、云计算的关键技术与挑战,主要内容,云计算的关键技术云计算的技术挑战Google file System介绍,云计算的关键技术云计算的技术挑战Google file System介绍,云计算的关键技术,快速部署资源调度多租户海量数据处理大规模消息通信大规模分布式存储许可证管理和计费,云计算的关键技术快速部署,并行部署技术将传统的顺序部署方式改为并行执行，同时执行多个部署任务，将虚拟机同时部署到多个物理及上制约部署速度因素部署服务器的读写能力或部署系统的有限网络带宽协同部署技术将虚拟机镜像在多个目标物理机之间的网络中传输，提高部署速度制约部署速度因素目标物理机之间的网络带宽总和,云计算的关键技术资
2、源调度,资源调度在特定的资源环境下，根据一定的资源使用规则，在不同的资源使用者之间进行资源调整的过程两种途径调整计算任务的资源使用量转移计算任务云计算的新挑战海量规模满足服务级别协定,云计算的关键技术多租户技术,多租户技术大量用户共享同一堆栈的软、硬件资源，每个用户按需使用资源，能够对软件服务进行客户化配置而不影响其他用户的使用采用多租户技术的SaaS应用特征SaaS应用基于Web的租户能够对SaaS平台本身进行扩展技术难点数据隔离、客户化配置、架构扩展、性能定制,云计算的关键技术海量数据处理,海量数据处理对大规模数据的计算和分析，通常数据的规模可达TB甚至PB级别典型例子搜索引擎并行计算模型
3、River编程模型MapReduce编程模型,云计算的关键技术大规模消息通信,同步消息通信异步消息通信云计算中新的挑战足够稳定能够伸缩保证安全高效率图,云计算的关键技术大规模分布式存储,大规模分布式存储技术分布式文件系统FranqupaniGoogle File System云存储服务Amazon Simple Storage ServiceGoogle BigTable,云计算的关键技术许可证管理与计费,IT基础设施的许可证管理与计费模式按需付费按使用计费大量提供商还未制定产品在云计算环境下的计费模式较成熟的是Amazon提供的EC2和S3的按量计费模式,云计算的关键技术云计算的技术挑战Go
4、ogle file System介绍,云计算的技术挑战,安全性可用性可伸缩性信息保密高性能标准化,云计算的技术挑战安全性,云计算特有的安全问题传统观念转变政策法规保障云中每个节点都可能受到攻击现有软件系统安全防护模式如何改变,云计算的技术挑战可用性,软件系统在一定时间内正常工作的时间占总时间的比重，通常用百分比衡量云计算环境能够在最大程度上减少资源的不可用对业务系统的影响通过技术创新，保证即使软、硬件出现问题服务仍然可用,云计算的技术挑战可伸缩性,通过资源的增加或减少来应对负载的变化，并保持一致的性能垂直伸缩在现有的服务节点上增加或减少资源水平伸缩在现有的服务节点上增加或减少服务节点,云计算的
5、技术挑战信息保密,信息的内容不应该被未授权的人得到非法用户访问难度较大数据在云的大规模分布式存储机制中，完整的数据实体被打散存储在不同的服务器上，而每个数据块可包含不同的数据实体根本方法从逻辑上甚至物理上将多个用户的数据隔离,云计算的技术挑战高性能,云环境所承担的计算、存储和通信方面的负载大于传统的计算环境服务器虚拟化技术CPU开销较小内存性能开销较大（访问冲突）大规模数据处理技术MapReduce适用性问题原语设计导致的性能问题大量的网络消息通信问题分布式存储技术面对网络不可控的环境,云计算的技术挑战标准化,维护多个云之间的数据同步、应用版本同步、应用在多个云之间的互操作云计算的标准化工作还
6、在酝酿之中开放式云宣言(Open Cloud Manifesto)总结云计算的特点和现有挑战建立开放的云基础设施将是未来云计算领域的发展趋势对开放标准的呼吁,云计算的关键技术云计算的技术挑战Google file System介绍,The Google File System(GFS),A scalable distributed file system for large distributed data intensive applicationsMultiple GFS clusters are currently deployed.The largest ones have:1000+
7、storage nodes300+TeraBytes of disk storageheavily accessed by hundreds of clients on distinct machines,Introduction,Shares many same goals as previous distributed file systemsperformance,scalability,reliability,etcGFS design has been driven by four key observation of Google application workloads and
8、 technological environment,Intro:Observations 1,1.Component failures are the normconstant monitoring,error detection,fault tolerance and automatic recovery are integral to the system2.Huge files(by traditional standards)Multi GB files are commonI/O operations and blocks sizes must be revisited,Intro
9、:Observations 2,3.Most files are mutated by appending new dataThis is the focus of performance optimization and atomicity guarantees4.Co-designing the applications and APIs benefits overall system by increasing flexibility,The Design,Cluster consists of a single master and multiple chunkservers and
10、is accessed by multiple clients,The Master,Maintains all file system metadata.names space,access control info,file to chunk mappings,chunk(including replicas)location,etc.Periodically communicates with chunkservers in HeartBeat messages to give instructions and check state,The Master,Helps make soph
11、isticated chunk placement and replication decision,using global knowledgeFor reading and writing,client contacts Master to get chunk locations,then deals directly with chunkserversMaster is not a bottleneck for reads/writes,Chunkservers,Files are broken into chunks.Each chunk has a immutable globall
12、y unique 64-bit chunk-handle.handle is assigned by the master at chunk creationChunk size is 64 MBEach chunk is replicated on 3(default)servers,Clients,Linked to apps using the file system API.Communicates with master and chunkservers for reading and writingMaster interactions only for metadataChunk
13、server interactions for dataOnly caches metadata informationData is too large to cache.,Chunk Locations,Master does not keep a persistent record of locations of chunks and replicas.Polls chunkservers at startup,and when new chunkservers join/leave for this.Stays up to date by controlling placement o
14、f new chunks and through HeartBeat messages(when monitoring chunkservers),Operation Log,Record of all critical metadata changesStored on Master and replicated on other machinesDefines order of concurrent operationsChanges not visible to clients until they propigate to all chunk replicasAlso used to
15、recover the file system state,System Interactions:Leases and Mutation Order,Leases maintain a mutation order across all chunk replicasMaster grants a lease to a replica,called the primaryThe primary choses the serial mutation order,and all replicas follow this orderMinimizes management overhead for
16、the Master,System Interactions:Leases and Mutation Order,Atomic Record Append,Client specifies the data to write;GFS chooses and returns the offset it writes to and appends the data to each replica at least onceHeavily used by Googles Distributed applications.No need for a distributed lock managerGF
17、S choses the offset,not the client,Atomic Record Append:How?,Follows similar control flow as mutationsPrimary tells secondary replicas to append at the same offset as the primaryIf a replica append fails at any replica,it is retried by the client.So replicas of the same chunk may contain different d
18、ata,including duplicates,whole or in part,of the same record,Atomic Record Append:How?,GFS does not guarantee that all replicas are bitwise identical.Only guarantees that data is written at least once in an atomic unit.Data must be written at the same offset for all chunk replicas for success to be
19、reported.,Replica Placement,Placement policy maximizes data reliability and network bandwidthSpread replicas not only across machines,but also across racksGuards against machine failures,and racks getting damaged or going offlineReads for a chunk exploit aggregate bandwidth of multiple racksWrites h
20、ave to flow through multiple rackstradeoff made willingly,Chunk creation,created and placed by master.placed on chunkservers with below average disk utilizationlimit number of recent“creations”on a chunkserverwith creations comes lots of writes,Detecting Stale Replicas,Master has a chunk version num
21、ber to distinguish up to date and stale replicasIncrease version when granting a leaseIf a replica is not available,its version is not increasedmaster detects stale replicas when a chunkservers report chunks and versionsRemove stale replicas during garbage collection,Garbage collection,When a client
22、 deletes a file,master logs it like other changes and changes filename to a hidden file.Master removes files hidden for longer than 3 days when scanning file system name spacemetadata is also erasedDuring HeartBeat messages,the chunkservers send the master a subset of its chunks,and the master tells
23、 it which files have no metadata.Chunkserver removes these files on its own,Fault Tolerance:High Availability,Fast recoveryMaster and chunkservers can restart in secondsChunk ReplicationMaster Replication“shadow”masters provide read-only access when primary master is downmutations not done until rec
24、orded on all master replicas,Fault Tolerance:Data Integrity,Chunkservers use checksums to detect corrupt dataSince replicas are not bitwise identical,chunkservers maintain their own checksumsFor reads,chunkserver verifies checksum before sending chunkUpdate checksums during writes,小结,云计算产生、发展、推广过程中的新技术云计算的不足和面临的挑战GFS中的主要技术,
