刘鹏
中国网格信息中转站：
中国云计算：
JP Navarro <navarro@>
*
报告简介：自2007年第4季度开始，“云计算”变成了炙手可热的一个词。Google、Amazon、IBM、微软等IT巨头们以前所未有的速度和规模推动云计算技术和产品的普及，一些学术活动迅速将云计算提上议事日程，支持的声音和反对的声音不绝于耳。那么，云计算到底是什么？它是不是一种炒作？从技术上看它是如何实现的？它与网格计算是什么关系？网格计算搞了这么多年，结果如何？网格计算和云计算的未来是什么？本次讲座试图基于大量的实例和数据分析这些问题…
主题报告： 网格计算与云计算
报 告 人： 刘 鹏 教授
报告时间： 2009年3月6日（周五）9:00～11:00
报告地点：上海大学延长校区行健楼707报告厅
JP Navarro <navarro@>
标志此页从他人的PPT引用而来
标志此页从他人的PPT改造而来
著作权声明：本PPT相当部分页面引用或改造自他人
PPT，其著作权属于其创作者。在此，向他们在网上共享
所创作的内容表示致敬和感谢！
JP Navarro <navarro@>
内 容 提 纲
JP Navarro <navarro@>
*
云计算是个新名词
JP Navarro <navarro@>
*
Cloud Computing
*
JP Navarro <navarro@>
Google云计算
JP Navarro <navarro@>
*
Google云计算数据中心
JP Navarro <navarro@>
*
User Centric
Data stored in the “Cloud”
Data follows you & your devices
Data accessible anywhere
Data can be shared with others
JP Navarro <navarro@>
Google的三大法宝
Google File System(GFS) BigTable MapReduce
*
GFS Architecture
Google
48%
MSN
19%
Yahoo
33%
Files broken into chunks (typically 64 MB)
Master manages metadata
Data transfers happen directly between clients/chunkservers
Client
Replicas
Masters
GFS Master
GFS Master
C0
C1
C2
C5
Chunkserver 1
C0
C2
C5
Chunkserver N
C1
C3
C5
Chunkserver 2
…
JP Navarro <navarro@>
Enter Title of Presentation Here
Google Confidential
*
GFS Usage @ Google
200+ clusters
Filesystem clusters of up to 5000+ machines
Pools of 10000+ clients
5+ Petabyte Filesystems
All in the presence of frequent HW failure
JP Navarro <navarro@>
*
Google的三大法宝
Google File System(GFS) BigTable MapReduce
BigTable
Data model
(row, column, timestamp) cell contents
JP Navarro <navarro@>
*
BigTable
Distributed multi-level sparse map
Fault-tolerance, persistent
Scalable
Thousand of servers
Terabytes of in-memory data
Petabytes of disk-based data
Self-managing
Servers can be added/removed dynamically
Servers adjust to load imbalance
JP Navarro <navarro@>
*
Google的三大法宝
Google File System(GFS) BigTable MapReduce
MapReduce – A New Model and System
Two phases of data processing
Map: (in_key, in_value) {(keyj, valuej) | j = 1…k}
Reduce: (key, [value1,…valuem]) (key, f_value)
JP Navarro <navarro@>
*
JP Navarro <navarro@>
*
JP Navarro <navarro@>
*
JP Navarro <navarro@>
*
JP Navarro <navarro@>
*
JP Navarro <navarro@>
*
JP Navarro <navarro@>
*
JP Navarro <navarro@>
*
JP Navarro <navarro@>
*
JP Navarro <navarro@>
*
亚玛逊云计算
JP Navarro <navarro@>
*
Amazon Simple Storage Service
JP Navarro <navarro@>
S3 SOAP/Query API
Service:
ListAllMyBuckets
Buckets:
CreateBucket
DeleteBucket
ListBucket
GetBucketAccessControlPolicy
SetBucketAccessControlPolicy
GetBucketLoggingStatus
SetBucketLoggingStatus
Objects:
PutObject
PutObjectInline
GetObject
GetObjectExtended
DeleteObject
GetObjectAccessControlPolicy
SetObjectAccessControlPolicy
JP Navarro <navarro@>
JP Navarro <navarro@>
Amazon Elastic Compute Cloud
JP Navarro <navarro@>
EC2 SOAP/Query API
Images:
RegisterImage
DescribeImages
DeregisterImage
Instances:
RunInstances
DescribeInstances
TerminateInstances
GetConsoleOutput
RebootInstances
Keypairs:
CreateKeyPair
DescribeKeyPairs
DeleteKeyPair
Image Attributes:
ModifyImageAttribute
DescribeImageAttribute
ResetImageAttribute
Security Groups:
CreateSecurityGroup
DescribeSecurityGroups
DeleteSecurityGroup
AuthorizeSecurityGroupIngress
RevokeSecurityGroupIngress
JP Navarro <navarro@>
Web-Scale Architecture
JP Navarro <navarro@>
微软云计算
JP Navarro <navarro@>
*
*
JP Navarro <navarro@>
*
*
JP Navarro <navarro@>
*
*
JP Navarro <navarro@>
*
Enterprise IT spending challenge
Source: IBM Corporate Strategy analysis of IDC data, Sept. 2007
Global Annual IT Spending
Estimated US$B 1996-2010
$0B
50
100
150
200
250
300
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
New Server Spending
Server Mgt and Admin Costs
Power and Cooling Costs
JP Navarro <navarro@>
*
Dream or Nightmare?
JP Navarro <navarro@>
新概念层出不穷
Utility Computing
SaaS = Software as a Service
?
?
Grid Computing
Cluster Computing
Cloud Computing
JP Navarro <navarro@>
*
可以分为两类
使用模式
基础设施
JP Navarro <navarro@>
*
云计算的技术背景
云计算是并行计算(Parallel Computing)、分布式计算(Distributed Computing)和网格计算(Grid Computing)的发展，或者说是这些计算机科学概念的商业实现。
云计算是虚拟化(Virtualization)、效用计算(Utility Computing)、IaaS(基础设施即服务)、PaaS(平台即服务)、SaaS(软件即服务)等概念混合演进并跃升的结果。
JP Navarro <navarro@>
*
云计算的定义
云计算是一种商业计算模型。它将计算任务分布在大量计算机构成的资源池上，使各种应用系统能够根据需要获取计算力、存储空间和各种软件服务。
JP Navarro <navarro@>
*
*
JP Navarro <navarro@>
*
云计算的概念模型
*
云计算的服务类型
*
云计算技术体系结构
*
云计算简化实现机制
*
云计算的特点
超大规模
虚拟化
高可靠性
通用性
高可扩展性
按需服务
极其廉价
JP Navarro <navarro@>
*
内 容 提 纲
JP Navarro <navarro@>
*
网格计算的定义
在动态变化、由多个机构组成的虚拟组织中协调资源共享和求解问题。
JP Navarro <navarro@>
*
*
虚拟组织
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
TeraGrid
JP Navarro <navarro@>
*
*
What is the TeraGrid?
Technology + Support = Science
NSF已投资亿美元
自2004年10月已处于生产运行阶段，目前已用高性能网络集成了每秒750万亿次计算能力、30PB存储空间和100多个学科的数据库资源。
JP Navarro <navarro@>
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
*
TeraGrid PI’s By Institution
TeraGrid PI’s
Blue: 10 or more PI’s
Red: 5-9 PI’s
Yellow: 2-4 PI’s
Green: 1 PI
JP Navarro <navarro@>
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
*
This Google Map mashup (using ) shows TeraGrid principal investigators for roughly 1,000 allocated projects as of June 2006.
Different icons represent different numbers of Pis per institution.
*
TeraGrid Used
JP Navarro <navarro@>
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
*
TeraGrid’s 3-pronged strategy to further science
DEEP Science: Enabling Terascale Science
Make science more productive through an integrated set of very-high capability resources
ASTA projects
WIDE Impact: Empowering Communities
Bring TeraGrid capabilities to the broad science community
Science Gateways
OPEN Infrastructure, OPEN Partnership
Provide a coordinated, general purpose, reliable set of services and resources
Grid interoperability working group
JP Navarro <navarro@>
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
*
DEEP- To ensure that we are harnessing the enormous power of the TeraGrid resources as an integrated system we have the Advanced Support for TeraGrid Applications (ASTA) program that involves our distributed user support team (~20 people across 8 sites, coordinated by GIG). The ASTA program places 1/4 to 1/2 of a support person in a scientific application team for 4 to 12 months, working with that team to exploit TeraGrid capabilities. The ASTA program supports roughly a dozen teams at a time, with a goal of 20-25 teams supported per year.
WIDE- For over 2 decades the NSF high performance computing program, including TeraGrid, has served several thousand users very effectively. However, NSF alone funds tens of thousands of scientists, most of whom have computational requirements that do not frequently require supercomputers. The TeraGrid Science Gateways program is a set of partnerships with discipline-specific teams who are providing computational infrastructure for their science communities. The partnerships involve integrating TeraGrid as a computational and data management “service provider” embedded in the science-community infrastructure. Roughly a dozen web portal style science gateways are part of this program, with more being added continually. In addition, this program includes a peer-grid interoperation effort with Open Science Grid and a desktop application effort that is leveraging an NIH-funded biomedical project directed by Rick Stevens at the University of Chicago.
OPEN- TeraGrid began as an infrastructure involving four partner sites, grew to nine sites, and is currently organized as a set of eight resource providers and a central “grid infrastructure group” (GIG) providing central services and support as well as management and operations. This new structure allows TeraGrid to grow to dozens of resource provider sites, making it an “open” partnership. In addition, TeraGrid architecture is service-oriented, stressing open source standards such as are deployed with key software including the Globus Toolkit GT4, Condor, and other tools.
*
Science Gateways
A new initiative for the TeraGrid
Provide “TeraGrid Inside” capabilities
Application programs running on users' machines but accessing services in TeraGrid
JP Navarro <navarro@>
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
*
TeraGrid leaders created the Science Gateways concept and initiative as a way to adapt the TeraGrid to the environments that users have adopted (. web portals), leveraging the work of these science communities in designing and building their infrastructure. This approach also leverages education efforts of these communities, and allows for an order of magnitude increase in the number of users served by TeraGrid. This increase is possible by changing the model of provision of resrouces and support, where the TeraGrid serves gateways as a “wholesale” supplier, working with the gateway providers who serve as the “retailers” providing tailored support to communities of users.
*
Gateways are growing in numbers
10 initial projects as part of TG proposal
>20 Gateway projects today
No limit on how many gateways can use TG resources
Prepare services and documentation so developers can work independently
Open Science Grid (OSG)
Special PRiority and Urgent Computing Environment (SPRUCE)
National Virtual Observatory (NVO)
Linked Environments for Atmospheric Discovery (LEAD)
Computational Chemistry Grid (GridChem)
Computational Science and Engineering Online (CSE-Online)
GEON(GEOsciences Network)
Network for Earthquake Engineering Simulation (NEES)
SCEC Earthworks Project
Network for Computational Nanotechnology and nanoHUB
GIScience Gateway (GISolve)
Biology and Biomedicine Science Gateway
Open Life Sciences Gateway
The Telescience Project
Grid Analysis Environment (GAE)
Neutron Science Instrument Gateway
TeraGrid Visualization Gateway, ANL
BIRN
Gridblast Bioinformatics Gateway
Earth Systems Grid
Astrophysical Data Repository (Cornell)
Many others interested
SID Grid
HASTAC
JP Navarro <navarro@>
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
Open Science Grid
JP Navarro <navarro@>
*
*
96 Resources across
production & integration infrastructures
20 Virtual Organizations +6 operations
Includes 25% non-physics.
~20,000 CPUs (from 30 to 4000)
~6 PB Tapes
~4 PB Shared Disk
Sustaining through OSG submissions:
3,000-4,000 simultaneous jobs .
~10K jobs/day
~50K CPUhours/day.
Peak test jobs of 15K a day.
Using production & research networks
OSG Snapshot
JP Navarro <navarro@>
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
NERSC
BU
UNM
SDSC
UTA
OU
FNAL
ANL
WISC
BNL
VANDERBILT
PSU
UVA
CALTECH
IOWA STATE
PURDUE
IU
BUFFALO
TTU
CORNELL
ALBANY
UMICH
INDIANA
IUPUI
STANFORD
UWM
UNL
UFL
KU
UNI
WSU
MSU
LTU
LSU
CLEMSON
MCGILL
UMISS
UIUC
UCR
UCLA
LEHIGH
NSF
ORNL
HARVARD
UIC
SMU
UCHICAGO
What is the Open Science Grid?
(+Brazil, Mexico, Tawain, UK)
JP Navarro <navarro@>
Close collaboration with EGEE/WLCG.
5 DOE Labs - BNL, Fermilab, NERSC, ORNL, SLAC. 65 Universities. 5 partner campus/regional grids.
>43,000 cores, 6 Petabyes disk cache, 10 Petabyes tape stores
15,000 CPU WallClock days/day. 1 petabyte data distributed/month.
100,000 application jobs/day. 20% cycles through resource sharing, opportunistic use.
*
OSG应用
Genome sequence analysis
STAR: 5 TB transfer
(SRM, GridFTP)
Sloan digital sky survey
Earth System Grid:
O(100TB) online data
JP Navarro <navarro@>
*
Earth System Grid
JP Navarro <navarro@>
*
Sloan digital sky survey
Chandra
Hubble
MMT
Sub-mm array
VLA
Antartica sub-mm
Magellan
Whipple g-ray
SIRTF
Oak Ridge
CO
LAMOST
Sloan digital sky survey
JP Navarro <navarro@>
*
*
EGEE
(Enabling Grids for E-sciencE)
JP Navarro <navarro@>
*
European Grid Initiative
JP Navarro <navarro@>
*
June 2, 2008
*
Archeology
Astronomy
Astrophysics
Civil Protection
Comp. Chemistry
Earth Sciences
Finance
Fusion
Geophysics
High Energy Physics
Life Sciences
Multimedia
Material Sciences
…
>250 sites
48 countries
>50,000 CPUs
>20 PetaBytes
>10,000 users
>150 VOs
>150,000 jobs/day
EGEE
June 2, 2008
*
*
EGEE workload in 2007
CPU: 114 Million hours
Data:
25PB stored
11PB transferred
JP Navarro <navarro@>
Chart1
11400000
Xfer
CPU
Storage
Sheet1
How much would it cost to run EGEE's 2007 workload on Amazon?
This page shows how the calcuation of cost of running EGEE workload from 2007 ((all VOs, all sites)) on Amazon S3 and EC2 was made
CPU: $11,400,000
EGEE in 2007 consumed 114million CPU-hours
Typical EGEE site CPU corresponds to Amazon "small instance" - GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform - $ /hour
Storage: $47,185,920
EGEE in 2007: 25Petabytes == 25600Terabytes == 26214400Gigabytes
Amazon S3 pricing (Europe): $ per GB-Month of storage used
314572800
$ * 26214400Gb * 12 months
Data Xfer: $1,397,983
EGEE aggregrate data xfer between all sites: == 11694Tb == 11974737Gb hence average of 997895Gb/month. Assume 50% in and 50% out
Amazon data Xfer pricing:
$ per GB - all data transfer in
$ per GB - first 10 TB / month data transfer out
$ per GB - next 40 TB / month data transfer out
$ per GB - next 100 TB data transfer out / month
$ per GB - data transfer out / month over 50 TB
TOTAL $59,983,903 1,397,
In Euros € 47,486,548 9,500,
69,421,
Sheet1
storage
CPU
Xfer
summary
EGEE07 figures put these into
storage (GB-months) US 314572800
data xfer-in
data xfer-out
small instance CPU hrs 114000000
Month Storage Xfer CPU total $ total euro total chf total gbp
feb'08 56,623, 1,397, 11,400, 69,421, € 47,486,548
may'08 47,288, 102, 11,400, 58,688, 37,734, 61,532, 29,977,
summary
Storage
CPU
Xfer
Sheet3
*
LCG
(LHC Computing Grid)
JP Navarro <navarro@>
*
LHC - Large Hadronic Collider
GRID Tutorial - How to use LCG
4 experiments:
ATLAS Alice CMS LHCb
27 km long pipe
7+7 TeV
JP Navarro <navarro@>
由欧洲原子能研究中心投资，已投入1亿欧元。专为全球化分布式处理大型强子对撞机产生的每年15PB数据而设计。
*
LCG - LHC Computing Grid
目前集成了33个国家的
140个计算中心。
2008年执行1亿个计
算任务。
JP Navarro <navarro@>
*
内 容 提 纲
JP Navarro <navarro@>
*
网络的迅猛发展
1986 年到2000年 计算机: × 500 网络: × 340,000
*
网络 vs. 计算机性能
处理器速度每18个月翻一番
存储密度
每12个月翻一番
网络速度
每9个月翻一番
1986 to 2000
计算机: x 500
网络: x 340,000
2001 to 2010
计算机: x 60
网络: x 4000
光速每秒30万公里，30公里*80亿/3600＝6600万公里/秒，是光速的200倍以上。
网络发展的必然结果…
*
网络 vs. 计算机性能
处理器速度每18个月翻一番
存储密度
每12个月翻一番
网络速度
每9个月翻一番
1986 to 2000
计算机: x 500
网络: x 340,000
2001 to 2010
计算机: x 60
网络: x 4000
光速每秒30万公里，30公里*80亿/3600＝6600万公里/秒，是光速的200倍以上。
网格计算与云计算的比较
*
云计算符合网格的笼统定义
网格是构筑在互联网上的一组新兴技术，它将高速互联网、高性能计算机、大型数据库、传感器、远程设备等融为一体，为科技人员和普通老百姓提供更多的资源、功能和交互性服务。
Ian Foster, The Grid, 1998
JP Navarro <navarro@>
*
网格计算与云计算理念相似
水坝
油井
电力网
*
内 容 提 纲
JP Navarro <navarro@>
*
反垃圾邮件网格
JP Navarro <navarro@>
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
�
*
上海视频网格
JP Navarro <navarro@>
*
校园网格
JP Navarro <navarro@>
*
电子商务网格
JP Navarro <navarro@>
*
网格书籍
*
JP Navarro <navarro@>
*
*
内 容 提 纲
JP Navarro <navarro@>
*
看看问题解决没有
JP Navarro <navarro@>
*
未来10年的科学
*
未来10年的商业
*
Cloud + Grid Computing
Service Catalog
Computing center
Infrastructure
Virtual Client service
Web Application Service
Compute Service
Database service
Storage service
Content Classification
Storage backup,
archive… service
Job Scheduling
Service
Collaboration Services
Datacenter
Infrastructure
云计算与网格计算将走向融合
JP Navarro <navarro@>
*
*
*
*
*
*
Enter Title of Presentation Here
Google Confidential
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
*
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
*
This Google Map mashup (using ) shows TeraGrid principal investigators for roughly 1,000 allocated projects as of June 2006.
Different icons represent different numbers of Pis per institution.
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
*
DEEP- To ensure that we are harnessing the enormous power of the TeraGrid resources as an integrated system we have the Advanced Support for TeraGrid Applications (ASTA) program that involves our distributed user support team (~20 people across 8 sites, coordinated by GIG). The ASTA program places 1/4 to 1/2 of a support person in a scientific application team for 4 to 12 months, working with that team to exploit TeraGrid capabilities. The ASTA program supports roughly a dozen teams at a time, with a goal of 20-25 teams supported per year.
WIDE- For over 2 decades the NSF high performance computing program, including TeraGrid, has served several thousand users very effectively. However, NSF alone funds tens of thousands of scientists, most of whom have computational requirements that do not frequently require supercomputers. The TeraGrid Science Gateways program is a set of partnerships with discipline-specific teams who are providing computational infrastructure for their science communities. The partnerships involve integrating TeraGrid as a computational and data management “service provider” embedded in the science-community infrastructure. Roughly a dozen web portal style science gateways are part of this program, with more being added continually. In addition, this program includes a peer-grid interoperation effort with Open Science Grid and a desktop application effort that is leveraging an NIH-funded biomedical project directed by Rick Stevens at the University of Chicago.
OPEN- TeraGrid began as an infrastructure involving four partner sites, grew to nine sites, and is currently organized as a set of eight resource providers and a central “grid infrastructure group” (GIG) providing central services and support as well as management and operations. This new structure allows TeraGrid to grow to dozens of resource provider sites, making it an “open” partnership. In addition, TeraGrid architecture is service-oriented, stressing open source standards such as are deployed with key software including the Globus Toolkit GT4, Condor, and other tools.
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
*
TeraGrid leaders created the Science Gateways concept and initiative as a way to adapt the TeraGrid to the environments that users have adopted (. web portals), leveraging the work of these science communities in designing and building their infrastructure. This approach also leverages education efforts of these communities, and allows for an order of magnitude increase in the number of users served by TeraGrid. This increase is possible by changing the model of provision of resrouces and support, where the TeraGrid serves gateways as a “wholesale” supplier, working with the gateway providers who serve as the “retailers” providing tailored support to communities of users.
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
*
NSF TeraGrid Review
January 10, 2006
Charlie Catlett (cec@)
*
Close collaboration with EGEE/WLCG.
5 DOE Labs - BNL, Fermilab, NERSC, ORNL, SLAC. 65 Universities. 5 partner campus/regional grids.
>43,000 cores, 6 Petabyes disk cache, 10 Petabyes tape stores
15,000 CPU WallClock days/day. 1 petabyte data distributed/month.
100,000 application jobs/day. 20% cycles through resource sharing, opportunistic use.
*
*
*
*
*
*
*
*
*
*
*
由欧洲原子能研究中心投资，已投入1亿欧元。专为全球化分布式处理大型强子对撞机产生的每年15PB数据而设计。
*
*
*
*
网络 vs. 计算机性能
处理器速度每18个月翻一番
存储密度
每12个月翻一番
网络速度
每9个月翻一番
1986 to 2000
计算机: x 500
网络: x 340,000
2001 to 2010
计算机: x 60
网络: x 4000
光速每秒30万公里，30公里*80亿/3600＝6600万公里/秒，是光速的200倍以上。
*
网络 vs. 计算机性能
处理器速度每18个月翻一番
存储密度
每12个月翻一番
网络速度
每9个月翻一番
1986 to 2000
计算机: x 500
网络: x 340,000
2001 to 2010
计算机: x 60
网络: x 4000
光速每秒30万公里，30公里*80亿/3600＝6600万公里/秒，是光速的200倍以上。
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
