HPC 软件栈与数据分析软件栈对比

HPC 软件栈与数据分析软件栈对比

HPC 软件栈

+------------------------------------------------------------+ Application | Applications and community Codes | Level +------------------------------------------------------------+

MiddleWare +--------------------------------+ +------+ Management | FORTRAN, C, C++, IDE | | | | | | | +--------------------------------+ | | +--------------------------------+ |SCI.V | | Domain Specific library | |IS. | | | | | +--------------------------------+ | | +----------------+ +---------------+ +--------------+| | |MPI/OPENMP +Acce| | Numercial | |Programming/ || | |lerator tools | |Libraries | |Debugging(eg. || | | | | | |PAPI) || | +----------------+ +---------------+ +--------------+| | +----------------+ +---------------+ +--------------+| | |Lustre(Parallel | | Batch Schedu | | System Monit || | |File System) | + ler(eg. SLURM)| | or Tools || | | | | | | || | +----------------+ +---------------+ +--------------++------+ ----------------------------------------------------------------------------------------- System +---------------------------------------------------+ Software | Containers (Singularity, Shifter, etc.) | | | +---------------------------------------------------+ +---------------------------------------------------+ | Linux OS Variant | | | +---------------------------------------------------+ ----------------------------------------------------------------------------------------- Cluster +---------------+ +----------+ +----------+ +-------+ Hardware | Infiniband + | |Local + | | X86 Rack+| |In-Situ| | Eth Switches | | SAN | |GPU or .. | |Proce. | +---------------+ +----------+ +----------+ +-------+

数据分析软件栈

+------------------------------------------------------------+ Application | Manhout R and Applications | Level +------------------------------------------------------------+

MiddleWare+------+ +------------+ +------------+ +--------+ +------+ Management| | | HIVE | | PIG | |FLUME | | | | | | | | | | | | | | | +------------+ +------------+ +--------+ | | | | |Zooke | | | +------------+ +------------+ +------------------+ |eper( | | cloud| |MAPREDUCE | | STORM | | SQOOP | |coordi| |serv | | | | | | | |nation| | | +------------+ +------------+ +------------------+ | | | | +---------------------------------------------------+| | | | | YARN || | | | +---------------------------------------------------+| | | | +---------------------------------------------------+| | | | | HBASE Bigtable(KV Store) || | | | +---------------------------------------------------+ | | +---------------------------------------------------+| | | | | HDFS || | | | +---------------------------------------------------++------+ --------------------------------------------------------------------------------------- System | | +---------------------------------------------------+ Software | | | Virtual Machine and CLoud Service Container | | | | | | | +---------------------------------------------------+ | | +---------------------------------------------------+ | | | Linux OS Variant | +------+ | | +---------------------------------------------------+ ----------------------------------------------------------------------------------------- Cluster +---------------+ +--------------+ +-----------------+ Hardware | Ethenet | | Local Storage| | X86 Racks | | Switches | | | | | +---------------+ +--------------+ +-----------------+

资源管理诉求差异

数据格式与文件系统差异

用户面差异

数据挖掘

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。