Dear readers, I’m a little bit late posting this information, but I have been up to my eyes with work!. Few weeks ago I attended the European Spark Summit held in Amsterdam. It was a wonderful opportunity to meet and talk with the spark community. The summit started with Matei Zaharia, who talked about how spark usage is evolving and Ion Stoica talking about how Spark is entering into enterprises. There were a lot of interesting presentations. You can download them and see the recorded video presentations on this web site. BSC also gave a presentation where we presented what we are doing with Spark in our research supercomputing center in Barcelona. Below you can find the slides that we presented. Attending the summit also gave us the opportunity [...]
Les animo a leer un gran libro, El maestro, que acaba de salir publicado, de un gran escritor como es Màrius Mollà. Aunque se trate de una novela ambientada en el siglo pasado, el tema nos invita a una reflexión muy actual y a mi parecer de máxima importancia en los momentos en que vivimos. Comparto con el autor que la transformación del sistema educativo es la base de todo cambio social y económico. ¡Espero que lo disfruten!
Spark 1.2 is out! More than 172 contributions! Check it at http://spark.apache.org/news/spark-1-2-0-released.html. Yesterday Databricks announced this new in his blog. Congratulations! This release contains more than 1000 patches from 172 contributors. Spark 1.2 introduces many new features along with scalability, usability and performance improvements: Optimizations in Spark’s core engine: Spark 1.2 includes several cross-cutting optimizations focused on performance for large scale workloads. Two new features Databricks developed for their world record petabyte sort with Spark are turned on by default in Spark 1.2. The first is a re-architected network transfer subsystem that exploits Netty 4’s zero-copy IO and off heap buffer management. The second is Spark’s sort based shuffle implementation. They say that these features give as much as 5X performance improvement for workloads with very large shuffles. Spark SQL [...]
In a previous post we introduced Spark, a framework that will play an important role in the Big Data area. You can find a good starting point to understand what is Spark following this page from DataBricks, however let me reproduce an overview in this post. Spark runs on top of existing Hadoop clusters to provide enhanced and additional functionality. Although Hadoop is effective for storing vast amounts of data cheaply, the computations it enables with MapReduce are highly limited. MapReduce is only able to execute simple computations and uses a high-latency batch model. Spark provides a more general and powerful alternative to Hadoop's MapReduce, offering rich functionality such as stream processing, machine learning, and graph computations. Spark provides out of the box support for deploying within an existing Hadoop [...]
Esta semana he hecho una visita al supercomputador Marenostrum 3 con los estudiantes de uno de los cursos que imparto este cuatrimestre (que actualmente ocupa el lugar 34 del ranquin mundial con 1 Petaflop/s , siendo el más potente de España y el número once de Europa). Cómo se renovó recientemente el supercomputador he tenido que preparar nuevamente mis notas con los datos principales. A veces algunos me piden estos datos, que sinceramente no me los sé de memoria, y por ello he pensado que estaría bien compartir las transparencias que he usado en el curso (que son básicamente las de Javier Bartolomé, BSC system head).
Invited presentation at e-InfraNet Green ICT Workshop in Helsinki (27th – 28th October 2010). This eInfranet workshop addressed some of the most original EU projects and initiatives in the field and it also reviewed the EU green strategies and policies. Workshop on Green and Environmental Computing - Helsinki from Barcelona Tech UPC - Barcelona Supercomputer Center BSC
“A converted chapel in Barcelona is the home for a supercomputing centre that is helping to revolutionise the way we will use cloud computing …“ in FUTURES Magazine Issue n.6 June 2010 Read the full article at MicrosoftMagazineFUTURESpp20-22 . (full issue). linkMicrosoftMagazineFUTURESpp20-22-portada
(english version) Debate: The ICT, a new opportunity Smart technology for a sustainable Europe Europe is worried about its sustainable growth. The traffic jams in the principal European cities have an estimated cost of 135 million Euros per year, which means that the car drivers spend a total of five days a year waiting in their cars. Around 40% of energy produced is wasted by the time it arrives to the consumer due to its inefficient management. At the moment, we are wasting energy and emitting carbon dioxide (CO2) into the atmosphere. The difficulty, now, is that if Europe can accomplish its ambitious goals to combat against climate change. The diagnosis is clear and the solution does not mean to increase the infrastructures, but to maintain the [...]
BSC at “Disruptive Solutions for Energy Efficient ICT” European Comission experts consultation meeting
Some orders of magnitude separate the energy efficiency of conventional computers from what is theoretically possible. Closing this gap would lead to a significant improvement in the energy efficiency of information and communication technology (ICT). Experts from a variety of disciplines held wide-ranging discussions about how to achieve this goal at the Disruptive Solutions for Energy Efficient ICT meeting in Brussels on 8-9 February 2010. This meeting brought together a diverse group of experts to brainstorm and elaborate where and in what ways FET(*) research could improve energy efficiency in ICT. The goal was to look for disruptive ideas with a high potential pay-off that cannot be reached with traditional funding instruments. The direct link to the report is http://cordis.europa.eu/fp7/ict/fet-proactive/docs/shapefetip-wp2011-12-10_en.pdf List of Participants Mats Brorsson, KTH School [...]