Although our focus on Baidu's work this year is focused on the Chinese search giant's initiatives in deep learning, many other key, though not so cutting-edge applications, present the challenges of big data.
As Baidu's Ouyang Jian talks about at this week's Hot Chips conference, Baidu sits on more than 1 EB of data, processes about 100 PB of data per day, updates 10 billion pages per day, and updates more than 1 PB of log updates every 24 hours. These numbers are indistinguishable from Google, as one might imagine. Baidu has adopted a Google-like approach to solve potential bottlenecks on a large scale.
As we just talked about, Google is looking for every possible way to beat Moore's Law, Baidu is doing the same exploration, and the exciting and fascinating machine learning work is fascinating, the core mission of the business is accelerating. The same is true, because it must be. Ouyang mentioned that the gap between the company's demand for high-end services based on its own data and the capabilities that the CPU can carry will gradually increase.
For Baidu’s tens of billions of questions, at the receiving end of all data is a series of frameworks and platforms for data analysis, from the company’s massive knowledge maps, multimedia tools, natural language processing frameworks, recommendation engines, and clickstream analysis. It’s all like this. In short, the primary problem with big data is this: a range of applications and matching overwhelming data.
When it comes to accelerating Baidu's big data analysis, several challenges are faced. It is difficult for Ouyang to talk about the core of abstract computing to find a universal method. "The diversity of big data applications and the changing types of computing make this a challenge. It's difficult to integrate all of this into a distributed system because of the changing platform and programming model (MapReduce, Spark, streaming, user defined, Etc.) There will be more data types and storage formats in the future."
Despite these obstacles, Ouyang talked about the common clues that their team found (between them). As he pointed out, the traditional SQL that links many of their data-intensive tasks together. “About 40% of our data analysis tasks are written in SQL, and other SQL rewrites are also available.†Further, he said that they can enjoy the benefits of existing SQL systems and can Matches existing frameworks such as Hive, Spark SQL, and Impala. The next thing to do is SQL query acceleration, Baidu found that FPGA is the best hardware.
These boards, called processing units (PE in the figure below), automatically handle key SQL functions when executing SQL. All of the things mentioned here are from speeches, and we are not responsible. To be exact, the FPGA mentioned here is a bit mysterious, perhaps deliberately. If Baidu gets an upgrade in the benchmark as shown below, then this is a competitive message. We will continue to introduce what is described here later. Simply put, the FPGA runs in the database, and when it receives the SQL query, the software designed by the team is tightly integrated.
Ouyang mentioned one thing, their accelerator is limited by the bandwidth of the FPGA, otherwise the performance can be higher. In the following evaluation, Baidu installed 2 12-core, 2.0 GHz intl E26230 CPU, running at 128G memory. SDA has 5 processing units (300MHz FPGA boards in the above figure) each handling different core functions (filter filter, sort sort, aggregate aggregate, joint join and group by)
In order to achieve SQL query acceleration, Baidu conducted a benchmark test for TPC-DS and created a special engine called a processing unit (PE) to accelerate five key functions in benchmarking, including filtering filters, sorting Sort, aggregate aggregate, join join, and group by group, (we didn't capitalize these words like SQL). SDA devices use an offload model. Accelerator cards with multiple different types of processing units make up the logic in the FPGA. The type of SQL function and the number of cards per card are determined by the specific workload. Since these queries are executed in Baidu's system, the data used for querying is pushed to the accelerator card in a column format (this makes the query very fast), and through a unified SDA API and driver, SQL query work is distributed to The correct processing unit and SQL operations are accelerated.
The SDA architecture uses a data flow model. Operations that are not supported by the acceleration unit are returned to the database system and then run locally there. The performance of the SQL accelerator card developed by Baidu is limited by the memory bandwidth of the FPGA card. Accelerator cards work across the entire cluster machine, by the way, but the exact principles of how data and SQL operations are distributed to multiple machines are not disclosed by Baidu.
We are limited to the details that Baidu is willing to disclose, but these benchmark results are very encouraging, especially for Terasort. We will follow Baidu’s footsteps after the Hot Chips conference to see if we can get information about how this is. The details of how to connect together and how to resolve the memory bandwidth bottleneck.
Ethernet Slip Ring is a device that allows Ethernet signals to be transmitted through a rotating assembly. This device is typically used in applications where it is not possible or practical to run Ethernet cables, such as in an application where the cables would have to travel through a hazardous area. It converts the regular Ethernet signals into rotary signals, which are then transmitted through the rotating assembly.
There is a growing trend in the industry for Ethernet slip rings. Ethernet slip rings are currently being used to interconnect computer networks. A slip ring is a transmission device that uses a counter-rotating encircling belt to transfer power and data between two or more devices.
Senripgrid Conductive Slip Ring has the following characteristics Direct unplugging of the RJ45 connector Upon reaching eight simultaneous ports, follow the standard cable connection method Work with either copper or gas-liquor joints Transmission to the external network (10M, 100M, 1Km Base T).
Ethernet Slip Ring,Generator Collector Rings,Through Bore Rotary Union,Rotary Pressure Joint
Dongguan Oubaibo Technology Co., Ltd. , https://www.sliprobs.com