The training course is read for undergraduates of the 1st year of study.
<iframe src="https://www.youtube.com/embed/videoseries?list=PLGESve048UzTbAqDil1EfsU4Lre3B09bt" width="640" height="360" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen>Big Data Technologies</iframe>
The course includes the following topics:
<ul>
<li>Hadoop</li>
<li>Spark</li>
<li>Ignite</li>
<li>Kafka</li>
<li>NoSQL</li>
<li>Docker</li>
<li>Kubernetes</li>
</ul>
<img src="/content/courses/big-data-course-qr.png" alt="image">

Big Data Technology

The course is taught for bachelors of 3-4 years of study.
<h2>About the course</h2>
The course is dedicated to the practical study of modern DevOps technologies. You will master the basic knowledge and skills of building pipelines, automating software development and operation processes, containerizing applications, deploying continuous code integration systems, building application assembly pipelines, deploying container orchestration systems, building application deployment processes on the execution platform, creating infrastructure and application monitoring systems, collecting and analyzing application logs.
<h2>You will learn</h2>
<ul>
<li>deploy continuous code integration systems</li>
<li>collect and analyze application logs</li>
<li>deploy container orchestration systems</li>
<li>create infrastructure and application monitoring systems</li>
</ul>
<h2>Topics</h2>
<ol>
<li>
Fundamentals of application containerization technology
The specifics of deploying distributed high-performance applications. Introduction to cloud services (IaaS, PaaS, SaaS). DevOps philosophy, continuous integration and code delivery. Version control system. Git. Typical operations. Repository of software code artifacts. Docker technology. Delimiting process spaces in the operating system using cgroups. Technology composition: docker files, image registry, containers, Docker daemon. File access layers. A cluster of Docker hosts. Creating a cluster. Scaling of containers. Preparing Debian-based virtual machines. Writing Dockerfile and docker-compose.yaml for local launch of React-mobi application with node.a js backend and a PostgreSQL database.
</li>
<li>
Installation and configuration of GitLabCI
Purpose and component modules of the GitLab CI assembly automation system. The format of the project build descriptor in the file .gitlab-ci.yml. Ways to scale GitLabCI to speed up the build and testing process. Installing and configuring GitLabCI in a docker-compose environment. Deploying the registry of GitLabCI Docker images.
</li>
<li>
Building applications with GitLabCI
Components of the GitLab CI pipeline, a system of stages in the GitLabCI pipeline. The pipeline starts according to the schedule. Publishing artifacts of the assembly line operation. Handling build errors. Writing.gitlab-ci.yml for testing, building, publishing a Docker image to the GitLabCI registry and local deployment of the project.
</li>
<li>
Kubernetes Orchestration System
Orchestration of large-scale application services. Problems and challenges. The composition of the Kubernetes cluster. The main elements of the deployment (Pod, Service, Route). Kybernetes cluster deployment options (mini cube, k8s, k3s, k8s managed). Managing a Kubernetes cluster via kubectl and Kubernetes Dashboard. Deploy the k3s and Kubernetes Dashboard cluster in a prepared Debian-based virtual machine.
</li>
<li>
Deploying applications to Kubernetes using GitLabCI
Automatic deployment of the application as a result of a successful build, deployment outlines. Create application deployment reports. Creating yaml deployment descriptors to the k3s cluster. Creation of elements for consolidation of deployment modules (Service) and routing (Ingress). Passing parameters via Config Map. Creating a GitLabCI pipeline for deploying an application in a Kubernetes cluster using the created descriptors.
</li>
<li>
Helm Deployment Template Engine
The release of applications, the composition of the release, the order of deployment of releases along the contours of operation. The purpose and use of the Helm template engine. Installing Helm. Creating Helm charts. The structure of Helm charts. Setting parameters (values) for the Helm chart configuration. Configuring the installation and updating of releases using Helm. Rollback a release using Helm.
Creating a GitLab CI pipeline for deploying an application in a Kubernetes cluster using helm artifacts.
</li>
<li>
Prometheus and Grafana monitoring technologies
Purpose and composition of the Prometheus-based metrics collection system. The subsystem for sending Prometheus warnings. Long-term storage of metrics, the use of PromQL for the analysis and aggregation of metrics in Prometheus. Grafana's capabilities as a Prometheus metrics visualization environment. Scaling up the monitoring infrastructure. Setting up monitoring of the operation infrastructure and applications using Prometheus and Grafana. Deployment of Prometheus and Grafana servers, deployment of node exporter to send metrics of the infrastructure (cluster nodes) to Prometheus. Creating a Grafana Dashboard. Collecting application performance metrics. Creating a GitLab CI pipeline for deploying an application in a Kubernetes cluster using helm artifacts.
</li>
<li>
Centralized collection of application logs
The purpose of centralized logging systems in cluster systems. The composition of centralized logging systems. Kafka message broker in centralized logging systems. Collecting standard container output using beat utilities. Logstash as a system for polling, filtering and transmitting messages from source to receiver. OpenSearch full-text search system as a log analysis tool. Deployment of modules of the centralized log collection system based on Logstash and OpenSearch technologies.
</li>
</ol>

DevOps Technologies

The training course is read for undergraduates of the 2nd year of study.
<iframe src="https://www.youtube.com/embed/videoseries?list=PLGESve048UzTYWps-8JjTUDThnGHOflg2" width="640" height="360" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen>Hybrid supercomputing technologies</iframe>
The course includes the following topics:
<ul>
<li>MPI</li>
<li>OpenMP</li>
<li>CUDA</li>
</ul>
<img src="/content/courses/hst-course-qr.png" alt="image">

Hybrid supercomputing technologies

Decentralized data storage systems allow consumers to access their data and perform
autonomous control over it. Fluid Storage benefits from any centralized solution
in a situation of increasing insurance sanctions and blockages from regional operators,
since it does not have a central storage.
Popularity of decentralized cloud storage is growing rapidly as a new generation of
platforms increases the flexibility and security of cloud storage, while reducing
costs for end users. Some decentralized storage networks seek to replace existing
cloud storage giants, while others seek to work together with them, expanding and
improving their capabilities.
Compared to its closest competitor BTFS, Fluid Storage uses more advanced replication
and routing technologies to reduce file access time and storage costs.

Fluid Storage - Decentralized storage system

<h2>Purpose</h2>
KubeSmart is designed for PaaS.
Key users are cloud service providers, large companies with large-scale Kubernetes infrastructure.
It is a Kubernetes add-on tool in the form of K8S Custom Resource Definition (CRD)
Analyzes the interaction of microservices with each other and redeploys them in such a way as to achieve the best characteristics.
<h2>Benefits</h2>
Advanced monitoring of the current state of nodes and microservices
Definition of microservice relationships
Visual display of interaction graphs for the user
Dynamic management of container placement on nodes to optimize resources and increase reliability.
The result is a reduction in hosting costs for users and hosting providers by 10-15%.
<h2>Relevance</h2>
Most modern cloud Platform as a Service (PaaS) systems are built on the basis of Docker application containerization technology. In this case, the server application (microservice) is “packed” together with all its dependencies into a Docker container, which simplifies its transfer from server to server.
The servers operate in cluster mode under the control of the container orchestration system – Kubernetes (K8S). The K8S cluster includes a scheduler that distributes containers across cluster nodes (servers).
The schedulers currently in use do not have intelligent algorithms for placing containers across nodes. They operate on the principle of placing in the “first suitable slot”.
KubeSmart takes into account the interaction pattern of modules placed in the cloud with each other to identify their interrelations and more effectively plan the use of cloud hardware resources using machine learning methods.
<h2>Technology</h2>
Predlfuftvsq KubeSmart approach is applicable to microservice systems, or distributed systems of streaming and batch computing, in which the locality of interaction of distributed processing components (microservices, executors, etc.) with each other plays a significant role.
<img src="/content/projects/kube-smart/1.png" alt="KubeSmart-1">
The figure shows an example of using an approach based on ML planning, as applied to microservice systems. Before starting the ML scheduler, microservices are distributed across nodes in an arbitrary manner. The ML scheduler analyzes the state of the cluster nodes and the microservices placed on them and proposes a new placement scheme. A variety of characteristics can be selected as optimization criteria: from the uniformity of temperature distribution inside the data center to the total throughput of all microservices.
The first version of KubeSmart proposes a method for improving the response time of a gateway microservice (a service that is accessed by an external user with a request, generating the execution of a chain of requests between microservices within a cluster). In this case, the ML scheduler determines which microservices most often interact with each other (form a call chain) and places them, if possible, close to each other on the same cluster nodes.
<img src="/content/projects/kube-smart/2.png" alt="KubeSmart-2">
The clustering model is packaged as a Kubernetes CronJob and deployed on an experimental cluster of 10 k3s ARM nodes (shown in the figure).
<img src="/content/projects/kube-smart/3.png" alt="KubeSmart-3">
Data collection is configured using the coroot eBPF framework, which collects performance metrics in Prometheus. The model runs periodically, reads metrics about the frequency of interactions between microservices within a certain sliding window, clusters microservices and writes data about the microservice belonging to a certain cluster to PostgreSQL.
The scheduler is developed in Go as a plugin for the main Kubernetes scheduler. It reads information about clusters from the database and builds a target placement picture for a specific Pod being placed. The decision is made based on how many microservices, also belonging to this cluster, are already placed on a certain node. Also, to make the system more “flexible”, a descheduler was developed that removes the Pod that deviated the most from the target placement scheme, thereby initiating its re-placement.
<img src="/content/projects/kube-smart/4.png" alt="KubeSmart-4">
Using smart container placement scheduling algorithms allows achieving a performance increase of at least 10% compared to the default Kubernetes scheduling.
<h2>Technology readiness level</h2>
TRL 4: A detailed mockup of the solution has been developed to demonstrate the functionality of the technology

Research projects

Fluid Storage - Decentralized storage system

KubeSmart - Intelligent PaaS system for Docker-container orchestration

Educational projects

Big Data Technology

DevOps Technologies

Hybrid supercomputing technologies

Address

Contacts