gar-oracledba

  Table of Contents Redpanda Demo Project Architecture Session Terminal 1 (Preparations) Session Terminal 2 (Kafka Producer) Session Terminal 3 (Kafka Consumer) Check (Monitor) Output Overview This repository contains some homework solutions from module 6 (Streaming) in DTC Data Engineering Zoomcamp 2024. Instead of Kafka, here will use Red Panda , which is a drop-in replacement for Kafka. Ensure we have the following set up : Docker (module 1) PySpark (module 5) For this homework we will be using the files from Module 5 homework i.e. : Green 2019-10 data from here Note: Don't run these all steps on Jupyter Notebook. Otherwise the ipynb script file will grow very quickly when running the producer and consumer steps. Please run on terminal. All scripts created using pyhton (.py) extention Redpanda Streaming Demo Architecture Session Terminal 1 (Preparations) Open Operating System terminal Creat

  DE Zoomcamp 2024 – Project1 This repository contains a brief description of my DE Zoomcamp 2024 Project 1 Problem statement The Retailrocket has collected a large dataset of E-commerce i.e a file with behaviour data (events.csv), a file with item properties (item_properties.сsv) and a file, which describes category tree (category_tree.сsv). The data has been collected from a real-world ecommerce website. It is raw data, i.e. without any content transformations, however, all values are hashed due to confidential issues. The purpose of publishing is to motivate researches in the field of recommender systems with implicit feedback. The goal of this project is to create a streamlined and efficient process for ingesting and analyzing e-commerce on Cloud by implementing Data Engineering concepts. About the Dataset Retailrocket recommender system The dataset consists of three context files i.e. : a file with behaviour data (events.csv) a file, which describes category tree (category_tree.сs