Commit 7d4e1bda authored by noureen.taj's avatar noureen.taj 🖖🏻

Update assignment1.md

parent 9420a93e
# Assignment 1
Hey there! Welcome to Knowledge Lens Intern Training Program.
This Assignment will serve as a quick refresher on the usage of NoSQL and Time-series databases.
There are three tasks in this assignment, on completion of which you'll learn:
* How to interact with Mongo DB
* Using Pandas Dataframe and generating your own excel reports
* Leveraging Kairos Time-series database for data ingestion and querying the same
* Publishing and Consuming messages via MQTT protocol
* Caching mechanism using Redis DB
Happy Coding! :tada:
## Task 1: Working with Mongo - Advanced
## :pushpin: Task 1: Working with Mongo - Advanced
## Areas covered:
### :golf: Areas covered:
- Timeseries Operation
- Working with NoSQL
- Working with Pandas
## Description:
### :books: Description:
You are given with a dataset of an inspection company in the form of a JSON file. The end goal of the project is to create an API interface that will provide the following:
1. Business name with maximum number of "`Violation Issued`".
......@@ -38,31 +48,32 @@ Sample Document:
Bonus Points: Use Mongo Aggregate framework
### Tools to use:
### :wrench: Tools to use:
1. Pycharm / VSCode
2. Robo3T / Studio3T / MongoDB Compass
3. PyMongo
### Reference:
https://www.mongodb.com/docs/manual/tutorial/query-documents/
https://www.mongodb.com/docs/manual/reference/operator/aggregation-pipeline/
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_excel.html
https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse
https://pymongo.readthedocs.io/en/stable/
### :mag: References:
* [Querying Documents on Mongo](https://www.mongodb.com/docs/manual/tutorial/query-documents/)
* [Quick Summary on Mongo Aggregation Stages](https://www.mongodb.com/docs/manual/reference/operator/aggregation-pipeline/)
* [Generating Excel Sheets from a Pandas Dataframe](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_excel.html)
* [How to return files on FastAPI response](https://fastapi.tiangolo.com/advanced/custom-response/#fileresponse)
* [PyMongo Official Documentation](https://pymongo.readthedocs.io/en/stable/)
_________________________________
## Task 2: Working with Timeseries
## :pushpin: Task 2: Working with Timeseries
## Areas covered:
### :golf: Areas covered:
- Timeseries Operation
- Working with Timeseries
- Working with Pandas
## Description:
### :books: Description:
You are given with a dataset of weather in the form of a CSV file. The end goal of the project is to create an API interface that will provide the following:
1. Get daily, monthly, weekly and monthly aggregate (min, max, and average) of the data and generate report in Excel format.
......@@ -74,20 +85,24 @@ Sample Document:
|10/03/2004 |18:00:00 |2.6 |1360 |150 |11.9 |1046 |166 |1056 |113 |1692 |1268 |13.6|48.9|0.7578|
### Tools to use:
### :wrench: Tools to use:
1. Pycharm / VSCode
2. Pandas
3. Kairos
### Reference:
https://kairosdb.github.io/docs/restapi/QueryMetrics.html
https://pypi.org/project/kairosdb-python/
### :mag: References:
* [How to query Kairos DB using Metrics](https://kairosdb.github.io/docs/restapi/QueryMetrics.html)
------------------------------------------------------
## Task 3: Working with MQTT & REDIS
## :pushpin: Task 3: Working with MQTT & REDIS
### :golf: Areas covered:
- MQTT Protocol
- Caching using Redis DB
# Description
### :books: Description
Hospital with 'N' doctor's on duty,
Receptionist need's to assign new patient's in round robin order.
......@@ -108,18 +123,15 @@ Patient details sample format:
Use Redis for caching information
create a snippet which produces patient information to MQTT
Create a snippet which produces patient information to MQTT.
create consumer's which consumes data from these topics and store to a redis db
Create consumer's which consumes data from these topics and store to a redis db
### Tools to use:
### :wrench: Tools to use:
1. Pycharm / VSCode
2. MQTT - (paho-mqtt)
3. REDIS - (redis)
### Reference:
https://www.emqx.com/en/blog/how-to-use-mqtt-in-python
https://docs.redis.com/latest/rs/references/client_references/client_python/
2. MQTT - (PIP package: `paho-mqtt`)
3. REDIS - (PIP package: `redis`)
### :mag: References:
* [Using MQTT in Python](https://www.emqx.com/en/blog/how-to-use-mqtt-in-python)
* [Connecting to Redis in Python](https://docs.redis.com/latest/rs/references/client_references/client_python/)
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment