Big Data Analytics A.A. 2020/21
WARNING: All lectures of the First Semester of the academic year 2020/21, until 31/12/2020, will be provided exclusively remotely, through the Teams team named “599AA 20/21 - BIG DATA ANALYTICS [WDS-LM]” (https://bit.ly/35yJ65c).
ATTENZIONE: Tutte le lezioni frontali del Primo Semestre dell’a.a. 2020/21, fino al 31/12/2020, verranno erogate esclusivamente in modalità a distanza, attraverso il canale Teams “599AA 20/21 - BIG DATA ANALYTICS [WDS-LM]” (https://bit.ly/35yJ65c).
Instructors - Docenti:
Timetable (http://bit.ly/unipi_timetable_2020)
Team Registration: build up teams of 3 or 4 students and register your team here, by September 27th: https://forms.gle/rbsV4dF6RuAnCBWz9
For students without a team: send an email to Luca Pappalardo to notify that you are without a team by September 30th.
Only for the registered teams, express your preference for the datasets by September 30th https://forms.gle/HVheaScCgQJw4o616
Dataset assignment: at thie following link, each team can find the dataset assigned for the project –> https://bit.ly/33eTfC9
Instructions for mid term 1: The first mid term presentation (data understanding and project proposal) will be on October 19th (BigProblem, Global, MMG, I TeamIDI) and October 20th (Bei Dati Acrobatici, Malucs, AMS Group).
presentation: prepare a presentation describing the data understanding and a proposal of the problem you want to solve. Motivate your decisions and choices (e.g., which variables you delete, how you deal with missing values and noise, the new variables you created, if you integrated your data with external datasets, etc.). The presentation should last 20 minutes (+ 10 minutes questions). The presentation must be sent through the google form (see below) in pdf format;
report: the report must be done in latex, using this template:
latex_template_bda.zip. It must be a maximum of 5 pages long. Summarize the data understanding and describe and motivate your project proposal. A zipped folder (.zip file) containing the .tex file, the .cls file, the .pdf file, and the files of all figures must be sent through the google form. In the report, put the name of the title of your project and the names of the members of your team.
code: the python code in .ipynb format (Jupyter Notebook) or .py format used to generate the computations and the plots must be sent through the google form. Please document adequately your notebooks using the markdown language.
-
name the files using the format midterm1_teamname_type, where teamname is the name of the team (do not use spaces, use lowercase only), type is the type of the file (i.e., presentation, report, or code). Examples: midterm1_iteamidi_presentation.pdf, midterm1_beidatiacrobatici_report.zip, midterm1_amsgroup_code.ipynb
Instructions for mid term 2: The second mid term presentation (model(s) implementation and evaluation) will be on November 16th (BigProblem, Global, MMG, I TeamIDI) and November 17th (Bei Dati Acrobatici, Malucs, AMS Group).
presentation: prepare a presentation describing the models you tried (e.g., Decision Trees, SVMs, etc.), the baselines used as a comparison (e.g., DummyClassifiers), how you perform the hyper-parameter tuning, the evaluation technique used (i.e., holdout, repeated holdout, cross validation), and the metrics chosen to evaluate the performance of the models. Motivate your decisions and choices (e.g., which evaluation metrics you chose, how you deal with the unbalancing of the dataset). The presentation should last 20 minutes (+ 10 minutes questions). The presentation must be sent through the google form (see below) in pdf format;
report: the report must be done in latex, using the same template as the first mid term:
latex_template_bda.zip. It must be a maximum of 10 pages long, including the part of the report regarding mid term 1. Summarize the model construction and evaluation and motivate your choices. A zipped folder (.zip file) containing the .tex file, the .cls file, the .pdf file, and the files of all figures must be sent through the google form. In the report, put the name of the title of your project and the names of the members of your team.
code: the python code in .ipynb format (Jupyter Notebook) or .py format used to generate the computations and the plots must be sent through the google form. Please document adequately your notebooks using the markdown language.
-
name the files using the format midterm2_teamname_type, where teamname is the name of the team (do not use spaces, use lowercase only), type is the type of the file (i.e., presentation, report, or code). Examples: midterm2_iteamidi_presentation.pdf, midterm2_beidatiacrobatici_report.zip, midterm2_amsgroup_code.ipynb
Paper presentation:
each student will present, during a talk of most 7 minutes, a paper on Big Data Analytics. The presentations of the papers are scheduled on November 23rd and 24th. The presentation should last 7 minutes (+ 3 minutes questions).
Express your preference for 5 papers here:
https://forms.gle/B9rCmpJ8jnQ4vzWN8. We'll take into account your preference as much as possible.
Fill the form by Nov 3rd. I'll assign you the paper within Nov 5th.
During the presentation (with slides) you should highlight the following aspects: the data set used, the feature engineering and/or selection (if any), the problem addressed, the models/algorithms used to solve the problem, and finally the explanations of the model constructed (if any).
-
Instructions for mid term 3: The third mid term presentation (model(s) interpretation and explanation) will be on December 7th (BigProblem, Global, MMG, I TeamIDI) and December 8th (Bei Dati Acrobatici, Malucs, AMS Group).
presentation: prepare a presentation in which you show how to interpret the model(s) you created and how to explain the reasoning the model.Example: if you use a decision tree (or similar) you can show the feature importance and show its structure to describe the rules it is composed of; if you use a (logistic or linear) regressor you can show the value of the computed weights; if you use a geometric model you can perform a dimensionality reduction and try to interpret it in two or three dimensions; if you use a non-interpretable model (e.g., a neural network) you can use explanability models (e.g., Shap, LIME). Provide also examples of how to interpret the model on specific records that are correctly classified and records that are incorrectly classified by your model. The presentation should last 20 minutes (+ 10 minutes questions). The presentation must be sent through the google form (see below) in pdf format;
report: the report must be done in latex, using the template as the other mid terms. It must be a maximum of 15 pages long. Extend/modify the previous report. A zipped folder (.zip file) containing the .tex file, the .cls file and the files of all figures/plots must be sent through the google form. In the report, put the name of the title of your project and the names of the members of your team.
code: the python code in .ipynb format (Jupyter Notebook) or .py format used to generate the computations and the plots must be sent through the google form. Please document adequately your notebooks using the markdown language.
-
name the files using the format midterm3_teamname_type, where teamname is the name of the team (do not use spaces, use lowercase only), type is the type of the file (i.e., presentation, report, or code). Examples: midterm3_iteamidi_presentation.pdf, midterm3_beidatiacrobatici_report.zip, midterm3_amsgroup_code.ipynb
Examples of projects from past years:
Instructions for exam:
remember to register in time to the first appello (Jan 14th) or the second appello (Feb 4th);
by the day before your appello (i.e., Jan 13th or Feb 3rd), upload the python code that will be used during the exam (in .ipynb format or .py format). Name the file as exam_teamname_code, where teamname is the name of your team;
during the exam, we'll provide you with a .txt file (using the same separator as the one in your dataset) with some records (with the corresponding labels) extracted from the dataset assigned to your team. You must run the code to generate predictions and explanations using your model (this means that the model should be already trained). Your code must also show which predictions are correct and which are not.
If you made some improvements or changes on your project, describe them in the report and upload it. Name the file exam_teamname_report, where teamname is the name of your team.
-
There will two exam sessions on Feb 14th: 10am - 12pm and 4pm - 6pm. Groups can connect to the session they prefer. We'll use the same Teams channel used for the lessons.
Learning goals
In our digital society, every human activity is mediated by information technologies, hence leaving digital traces behind. These massive traces are stored in some, public or private, repository: phone call records, movement trajectories, soccer-logs and social media records are all examples of “Big Data”, a novel and powerful “social microscope” to understand the complexity of our societies. The analysis of big data sources is a complex task, involving the knowledge of several technological and methodological tools.
This course has three objectives:
introducing to the emergent field of big data analytics and social mining;
introducing to the technological scenario of big data, like programming tools to analyze big data, query NoSQL databases, and perform predictive modeling;
guide students to the development of a open-source and reproducible big data analytics project, based on the analyis of real-world datasets.
Module 1: Big Data Analytics and Social Mining
In this module, analytical methods and processes are presented thought exemplary cases studies in challenging domains, organized according to the following topics:
Module 2: Big Data Analytics Technologies
This module will provide to the students the technologies to collect, manipulate and process big data. In particular the following tools will be presented:
Python for Data Science
The Jupyter Notebook: developing open-source and reproducible data science
MongoDB: fast querying and aggregation in NoSQL databases
GeoPandas: analyze geo-spatial data with Python
Scikit-learn: machine learning in Python
Keras: deep learning in Python
Module 3: Laboratory for Interactive Project Development
During the course, teams of students will be guided in the development of a big data analytics project. The projects will be based on real-world datasets covering several thematic areas. Discussions and presentation in class, at different stages of the project execution, will be performed.
1st Mid Term: Data Understanding and Project Formulation
2nd Mid Term: Model(s) construction and evaluation
3rd Mid Term: Model interpretation/explanation
Exam: Final Project results
Calendar
14/09 (Mod. 1) Introduction to the course, The Big Data scenario lesson1_introduction_to_the_course_bda2021.pdf
15/09 (Mod. 2) Python for Data Science and the Jupyter Notebook: developing open-source and reproducible data science
21/09 No Lesson (Election Day in Italy)
22/09 (Mod. 3) Presentation of datasets for projects bda20_21_datasets_1_.pdf
28/09 (Mod. 2) Scikit-learn: programming tools for data mining (part 1): http://bit.ly/bda_notebooks_2
29/09
05/10 No Lesson (SocInfo2020 conference)
06/10 No Lesson (SocInfo2020 conference)
12/10 (Mod. 2) Geopandas and scikit-mobility: managing geographic data in Python (part 1) bda2021_geopandas.zip
13/10 (Mod. 2) Geopandas and scikit-mobility: managing geographic data in Python (part 2) https://github.com/scikit-mobility/tutorials/tree/master/mda_masterbd2020
19/10 (Mod. 3) 1st Mid Term - first group of teams
20/10 (Mod. 3) 1st Mid Term - second group of teams
26/10 (Mod. 3) Discussion and group working on projects
27/10 (Mod. 3) Discussion and group working on projects
02/11 (Mod. 1) Nowcasting well-being with big data bda_wellbeing.pdf
03/11 (Mod. 1) Injury prediction in sports with AI bda_2020_injury_forecasting.pdf
09/11 (Mod. 3) Discussion and group working on projects
10/11 (Mod. 1) Trustworthy data mining and Explainable AI parti1.explainableai-10.11.2020.pdf
16/11 (Mod. 3) 2nd Mid Term - first group of teams
17/11 (Mod. 3) 2nd Mid Term - second group of teams
23/11 (Mod. 3) Discussion and group working on projects
24/11 - No Lesson
30/11 (Mod. 3) Paper presentation
01/12 (Mod. 3) Paper presentation
07/12 (Mod. 3) 3rd Mid Term - first group of teams
08/12 (Mod. 3) 3rd Mid Term - second group of teams
Exam (Appelli)
January 14th, 2021
February 4th, 2021
Previous Big Data Analytics websites