====== Data Mining A.A. 2016/17 ======
===== DM 1: Foundations of Data Mining =====
Instructors - Docenti:
* **Dino Pedreschi, Anna Monreale**
* KDD Laboratory, Università di Pisa ed ISTI - CNR, Pisa
* [[http://www-kdd.isti.cnr.it]]
* [[dino.pedreschi@unipi.it]]
* [[anna.monreale@unipi.it]]
* [[mirco.nanni@isti.cnr.it]]
Teaching assistant - Assistente:
* **Riccardo Guidotti**
* KDD Laboratory, Università di Pisa and ISTI - CNR, Pisa
* [[guidotti@di.unipi.it]]
===== DM 2: Advanced topics on Data Mining and case studies =====
Instructors:
* **Mirco Nanni, Dino Pedreschi**
* KDD Laboratory, Università di Pisa and ISTI - CNR, Pisa
* [[http://www-kdd.isti.cnr.it]]
* [[mirco.nanni@isti.cnr.it]]
* [[dino.pedreschi@unipi.it]]
====== News =====
* **The results of the written exam for DM2 held on September 6th, 2017 are out!** Link: {{ :dm:results.2017.09.06.pdf | Results DM2 6.9.2017}}. The students interested in having the oral exam in this session are requested to contact the instructor at [[mirco.nanni@isti.cnr.it]]. We remind you that the project report must be submitted at least 2 days before the oral exam.
* The results of the written exam for DM2 held on July 4th, 2017 are out! Link: {{ :dm:results.2017.07.04.pdf | Results DM2 4.7.2017}}. The students interested in having the oral exam in this session are requested to contact the instructor at [[mirco.nanni@isti.cnr.it]]. We remind you that the project report must be submitted at least 2 days before the oral exam.
* The results of the written exam for DM2 held on June 13th, 2017 are out! Link: {{ :dm:results.2017.06.13.pdf |Results 13.6.2017}}. The students interested in having the oral exam in this session are requested to contact the instructor at [[mirco.nanni@isti.cnr.it]]. We remind you that the project report must be submitted at least 2 days before the oral exam.
* The results of the 2nd mid-term held on June 6th, 2017 are out! Link: {{ :dm:results.2017.06.06.pdf | Results 6.6.2017}}.
* Reminder: 2nd mid-term exam is on 6.6.2017 at 11.00 a.m. in rooms A and B.
* All the projects for DM2 are now available! Check the exam section.
* The results of the mid-term held on April 7th, 2017 are out! Link: {{ :dm:results.2017.04.07.pdf | Results 7.4.2017}}.
* The lecture of May 5, 2017 is cancelled due to the review of a European project where the instructors are committed.
* Erratum: Date for mid-term exam is out: April 7th, 2017 at 11:00 (instead of 9:00) in Rooms A1 + C1.
* New dates for DM1 oral exams: 2017/02/21 15:00 (3 seats available); 2017/02/23 09:30 (6 seats available) office of Prof. Pedreschi. Please, send an email to BOTH me and Riccardo to book the oral exam.
* Results DM1 Written Exam 2017/02/08: {{:dm:results-08022017.pdf|Results DM1}}
* Oral Exam Calendar 2017/02/13 14.00 (seats available), 2017/02/15 14.00 (seats available) office of Prof. Pedreschi. If you need to do the oral exam before the 2017/02/17 but these dates do not fit your timetable please contact us as soon as possible. The next oral exam will be on June.
* Results DM1 Written Exam 2017/01/19: {{:dm:votidm-january.pdf|Results DM1}}: students who want to do the oral exam the first date available is 2017/01/30 10.00 office of Prof. Pedreschi. Please write an email to [[anna.monreale@unipi.it]] if you want to do the exam on Monday. Other dates will be scheduled and published on Monday
* Oral Exam Calendar 2017/01/23 14.00 (seats available), 2017/01/24 14.00 (completed), 2017/01/30 10.00 (seats available) office of Prof. Pedreschi.
* 2017/01/23: Shtjefni, Cei; 2017/01/24: Inversi, Savasta, Semeraro, Bonfanti, Tanga, Briganti, Di Sarli, Pioli
* A new project is now available! (see Exam section for details) We recommend to follow the [[dm:start:guidelines|guidelines]].
* Results of the second mid-test of DM1: {{:dm:secondmidtest.pdf| Results-21Dec2016}}. For opening the file you need a password that I will send you by email. If you did not receive the email you can require it by email to: [[anna.monreale@unipi.it]]. RULES: 1) Students having an AVG Mark in the file may do the oral exam. 2) Students having a vote >= 18 in only one of the tests can do the written exam for the part without a sufficient mark. 3) For the oral exam students must come to the written exam and decide together with the teachers the dates of the oral exam. It is possible to do the oral exam also during the written exam.
* Project deadline extension 23.59 of 10/01/2016
* Results of the first mid-test of DM1: {{:dm:first-midtest.pdf|Results-04Nov2016}}. For opening the file you need a password that you can require by email to: [[anna.monreale@unipi.it]]
* To be included in the course mailing list for urgent communications, please send as soon as possible a mail to [[anna.monreale@unipi.it]] with the following data: **subject**= “DM1” and **text:** name and surname
* Datasets for exercises - Iris: http://archive.ics.uci.edu/ml/datasets/Iris, Titanic: https://www.kaggle.com/c/titanic/data, Adult: https://archive.ics.uci.edu/ml/datasets/Adult
* The first project is now available! Details in the Exam Section.
====== Learning goals -- Obiettivi del corso ======
** ... a new kind of professional has emerged, the data scientist, who combines the skills of software programmer, statistician and storyteller/artist to extract the nuggets of gold hidden under mountains of data. Hal Varian, Google’s chief economist, predicts that the job of statistician will become the "sexiest" around. Data, he explains, are widely available; what is scarce is the ability to extract wisdom from them. **
//Data, data everywhere. The Economist, Special Report on Big Data, Feb. 2010.//
La grande disponibilità di dati provenienti da database relazionali, dal web o da altre sorgenti motiva lo studio di tecniche di analisi dei dati che permettano una migliore comprensione ed un più facile utilizzo dei risultati nei processi decisionali. L'obiettivo del corso è quello di fornire un'introduzione ai concetti di base del processo di estrazione di conoscenza, alle principali tecniche di data mining ed ai relativi algoritmi. Particolare enfasi è dedicata agli aspetti metodologici presentati mediante alcune classi di applicazioni paradigmatiche quali il Basket Market Analysis, la segmentazione di mercato, il rilevamento di frodi. Infine il corso introduce gli aspetti di privacy ed etici inerenti all’utilizzo di tecniche inferenza sui dati e dei quali l’analista deve essere a conoscenza. Il corso consiste delle seguenti parti:
- i concetti di base del processo di estrazione della conoscenza: studio e preparazione dei dati, forme dei dati, misure e similarità dei dati;
- le principali tecniche di datamining (regole associative, classificazione e clustering). Di queste tecniche si studieranno gli aspetti formali e implementativi;
- alcuni casi di studio nell’ambito del marketing e del supporto alla gestione clienti, del rilevamento di frodi e di studi epidemiologici.
- l’ultima parte del corso ha l’obiettivo di introdurre gli aspetti di privacy ed etici inerenti all’utilizzo di tecniche inferenza sui dati e dei quali l’analista deve essere a conoscenza
===== Reading about the "data scientist" job =====
* Data, data everywhere. The Economist, Feb. 2010 {{:dm:economist--010.pdf|download}}
* Data scientist: The hot new gig in tech, CNN & Fortune, Sept. 2011 [[http://tech.fortune.cnn.com/2011/09/06/data-scientist-the-hot-new-gig-in-tech/|link]]
* Welcome to the yotta world. The Economist, Sept. 2011 {{:dm:economist-2012-dm.pdf|download}}
* Data Scientist: The Sexiest Job of the 21st Century. Harvard Business Review, Sept 2012 [[http://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century/ar/1|link]]
* Il futuro è già scritto in Big Data. Il SOle 24 Ore, Sept 2012 [[http://www.ilsole24ore.com/art/tecnologie/2012-09-21/futuro-scritto-data-155044.shtml?uuid=AbOQCOhG|link]]
* Special issue of Crossroads - The ACM Magazine for Students - on Big Data Analytics {{:dm:crossroadsxrds2012fall-dl.pdf|download}}
* Peter Sondergaard, Gartner, Says Big Data Creates Big Jobs. Oct 22, 2012: [[https://www.youtube.com/watch?v=mXLy3nkXQVM|YouTube video]]
* Towards Effective Decision-Making Through Data Visualization: Six World-Class Enterprises Show The Way. White paper at FusionCharts.com. [[http://www.fusioncharts.com/whitepapers/downloads/Towards-Effective-Decision-Making-Through-Data-Visualization-Six-World-Class-Enterprises-Show-The-Way.pdf|download]]
====== Hours - Orario e Aule ======
===== DM 1 =====
**Classes - Lezioni**
^ Day of Week ^ Hour ^ Room ^
| Lunedì/Monday | 11:00 - 13:00 | Aula C |
| Venerdì/Friday | 14:00 - 16:00 | Aula A1 |
**Office hours - Ricevimento:**
* Prof. Pedreschi/Monreale: Lunedì/Monday h 14:00 - 16:00, Dipartimento di Informatica
===== DM 2 =====
**Classes - Lezioni**
^ Day of week ^ Hour ^ Room ^
| Tuesday | 16:00 - 18:00 | B |
| Friday | 16:00 - 18:00 | B |
**Office hours - Ricevimento:**
* Nanni : appointment by email, c/o ISTI-CNR
====== Learning Material -- Materiale didattico ======
===== Textbook -- Libro di Testo =====
* Pang-Ning Tan, Michael Steinbach, Vipin Kumar. **Introduction to Data Mining**. Addison Wesley, ISBN 0-321-32136-7, 2006
* [[http://www-users.cs.umn.edu/~kumar/dmbook/index.php]]
* I capitoli 4, 6, 8 sono disponibili sul sito del publisher. -- Chapters 4,6 and 8 are also available at the publisher's Web site.
* Berthold, M.R., Borgelt, C., Höppner, F., Klawonn, F. **GUIDE TO INTELLIGENT DATA ANALYSIS.** Springer Verlag, 1st Edition., 2010. ISBN 978-1-84882-259-7
===== Slides of the classes -- Slides del corso =====
* The slides used in the course will be inserted in the calendar after each class. Most of them are part of the the slides provided by the textbook's authors [[http://www-users.cs.umn.edu/~kumar/dmbook/index.php#item4|Slides per "Introduction to Data Mining"]].
//Le slide utilizzate durante il corso verranno inserite nel calendario al termine di ogni lezione. In buona parte esse sono tratte da quelle fornite dagli autori del libro di testo: [[http://www-users.cs.umn.edu/~kumar/dmbook/index.php#item4|Slides per "Introduction to Data Mining"]]//
===== Past Exams =====
* Some exercises (partially with solutions) on **sequential patterns** and **time series** can be found in the following texts of exams from the last years:
* {{ :dm:dm2_exam.2015.04.13.results.pdf|}}, {{ :dm:dm2_exam.2016.04.4_sol.pdf |}}, {{ :dm:dm2_exam.2016.04.5_sol.pdf |}}, {{ :dm:dm2_exam.2016.06.20_sol.pdf |}}, {{ :dm:dm2_exam.2016.07.08_sol.pdf |}}
* Some very old exercises (part of them with solutions) are available here, most of them in Italian, not all of them on topics covered in this year program:
* {{tdm:verifica2006.pdf|Verifica 2006}}, {{tdm:verifica2005.pdf|Verifica 2005 (con soluzioni)}}, {{tdm:verifica2004.pdf|Verifica 2004}}
* {{dm:verifica.05.06.2007.pdf|Verifica 5 giugno 2007}}, {{dm:verifica.26.06.2007.pdf|Verifica 26 giugno 2007}}, {{dm:verifica.24.07.2007_corretto.pdf|Verifica 24 luglio 2007}} (e {{dm:verifica.24.07.2007_soluzioni.pdf|Soluzioni}})
* {{:dm:verifica.2008.04.03.pdf|Verifica 3 aprile 2008}} (e {{:dm:soluzioni.2008.04.03.pdf|Soluzioni}}), {{:dm:dm-tdm.appello_2008_07_18_parte1.pdf|Verifica 18 luglio 2008 - parte 1}}, {{:dm:dm-tdm.appello_2008_07_18_parte2.pdf|Verifica 18 luglio 2008 - parte 2}}
* {{:dm:appello.2010.06.01_soluzioni.pdf| Exam with solution 2010-06-01}} {{:dm:appello.2010.06.22_soluzioni.pdf|Exam with solution 2010-06-22}} {{:dm:appello.2010.09.09_soluzioni.pdf|Exam with solution 2010-09-09}}{{:dm:appello.2010.07.13_soluzioni.pdf| Exam with solution 2010-07-13}}
===== Data mining software=====
* [[http://www.knime.org | KNIME ]] The Konstanz Information Miner. [[http://www.knime.org/download-desktop| Download page ]]
* [[https://www.continuum.io/downloads | Python - Anaconda (2.7 version!!!)]]: Anaconda is the leading open data science platform powered by Python. [[https://www.continuum.io/downloads | Download page]] (the following libraries are already included)
* Scikit-learn: python library with tools for data mining and data analysis [[http://scikit-learn.org/stable/ | Documentation page]]
* Pandas: pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. [[http://pandas.pydata.org/ | Documentation page]]
* [[http://www.cs.waikato.ac.nz/ml/weka/ | WEKA ]] Data Mining Software in JAVA. University of Waikato, New Zealand [[http://www.cs.waikato.ac.nz/ml/weka/ | Download page ]]
====== Class calendar - Calendario delle lezioni (2016-2017) ======
===== First part of course, first semester (DMF - Data mining: foundations) =====
^ ^ Day ^ Aula ^ Topic ^ Learning material ^ Instructor ^
|1.| 19.09.2016 11:00-13:00 | C |Canceled | | - |
|2.| 23.09.2016 14:00-16:00 | A1 | Introduction |{{:dm:1.courseoverview.pdf|Course Overview}}{{:dm:2.dm_ml_introduction.pdf|DM Introduction}}| Monreale |
|3.| 26.09.2016 11:00-13:00 | C | Data Understanding|{{:dm:3.dataunderstanding.pdf|}} {{:dm:3.data-understanting-appendix.pdf|}}| Monreale |
|4.| 30.09.2016 14:00-16:00 | A1 |Data Preparation |{{:dm:4.data_preparation.pdf|}} | Monreale |
|5.| 03.10.2016 11:00-13:00 | C | Introduction to Python, Knime | {{:dm:python_tutorial.zip}}| Monreale/Guidotti |
|6.| 07.10.2016 14:00-16:00 | A1 | Exercises on Data Understanding.| {{:dm:exercises-dm1.pdf|}} | Monreale/Guidotti |
|7.| 10.10.2016 11:00-13:00 | C | Centroid-based methods.|{{:dm:dm2014_clustering_intro.pdf|}} {{:dm:dm2014_clustering_kmeans.pdf|}} | Monreale |
|8.| 14.10.2016 14:00-16:00 | A1 | Hierarchical methods.Density Based Clustering | {{:dm:dm2014_clustering_hierarchical.pdf|}} {{:dm:knime_slides_mains.pdf|}} | Monreale |
|9.| 17.10.2016 11:00-13:00 | C | Knime - Python: Data Understanding | {{:dm:start:python_data_understanding.zip}} {{:dm:start:knime_data_manipulation_iris.zip}} {{:dm:start:knime_data_manipulation_adult.zip}} | Monreale/Guidotti |
|10.| 21.10.2016 14:00-16:00 | A1 | Clustering Validation | {{:dm:dm2014_clustering_validation.pdf}} | Monreale |
|11.| 24.10.2016 11:00-13:00 | C | Knime - Python: Clustering| {{:dm:exercise-hc-ga.pdf|HC with Group Average}} {{:dm:exercises-clustering.pdf|}} {{:dm:start:knime_clustering_iris.zip}} {{:dm:start:titanic_clustering.ipynb.zip}} | Monreale/Guidotti |
|12.| 28.10.2016 14:00-16:00 | A1 | Exercises on Clustering | {{:dm:exercise-hc-ga.pdf|HC with Group Average}} {{:dm:exercises-clustering.pdf|}}| Monreale/Guidotti |
| | 04.11.2016 9:00-11:00 | A | First Mid-term test | | Monreale/Guidotti |
|13.| 07.11.2016 11:00-13:00 | C | Frequent Patterns & Association Rules| {{:dm:4-5tdm-restructured_assoc.pdf|}} | Monreale |
|14.| 11.11.2016 14:00-16:00 | A1 | Event on Big Data: Aula Magna | | |
|15.| 14.11.2016 11:00-13:00 | C | Frequent Patterns & Association Rules | | |
|16.| 18.11.2016 14:00-16:00 | A1 | Knime - Python: Frequent Pattern & Association Rules | {{:dm:knime_pattern.zip}} {{dm:knime_pattern_titanic2.zip}} {{dm:titanic_frequent_patterns.ipynb.zip}} (http://www.borgelt.net/apriori.html) | |
|17.| 21.11.2016 11:00-13:00 | C | Classification | {{:dm:chap4_basic_classification.pdf|}} | |
|18.| 25.11.2016 14:00-16:00 | A1 | Classification | | |
|19.| 28.11.2016 11:00-13:00 | C | Classification | | |
|20.| 02.12.2016 14:00-16:00 | A1 | Exercises on Patterns & Classification | | |
|21.| 05.12.2016 11:00-13:00 | C | Canceled | | |
|22.| 09.12.2016 14:00-16:00 | A1 | Canceled | | |
|23.| 12.12.2016 11:00-13:00 | C | Exercises on Patterns & Classification | {{:dm:knime_classification_iris.zip}} {{:dm:titanic_classification.ipynb.zip}} | Guidotti / Pedreschi |
|24.| 16.12.2016-18.12.2015 | A1 | Knime - Python: Classification | | Guidotti / Pedreschi |
| | 21.12.2016 9:00-11:00 | A | Second Mid-term test | | Monreale/Guidotti|
===== Second part of course, second semester (DMA - Data mining: advanced topics and case studies) =====
^ ^ Day ^ Room (Aula) ^ Topic ^ Learning material ^ Instructor (default: Nanni)^
|1.| 21.02.2017 16:00-18:00 | B | Introduction + Sequential patters/1 | {{ :dm:dm2_2017_intro.pdf |Introduction}} {{ :dm:sequential_patterns.pdf |Sequential patters}} | Nanni + Pedreschi |
|2.| 24.02.2017 16:00-18:00 | B | Sequential patterns/2 | | |
|3.| 28.02.2017 16:00-18:00 | B | Sequential patterns/3 | Link to [[http://www.philippe-fournier-viger.com/spmf/|SPMF]], a tool for seq. patterns and {{:dm:sequences_of_poits.zip|sample dataset}}. Exercises: {{:dm:exercises_sp_graphs_ts.pdf|Text 1}} and {{:dm:ex_2014-15.pdf|Text 2}} | |
| | 03.03.2017 16:00-18:00 | B | //cancelled// | | |
|4.| 07.03.2017 16:00-18:00 | B | Time series/1 | {{ :dm:time_series_2017.pdf |Time series}} | |
|5.| 10.03.2017 16:00-18:00 | B | Time series/2 | {{ :dm:python_timeseries.zip |Python examples}}, {{ :dm:knime_timeseries.zip |Knime examples}}, link to [[https://www.dropbox.com/s/c12fmsctfwwov5d/sounds.zip|sounds dataset]] (source: [[http://nbviewer.jupyter.org/github/pierre-rouanet/dtw/blob/master/speech-recognition.ipynb|speech recognition example]]) | |
|6.| 14.03.2017 16:00-18:00 | B | Time series/3 | {{ :dm:python_timeseries2.zip |Python examples/2}} | |
|7.| 17.03.2017 16:00-18:00 | B | Time series/4 | {{ :dm:python_timeseries3.zip |Python examples/3}}, {{ :dm:02_example_for_predicting_time_series_-_outliers.zip |Knime example}} | |
|8.| 21.03.2017 16:00-18:00 | B | DM Process/1 | {{ :dm:dm2_2017_crm_airmiles.pdf |Example AMRP}} (also described in this {{ :dm:case_studies.pdf |report}}, in Italian), {{ :dm:dm2_2015_crispdm_mains.pdf |CRISP-DM}}, Link to the [[https://www.the-modeling-agency.com/crisp-dm.pdf|CRISP-DM 1.0 guide]] (by SPSS) | |
|9.| 24.03.2017 16:00-18:00 | B | DM Process/2 | {{:dm:0.crm_intro_gordonparsonschultz.pdf|Intro_CRM}} {{:dm:1.crm_churn_2015.pdf|Churn}} | |
|10.| 28.03.2017 16:00-18:00 | B | DM Process/3 | {{ :dm:2.st_events_2015.pdf |Collective churn analysis}}, {{ :dm:dm2_2017_crm_coop_promotions.pdf |Promotions}}, {{ :dm:crm2014_pennacchioli_bigdata13.pdf |Sophistication}}. Sample reports made by students and (loosely) following CRISP-DM: {{ :dm:dm2_report1_ita.pdf |Report 1 (Italian)}}, {{ :dm:dm2_report2_eng.pdf |Report 2 (English)}}, {{ :dm:dm2_report3_ita.pdf |Report 3 (Italian)}}. Exercise on CRISP-DM: {{ :dm:project_simultion_2017.pdf |understanding churn}}| |
| | 31.03.2017 16:00-18:00 | B | Cancelled | | |
|11.| 04.04.2017 16:00-18:00 | B | Exercises | Exercise on {{ :dm:project_simultion_2017_full.pdf |Understanding churn (with a solution)}}. See also exercises in section [[dm:start#past_exams|Past Exams]] | |
| | 07.04.2017 11:00-13:00 | A1 + C1 | **Mid-term exams** | | |
|12.| 21.04.2017 16:00-18:00 | B | Classification: alternative methods/1 | {{ :dm:lezioneadvancedclassificationmethods1-knn_nb.pdf |slides}} on K-nearest neighbours and Naive Bayes | Pedreschi |
|13.| 28.04.2017 16:00-18:00 | B | Classification: alternative methods/2 | {{ :dm:lezioneadvancedclassificationmethods2-ann_svm.pdf |slides}} on Artificial Neural Networks and Support Vector Machines | Pedreschi |
|14.| 02.05.2017 16:00-18:00 | B | Classification: alternative methods/3 | {{ :dm:lezioneadvancedclassificationmethods3_rules-ensemble.pdf |slides}} on ensemble methods and {{ :dm:ensemblemethod_wisdomofthecrowd.pdf | slides}} on the wisdom of the crowds {{ :dm:voxpopuli-galton-1907.pdf | original 1907 Nature paper by Francis Galton "Vox populi" }} | Pedreschi |
|15.| 05.05.2017 16:00-18:00 | | Lecture canceled | | |
|16.| 09.05.2017 16:00-18:00 | B | Classification: validation methods/1 | {{ :dm:a_lecture_5_-_model_performance_analytics.pdf |Slides from P. Adamopoulos}}, {{ :dm:b_12_what_is_a_good_model.pdf |Slides from J.F. Ehmke}} | |
|17.| 12.05.2017 16:00-18:00 | B | Classification: validation methods/2 | {{ :dm:unbalanced_data.pdf |Imbalanced data & evaluation}}, {{ :dm:knime_classification_iris_alternative_methods.zip |Knime sample classification & evaluation}}, {{ :dm:titanic_classification_dm2.ipynb.zip |Python sample classification & evaluation}} | |
|18.| 16.05.2017 16:00-18:00 | B | Classification: validation methods/3 | | |
|19.| 19.05.2017 16:00-18:00 | B | Exercises | {{ :dm:exercises_classification_1.pdf|Ex. from past exams 1}}, {{ :dm:exercises_classification_2.pdf |Ex. from past exams 2}}, {{ :dm:neural_networks_svm_validation.pdf |Mixed Exercises}}, {{ :dm:lift_chart.pdf |Lift chart}} | |
|20.| 23.05.2017 16:00-18:00 | B | Outlier Detection/1 | [[https://www.siam.org/meetings/sdm10/tutorial3.pdf|Slides from SDM2010 tutorial]] | |
|21.| 26.05.2017 16:00-18:00 | B | Outlier Detection/2 | {{ :dm:outliers_python.zip |Python examples}}, {{ :dm:outliers_knime.zip |Knime examples}}, link to [[https://elki-project.github.io/|ELKI framework]], {{ :dm:elki_dataset.txt.zip |test dataset for ELKI}} | |
|22.| 30.05.2017 16:00-18:00 | B | Exercises | {{ :dm:outliers.pdf |Exercises on outliers detection}}, {{ :dm:validation_ensembles.pdf |Exercises on ensembles and ROC/Lift chart}} | |
| | 06.06.2017 11:00-13:00 | A + B | **Mid-term exams** | | |
====== Exams ======
===== Exam DM part I (DMF) ======
The exam is composed of three parts:
* A **written exam**, with exercises and questions about methods and algorithms presented during the classes. It can be substitute with the first and second mid-term tests of November and December.
* An **oral exam**, that includes: (1) discussing the project report with a group presentation; (2) discussing topics presented during the classes, including the theory of the parts already covered by the written exam.
* A **project** consists in exercises that require the use of data mining tools for analysis of data. Exercises include: data understanding, clustering analysis, frequent pattern mining, and classification. The project has to be performed by max 3 people. It has to be performed by using Knime, Python or a combination of them. The results of the different tasks must reported in a unique paper. The total length of this paper must be max 20 pages of text including figures. ** The project must be delivered at least 2 days before the oral exam**. The paper must emailed to [[datamining.unipi@gmail.com]]. Please, use “[DM 2016-2017] Project” in the subject. Tasks of the project:
- ** Data Understanding (Assigned on: 17/10/2016): ** Explore the dataset with the analytical tools studied and write a concise “data understanding” report describing data semantics, assessing data quality, the distribution of the variables and the pairwise correlations.
- ** Clustering analysis (Assigned on: 14/11/2016): ** Explore the dataset using various clustering techniques. Carefully describe your's decisions for each algorithm and which are the advantages provided by the different approaches. (see Guidelines for details)
- ** Association Rules (Assigned on: 21/11/2016): ** Explore the dataset using frequent pattern mining and association rules extraction. Then use them to predict a variable either for replacing missing values or to predict the hotel type. (see Guidelines for details)
- ** Classification (Assigned on: 12/12/2016): ** Explore the dataset using classification trees and random forest. Use them to predict the hotel type. (see Guidelines for details)
* Project 1
- Dataset: **Expedia** (Hotel Recommendations)
- Assigned: 11/01/2017
- Deadline: 11/02/2017 ** Deadline extension 23.59 of 13/02/2017 **
- Link: https://www.kaggle.com/c/expedia-hotel-recommendations
- Hint: if the dataset is too big to be analyzed by your computer you can use a representative sample of the entire dataset. You must specify in the project report how you selected this sample and justify your choices.
* Project 2
- Dataset: **Pima** (Diabets Detection)
- Assigned: 05/05/2017
- Deadline: 02/06/2017 ** Deadline extension 23.59 of 05/06/2017 **
- Links: https://archive.ics.uci.edu/ml/datasets/pima+indians+diabetes, https://www.kaggle.com/uciml/pima-indians-diabetes-database (same dataset)
**Guidelines for the project are [[dm:start:guidelines|here]].**
===== Exam DM part II (DMA) ======
The exam is composed of three parts:
* A **written exam**, with exercises and questions about methods and algorithms presented during the classes. It can be substitute with the first and second mid-term tests of April and June.
* An **oral exam**, that includes: (1) discussing the project report with a group presentation; (2) discussing topics presented during the classes, including the theory of the parts already covered by the written exam.
* A **project** consists in exercises that require the use of data mining tools for analysis of data. Exercises include: sequential patterns, time series, classification (alternative methods and validation), outlier detection. The project has to be performed by max 3 people. It has to be performed by using Knime, Python, other software or a combination of them. The results of the different tasks must reported in a unique paper. The total length of this paper must be max 20 pages of text including figures. The project must be delivered at least 2 days before the oral exam.
* **Sequential patterns**. Apply sequential pattern mining (with temporal constraints, if needed) to a dataset that encodes 100 Bach's chorales as sequences of numbers. Two files are provided: one encodes only notes (MIDI pitch integer numbers), the other encodes notes & durations as a single number (number = duration*100 + note). **Objective**: Find the top-5 most frequent sequences with at least 5 notes, and the top-5 //contiguous// sequences (i.e. contiguous strings of notes) with at least 4 notes. Repeat the experiments on both the datasets, using appropriate algorithms and adjusting parameters. **Dataset**: {{ :dm:uci_chorales.zip |Preprocessed data}}, see also the original data and further details on the [[https://archive.ics.uci.edu/ml/datasets/Bach+Chorales|UCI page]].
* **Time series**. It is given a dataset of the homicides recorded in the USA over 35 years, expressed as timeseries of yearly counts of homicides for each state. You are asked to look for similarities across the states. **Objectives**: check whether there is some periodicity in the timeseries; look for clusters over the time series using (i) DBSCAN with DTW, (ii) DBSCAN with Euclidean distance, (iii) K-means with Euclidean distance, each time searching the best parameters and commenting the results. In case of empty values (i.e. no records provided for some year/month in some state), fill them with a zero or another reasonable value. **Dataset**: download it from [[https://www.kaggle.com/murderaccountability/homicide-reports|Kaggle]] (11 MB , zipped); you can also (optionally) use the following {{ :dm:timeseries_homicidies.zip|preprocessing python script}} to extract the relevant data from the dataset.
* **Classification**. Using the Titanic dataset with target variable "Survived", extract one classification model for each of the following approaches: kNN, SVM, neural networks, naive Bayes. For at least one of them, perform a search to select the parameters that optimize accuracy. For all the others, simply choose reasonable parameters values. After dividing the dataset in training and test sets, provide for all the models the confusion matrix, accuracy, precision & recall for the positive class.
* **Outlier detection**. Given the 2-d dataset provided below, apply the distance-based outlier detection method (DB(epsilon,n)) fitting the parameters in order to have a 5% of outliers. On the same dataset apply the LOF method, and select the top 5% outliers according to it. Compare the outputs, showing the differences and trying to explain the results. **Dataset**: download {{ :dm:outlier_dataset.zip |here}}.
====== Appelli di esame ======
===== Mid-term exams =====
^ ^ Date ^ Hour ^ Place ^ Notes ^ Marks ^
| First Mid-term 2016 | 4.11.2016 | 9:00 - 11:00 | Room A | | |
| Second Mid-term 2016| 21.12.2016 | 9:00 - 11:00 | Room A | | |
^ ^ Date ^ Hour ^ Place ^ Notes ^ Marks ^
| 1st Mid-term 2017 | 7.4.2017 | 11:00 - 13:00 | Rooms A1 + C1 | {{ :dm:dm2_mid-term_exam.2017.04.07_solutions.pdf |Solutions}} | {{ :dm:results.2017.04.07.pdf |Results 7.4.2017}} |
| 2nd Mid-term 2017 | 6.6.2017 | 11:00 - 13:00 | Rooms A + B | {{ :dm:dm2_mid-term_exam.2017.06.06_solutions.pdf |Solutions}} | {{ :dm:results.2017.06.06.pdf |Results 6.6.2017}} |
===== Appelli regolari / Exam sessions =====
^ Session ^ Date ^ Time ^ Room ^ Notes ^ Solutions ^ Marks ^
| 1. | 19 Jan 2017 |09:00 | C | In the same date we will define the dates for the oral exam. | | |
| 2. | 08 Feb 2017 |14:00 | C | In the same date we will define the dates for the oral exam. | | |
| |
| 3. |08 June 2017 |14:00| A1 | (1) Oral exam of DM1 for students having already the vote for the written exam of DM1. (2) Oral exam of DM2 for students having already the vote for the written exam of DM2. Please, use the system for registration: https://esami.unipi.it/| | |
| 4. |09 June 2017 |10:00| A1 | (1) Oral exam of DM1 for students having already the vote for the written exam of DM1. (2) Oral exam of DM2 for students having already the vote for the written exam of DM2. Please, use the system for registration: https://esami.unipi.it/| | |
| 5. |13 June 2017 |11:00 | A1 | Written exam of DM1/DM2. In the same date we will do oral exam for students already having the written vote and we will define the dates for the oral exam. Please, use the system for registration: https://esami.unipi.it/| {{ :dm:dm2_exam.2017.06.13_solutions.pdf |Solutions}} | {{ :dm:results.2017.06.13.pdf| Results DM2 13.6.2017}} |
| 6. |04 July 2017| 09:00 | A1 | Written exam of DM1/DM2. In the same date we will do oral exam for students already having the written vote and we will define the dates for the oral exam. Please, use the system for registration: https://esami.unipi.it/ | {{ :dm:dm2_exam.2017.07.04_solutions.pdf |Solutions}} | {{ :dm:results.2017.07.04.pdf | Results DM2 4.7.2017}} |
| 7. |06 September 2017| 09:00 | A1 | Written exam of DM1/DM2. In the same date we will do oral exam for students already having the written vote and we will define the dates for the oral exam. Please, use the system for registration: https://esami.unipi.it/ | {{ :dm:dm2_exam.2017.09.06_solutions.pdf |Solutions}} | {{ :dm:results.2017.09.06.pdf | Results DM2 6.9.2017}} |
===== Appelli straordinari A.A. 2015/16 / Extra sessions A.A. 2015/16=====
^ Date ^ Time ^ Room ^ Notes ^ Results ^
| | | | | |
| | | | | |
====== Edizioni anni precedenti =====
* [[dm.2015-16]]
* [[dm.2014-15]]
* [[dm.2013-14]]
* [[dm.2012-13]]
* [[dm.2011-12]]
* [[dm.2010-11]]
* [[dm.2009-10]]
* [[dm.2008-09]]
* [[dm.2007-08]]
* [[dm.2006-07]]
* [[PhDWorkshop2011]]
* [[SNA.Ingegneria2011]]
* [[SNA.IMT.2011]]
* [[MAINS.SANTANNA.2011-12]]
* [[MAINS.SANTANNA.DM4CRM.2012]]
* [[MAINS.SANTANNA.DM4CRM.2016]]
* [[MAINS.SANTANNA.DM4CRM.2017 | Data Mining for Customer Relationship Management 2017]]