This document describes what is expected of a final project and proposes some possible project ideas.
What is Expected
Good class projects can vary dramatically in complexity, scope, and topic. The topic should be something related to databases and/or analysis of data. It should have some programmatic component, e.g., doing analysis using Excel does not count. You will have to write some scripts that process the data.
You can use what you have used in DS 200, but, you have to have something new you did in this class. For example, a design and compare different database options to select a good backend for your application may be acceptable. Simply using a project from another course is not allowed. You can reuse code, but you have to clearly identify what you are doing new in this class.
What to Hand In
There are two written deliverables: a proposal due on Sep 28, midway report including a project proposal, due on Nov 1, and a final report, due on the last day of class.
Project Proposal: The proposal should consist of 1-2 pages describing the problem you plan to solve, outlining how you plan to solve it, and describing what you will "deliver" for the final project. In the project proposal, you need to clearly identify the non-trivial use of databases and data analytic techniques you will use.
Final Report: You should prepare a report on your project with maximum length of 15 pages (10 pt font or larger, one or two columns, 1 inch margins, single or double spaced -- more is not better.) Your report should introduce and motivate the problem your project addresses, describe related work in the area, discuss the elements of your solution, and present results that measure the behavior, performance, or functionality of your system (with comparisons to other related systems as appropriate.)
Because this report is the primary deliverable upon which you will be graded, do not treat it as an afterthought. Plan to leave at least a few days to do the writing, and make sure your proofread and edit carefully!
Project Ideas
The following is a list of possible project ideas from past years. You are not required to choose from this list -- in fact, we encourage you to try to solve a problem of your own choosing! Note that these are not meant to be complete project proposals, but just suggestions for areas to explore -- you will need to flesh them out into complete projects.
· Loading large datasets (> 10 GB) can be very slow, especially when there are many indexes on tables. One simple trick is to sort data before inserting it, or to drop indexes, insert data, and reload indexes. The goal of this project would be to investigate other techniques to improve load performance in an open source database such as Postgres or MySQL/Innodb. One concrete idea might be "fast bulk load" a la Hbase for MySQL. HBase uses a MapReduce job to build its indexes. One possible project would be to build a prototype that does the same for a single (or multiple) MySQL Boxes.
· Twitter provides a fire hose of data. Automatically filtering, aggregating, analyzing such data can allow to harness the full value of the data, extracting valuable information. The idea of this project is to investigate stream processing technology to operate on social streams, or extract structure from them and store them in a database system.
· Compare and contrast the performance of different SQL/NoSQL solutions on different workloads (e.g., MySQL vs Postgres vs Hive vs Pig vs Voldemort vs Mongo vs ...)
· Build an automatic tool to extract data from websites such as the car sales website and populate the data into a database. You can use off-the-shelf software to extract the data from the sites. Design a set of queries someone may want to run on the data from these sites. Justify the basis for your choice of database.
· Build a tool that scrapes data from the web related to a domain, e.g., football or healthcare. Identify five major queries that will be supported by the website. Build a database and querying interface that will allow you to query the data.
Adapted from: http://db.csail.mit.edu/6.830/assignments/final_proj.html