Big Data Management Question Paper

Magnum_1993

1.2DataWrangling.pptx

Home >Computer Science homework help >Big Data Management Question Paper

ppt/presentation.xml

ppt/slideMasters/slideMaster1.xml

Click to edit Master title style Edit Master text styles Second level Third level Fourth level Fifth level Big Data Management ‹#›

ppt/slides/slide1.xml

Data Mashups Big Data Management Phil Bartie [email protected] EM G.29 Based on material from: Alasdair Gray , Heriot-Watt University

ppt/slides/slide2.xml

Materials released under CC-BY License You are free to: Share — copy and redistribute the material in any medium or format Adapt — remix, transform, and build upon the material for any purpose, even commercially. The licensor cannot revoke these freedoms as long as you follow the license terms. Under the following terms: Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. Big Data Management 2

ppt/slides/slide3.xml

Outline Data Mashup Open Data Development steps Data wrangling Moving from data silos to data lakes Big Data Management 3

ppt/slides/slide4.xml

Mashups Big Data Management http:// www.zwarm.com /blog/2011/1/25/data- wrangling.html

ppt/slides/slide5.xml

Mashup Taking data from multiple sources and overlaying it in a visualisation. Data may be cached locally or accessed through an API. Data may be open or closed. Should respect the data license. Big Data Management 5

ppt/slides/slide6.xml

Open Data in Use Tim Bernes -Lee, TED Talk 2010 Big Data Management 6 https:// www.ted.com /talks/ tim_berners_lee_the_year_open_data_went_worldwide?language = en

ppt/slides/slide7.xml

London Fire Mashup Big Data Management 7

ppt/slides/slide8.xml

London Fire Mashup by ODI http://london-fire.labs.theodi.org/explore/ Proposal to shut 10 fire stations to cut cost but required to keep response time unchanged. Mashup takes data from London Fire Brigade: Incident Records Office for National Statistics: Regional Profiles Key Statistics London Big Data Management 8

ppt/slides/slide9.xml

Initial Display Big Data Management 9 http://www.cs.man.ac.uk/~alvaro/publications/sensors-11-08855.pdf

ppt/slides/slide10.xml

Sensor Data Big Data Management 10

ppt/slides/slide11.xml

Sea-state Forecast Model Big Data Management 11

ppt/slides/slide12.xml

SemSorGrid4Env EU research project 2008 – 2011 Smashup : semantically enabled mashup Real-time overlaying of different sources of data Open weather and traffic streams Commercial sea-defence data Private sea-state sensors and model data Enabled response planning for predicted and on-going flood events Big Data Management 12

ppt/slides/slide13.xml

Flight Radar 24 https://www.flightradar24.com/39.56,32.71/4 Big Data Management 13

ppt/slides/slide14.xml

Real Estate Agents House prices, maps, newspapers, school zones + other social demographic datasets Big Data Management 14 Schools layers e.g. https://www.zoopla.co.uk/ https://www.rightmove.co.uk/ style.visibility style.visibility style.visibility

ppt/slides/slide15.xml

Visualising Aggregated Data For example estate agents summarizing the spatial pattern of sale prices as a Heat Map Big Data Management 15

ppt/slides/slide16.xml

City Mapper Big Data Management 16 Integrate timetable data from different transport companies, for different modes of transport.

ppt/slides/slide17.xml

CDRC Maps Big Data Management 17 https://maps.cdrc.ac.uk/

ppt/slides/slide18.xml

UK - Ordnance Survey – Master Map Data Big Data Management 18 https://www.ordnancesurvey.co.uk/business-and-government/products/open-mastermap.html https://digimap.edina.ac.uk/ OS Master Map for the UK

ppt/slides/slide19.xml

OpenStreetMap – World Wide Map Data Big Data Management 19 https://www.openstreetmap.org

ppt/slides/slide20.xml

Data Brushing + Dashboards Big Data Management 20 https://plot.ly/products/dash/ https://dash-gallery.plotly.host/dash-oil-and-gas

ppt/slides/slide21.xml

CoronaVirus 2019/20 Big Data Management 21 https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6 The case data visualized is collected from various sources, including WHO , U.S. CDC , ECDC China CDC ( CCDC ), NHC and DXY . DXY is a Chinese website that aggregates NHC and local CCDC situation reports in near real-time, providing more current regional case estimates than the national level reporting organizations are capable of, and is thus used for all the mainland China cases reported in our dashboard (confirmed, suspected, recovered, deaths). U.S. cases (confirmed, suspected, recovered, deaths) are taken from the U.S. https://systems.jhu.edu/research/public-health/ncov/

ppt/slides/slide22.xml

Big Data Management 22

ppt/slides/slide23.xml

Grafana - Dashboard Big Data Management 23 https://play.grafana.org/d/000000012/grafana-play-home

ppt/slides/slide24.xml

Mashup Steps Discover data: Find relevant data, e.g. through datahub.io , govt websites Understanding: Interpret data Data cleaning: Manipulating data Data extraction: Select data subset Data display: Visualisations (Covered in F20DV) Juxtaposition of data See previous examples Big Data Management 24

ppt/slides/slide25.xml

Discovering Data Publishing Open Data Big Data Management http:// www.experiensense.com / wp -content/uploads/2012/04/ Transparencia -Open- Data.jpg

ppt/slides/slide26.xml

US Government Data is Open My Administration is committed to creating an unprecedented level of openness in Government. We will work together to ensure the public trust and establish a system of transparency , public participation , and collaboration . Openness will strengthen our democracy and promote efficiency and effectiveness in Government. – PRESIDENT OBAMA, 21/01/2009 http:// www.whitehouse.gov /open Big Data Management 26

ppt/slides/slide27.xml

Open Government Data Big Data Management 27 style.visibility style.visibility style.visibility style.visibility style.visibility style.visibility style.visibility

ppt/slides/slide28.xml

UAE – data examples UAE https://bayanat.ae/ Dubai specifically https://www.dubaipulse.gov.ae/ https://www.dm.gov.ae/en/OpenData/Pages/Default.aspx Big Data Management 28

ppt/slides/slide29.xml

Definitions Open Data Data that is licensed as available for anyone to use, for any purpose, at no cost. [ Open Data Institute ] [ UK Big Data Centre ] Commercial Data Data that is licensed as available for use according to the license terms, generally at cost. Closed Data Data that is unlicensed or under restricted license. Big Data Management 29

ppt/slides/slide30.xml

Why Open Data? Benefits Enable innovation Mash-ups of data reveal hidden value Ease information sharing Transparency Self-empowerment Services built around data Digital economy Examples Commuter apps Travel advisory alerts Accident black spots Crime hot spots http:// data.gov.uk /apps Real-time updates following Haiti earthquake Big Data Management 30

ppt/slides/slide31.xml

Privacy vs Availability What data should be Open? Government data, e.g. public transport, council spending? Personal data, e.g. health record? Anonymised data? How can data be effectively anonymised? Big Data Management 31 Google Transity Feed https://support.google.com/transitpartners/answer/1111577?hl=en

ppt/slides/slide32.xml

Discovery: Datahub.io Big Data Management 32

ppt/slides/slide33.xml

Datahub.io Registry of open datasets CKAN: open source software that powers datahub Multiple deployments UK Government US Government Urban Big Data Centre … Big Data Management 33

ppt/slides/slide34.xml

Data Wrangling Cleaning data to enable storage and analysis Big Data Management 80% of the work in any data project is cleaning the data. DJ Patil , Data Jujitsu https:// dataladder.com / wp -content/uploads/2016/03/dl2-blog.jpg

ppt/slides/slide35.xml

Real Data is Dirty Missing values: NULL or omitted values Typos: ‘ Jne ’ instead of ‘Jane’ or ‘June’ Different date types: ‘7/1/2015’ or ‘1/7/2015’ Different units: Speed limit of 30 Errors or outliers: Age value of 150 years Big Data Management 35

ppt/slides/slide36.xml

Data Completeness: n=All US 1936 Presidential Election: Roosevelt vs Landon Roosevelt won with 62% Big Data Management 36 Literary Digest Poll Sample: 2.4 million Prediction: Roosevelt 43% Biased sample George Gallup Poll Sample: 50,000 Prediction: Roosevelt 56% Random sample style.visibility style.visibility style.visibility

ppt/slides/slide37.xml

Data Assumptions What are your assumptions? Are they valid? What is the bias in the data? Is it representative of the population? Size isn’t everything! What are your biases? Big Data Management 37

ppt/slides/slide38.xml

Usability, Credibility, Useful Data is usable if it can be parsed and manipulated by computational tools. Cleaning data: make it “ fit for purpose ” Data is credible if it is suitably representative of a phenomenon to enable productive analysis Subjective assessment Data is useful if it is usable, credible, and responsive to one’s inquiry. Big Data Management 38

ppt/slides/slide39.xml

Cleaning Approaches Custom code: Scripting Python Pandas Spreadsheets Manual manipulation Workflow: Extract-Transform-Load (ETL) Map schema Transform data Load data Big Data Management 39 https:// www.kdnuggets.com / wp -content/uploads/data- sorted.jpg

ppt/slides/slide40.xml

Cleaning Process Audit your data Identify issues Define operations to correct data Test operations Run operation Repeat Big Data Management 40 OpenRefine Demo! https://youtu.be/B70J_H_zAWM https://www.youtube.com/watch?v=cO8NVCs_Ba0 style.visibility

ppt/slides/slide41.xml

Finding Open Data https://toolbox.google.com/datasetsearch Big Data Management 41 style.visibility

ppt/slides/slide42.xml

Findability - metadata Search Engine Optimisation Linked data Big Data Management 42 https://www.gov.uk/government/news/new-guide-helps-find-and-classify-geospatial-datasets-across-search-engines FAIR = Findable, Accessible, Interoperable, and Reusable (FAIR) – Dr Alasdair Gray , 2016, Nature Science Data style.visibility style.visibility style.visibility

ppt/slides/slide43.xml

The Future of Government Open Data in UK Impact of Brexit? Big Data Management 43 https://amp.theguardian.com/media-network/2016/jul/21/what-does-brexit-mean-open-data-uk “Government could use Brexit as an excuse to stop maintaining datasets that are produced to support EU programmes like Inspire and Eurostat , or to meet EU targets on air pollution and water quality . The UK would also no longer be bound by the PSI Directive , which underpins our regulatory framework for re-use of public sector information”….. 2016 style.visibility ppt_x ppt_y

ppt/slides/slide44.xml

Summary Open data has huge potential Governments empowering their citizens Also commercially Mashups enable human interpretation Mashup steps Discover Interpret data Clean data: Not as easy as it sounds Extract data Visualise the data: tell a story Big Data Management 44 https:// www.computescotland.com /images/e73Zh22gZAmSez6n7DDr09q09e.jpg

ppt/notesMasters/notesMaster1.xml

Data Mashups Click to edit Master text styles Second level Third level Fourth level Fifth level Big Data Management ‹#›

ppt/handoutMasters/handoutMaster1.xml

Data Mashups Big Data Management ‹#›

ppt/commentAuthors.xml

ppt/presProps.xml

ppt/viewProps.xml

ppt/theme/theme1.xml

ppt/tableStyles.xml

ppt/slideLayouts/slideLayout1.xml

Click to edit Master title style Click to edit Master subtitle style Big Data Management ‹#›

ppt/slideLayouts/slideLayout2.xml

Click to edit Master title style Edit Master text styles Second level Third level Fourth level Fifth level Big Data Management ‹#›

ppt/slideLayouts/slideLayout3.xml

Click to edit Master title style Click to edit Master subtitle style Big Data Management Click icon to add picture

ppt/slideLayouts/slideLayout4.xml

Click to edit Master title style Edit Master text styles Big Data Management ‹#›

ppt/slideLayouts/slideLayout5.xml

Click to edit Master title style Edit Master text styles Second level Third level Fourth level Fifth level Edit Master text styles Second level Third level Fourth level Fifth level Big Data Management ‹#›

ppt/slideLayouts/slideLayout6.xml

Click to edit Master title style Edit Master text styles Edit Master text styles Second level Third level Fourth level Fifth level Edit Master text styles Edit Master text styles Second level Third level Fourth level Fifth level Big Data Management ‹#›

ppt/slideLayouts/slideLayout7.xml

Click to edit Master title style Big Data Management ‹#›

ppt/slideLayouts/slideLayout8.xml

‹#›

ppt/slideLayouts/slideLayout9.xml

Click to edit Master title style Edit Master text styles Second level Third level Fourth level Fifth level Edit Master text styles Big Data Management ‹#›

ppt/slideLayouts/slideLayout10.xml

Click to edit Master title style Click icon to add picture Edit Master text styles Big Data Management ‹#›

ppt/slideLayouts/slideLayout11.xml

Click to edit Master title style Click to edit Master subtitle style Big Data Management Click icon to add picture

ppt/slideLayouts/slideLayout12.xml

Click to edit Master title style Click to edit Master subtitle style Big Data Management Click icon to add picture Click icon to add picture

ppt/slideLayouts/slideLayout13.xml

Click to edit Master title style Click to edit Master subtitle style Big Data Management Click icon to add picture Click icon to add picture Click icon to add picture

ppt/slideLayouts/slideLayout14.xml

Click to edit Master title style Edit Master text styles Second level Third level Fourth level Fifth level Big Data Management ‹#›

ppt/slideLayouts/slideLayout15.xml

Click to edit Master title style Edit Master text styles Second level Third level Fourth level Fifth level Big Data Management ‹#›

ppt/theme/theme2.xml

ppt/theme/theme3.xml

ppt/notesSlides/notesSlide1.xml

Embedded videos so need sound Remember to print out refine script for demo ROI and KPI (Return on Investment and Key Performance Indicator) - Driven by business questions: Even small companies can have a big data approach - Identify required data: move from data silos to a data lake - Combine data into model - Perform analysis based on hypothesis Draw conclusion 1 A.J.G. Gray – Big Data Management Data Mashups

ppt/media/image1.png

ppt/notesSlides/notesSlide2.xml

Data Mashups A.J.G. Gray – Big Data Management 3

ppt/notesSlides/notesSlide3.xml

Mashups are about combining data from multiple data sources Either through API calls or by caching the data locally Bringing different datasets together and using them to show something new, or learn more about them through integration. 4 A.J.G. Gray – Big Data Management Data Mashups

ppt/media/image2.png

ppt/media/media1.mp4

TED: Tim Berners-Lee (2010 University)

TED

2010

TEDTalks

To learn more about this speaker, find other TEDTalks, and subscribe to this Podcast series, visit www.TED.com Feedback: [email protected]

Podcast

ppt/notesSlides/notesSlide4.xml

{CLICK VIDEO} Note: Many of these involve maps – geospatial location is a useful way to integrate data – talk about Geographic Information Systems in a few weeks. Data Mashups A.J.G. Gray – Big Data Management 6

ppt/media/image3.png

ppt/notesSlides/notesSlide5.xml

{CLICK – IMAGE} Show live demo by clicking on image! {ENABLE ANALYSIS MODE - click on stations to shut them} {Note call out time profile for each London Borough by clicking on the Borough} ========================================== Proposal to shut 10 fire stations to cut cost but required to keep response time unchanged. Mashup takes data from London Fire Brigade: Incident Records Office for National Statistics: Regional Profiles Key Statistics London 7 A.J.G. Gray – Big Data Management Data Mashups

ppt/media/image4.png

ppt/notesSlides/notesSlide6.xml

Showing various data layers (roads, populated places) Real time data - ship positions Lots of sensors - often issue in bringing data together is in understanding what they are measuring, the units etc... especially worldwide. One research area is using Semantic Web to provide details about the values with the data. Alasdair will talk more about Semantic Web later on in this course. Data Mashups Big Data Management 9

ppt/media/image5.png

ppt/notesSlides/notesSlide7.xml

Can visualize data from sensors using graduated symbology ( eg waveheight ) Sensors provide real-time data as well as units ( eg m or knots) + capabilities such as sample rates Data Mashups Big Data Management 10

ppt/media/image6.png

ppt/notesSlides/notesSlide8.xml

Forecast the sea-state based on a model and the sensor inputs... Spatial - temporal model outputs - mapped to see which coastal regions are likely to be flooded. Data Mashups A.J.G. Gray – Big Data Management 11

ppt/media/image7.png

ppt/notesSlides/notesSlide9.xml

Web map data from Google combined with flight locations, combined with database of flights and plane images. Variety of data sources - ADS-B, MLAT and radar data - integrated with the airline flight details to get origin and destination details Main data source is ADS-B = automatic dependent surveillance-broadcast Flightradar24 started as a hobby project in 2006 when two Swedish aviation geeks decided to build a network of ADS-B receivers in Northern and Central Europe. In 2009 we opened up the network, and made it possible for anyone with an ADS-B receiver to upload data to the network. FlightRadar24 access 17,000 ADS-B receivers around the world. Data Mashups A.J.G. Gray – Big Data Management 13

ppt/media/image8.png

ppt/notesSlides/notesSlide10.xml

{CLICK - layers} {CLICK - school details} The house location, price and photos are available but so are other layers of information such as PARKS, TRANSPORT (Bus, train stations), SCHOOLS etc.. Some sites let you explore the school data, and even demographic data on newspapers read in the region, crime data, and professions. Data Mashups Big Data Management 14

ppt/media/image9.png

ppt/media/image10.png

ppt/notesSlides/notesSlide11.xml

Data Mashups Big Data Management 15

ppt/media/image11.png

ppt/notesSlides/notesSlide12.xml

{CLICK ON MAP - to go to web page} Data Mashups Big Data Management 16

ppt/media/image12.png

ppt/notesSlides/notesSlide13.xml

{CLICK on MAP} Data Mashups Big Data Management 17

ppt/media/image13.png

ppt/media/image14.png

ppt/media/image15.png

ppt/notesSlides/notesSlide14.xml

{CLICK to launch web page] Data Mashups Big Data Management 19

ppt/media/image16.png

ppt/notesSlides/notesSlide15.xml

Data Mashups Big Data Management 20

ppt/media/image17.png

ppt/media/image18.png

ppt/media/image19.png

ppt/media/image20.png

ppt/notesSlides/notesSlide16.xml

http://quakelive.co.nz/Browse/ -- eg. Bringing quake data to a wider audience through good interactive visualisations Data Mashups Big Data Management 24

ppt/media/image21.png

ppt/media/image22.jpg

ppt/notesSlides/notesSlide17.xml

{CLICK} UK, Belgium, Canadian, Austrian, NZ {CLICK} Developing world too: Indonesian, Kenyan, Indian 27 A.J.G. Gray – Big Data Management Data Mashups

ppt/media/image23.png

ppt/media/image24.png

ppt/media/image25.png

ppt/media/image26.png

ppt/media/image27.png

ppt/media/image28.png

ppt/media/image29.png

ppt/media/image30.png

ppt/media/image31.png

ppt/notesSlides/notesSlide18.xml

{CLICK to OPEN the 2 LINKS} Data Mashups Big Data Management 29

ppt/notesSlides/notesSlide19.xml

Google Transit Feed link  asking transport companies to make their data available in a particular format so that Google maps can use it.. Data Mashups Big Data Management 31

ppt/notesSlides/notesSlide20.xml

Number of openly available datasets Datasets in different formats Specialist deployments by governments: UK, US, Urban Big Data Centre We can find data, now what? Data Mashups A.J.G. Gray – Big Data Management 32

ppt/media/image32.png

ppt/notesSlides/notesSlide21.xml

Data wrangling is the process of preparing data 34 A.J.G. Gray – Big Data Management Data Mashups

ppt/media/image33.tiff

ppt/notesSlides/notesSlide22.xml

Date: 7 January or 1 July? Speed limit 30m.p.h or 30km/h often find both in the same field Can a human be 150 years old? Does your dataset cover the whole population? 35 A.J.G. Gray – Big Data Management Data Mashups

ppt/notesSlides/notesSlide23.xml

Big Data assumption that n=all Size isn’t everything BIAS - like the pothole app which would bias those with smartphones aware of the app - the richer part of the city ... not so much in the country... {CLICK to reveal Rossevelt won with 62% of vote} 36 A.J.G. Gray – Big Data Management Data Mashups

ppt/media/image34.tiff

ppt/notesSlides/notesSlide24.xml

Show example of using OpenRefine Remember the script! Screencast available as backup, see Vision Check Video Links to GOOGLE's DEMOs 40 A.J.G. Gray – Big Data Management Data Mashups

ppt/notesSlides/notesSlide25.xml

Searching for Data which has good Metadata Data Mashups Big Data Management 41

ppt/media/image35.png

ppt/notesSlides/notesSlide26.xml

Data Mashups Big Data Management 42

ppt/media/image36.png

ppt/media/image37.png

ppt/notesSlides/notesSlide27.xml

Article from 2016 It costs money to make data available (web space, data cleaning etc )… could UK govt use Brexit as an excuse to stop doing this? {CLICK to show paragraph about without EU pressure UK might not maintain OpenData } We don’t know yet… Data Mashups Big Data Management 43

ppt/media/image38.png

ppt/notesSlides/notesSlide28.xml

Nike opened up their supply chain details to eliminate myth of using sweat-shops Result ended up in more open competition to be their supplier http:// blog.okfn.org /2011/07/27/and-so-corporations-begin-to-open-data/ Data Mashups A.J.G. Gray – Big Data Management 44

ppt/media/image39.tiff

ppt/changesInfos/changesInfo1.xml

ppt/revisionInfo.xml

docProps/thumbnail.jpeg

docProps/core.xml

F28DM Database Management Systems Transaction Management Alasdair Gray Microsoft Office User 430 2016-02-03T11:21:06Z 2014-01-08T14:06:12Z 2020-01-30T11:27:12Z

docProps/app.xml

lectures 7336 2969 Microsoft Macintosh PowerPoint Widescreen 502 44 28 5 1 false Fonts Used 6 Theme 1 Slide Titles 44 Arial Calibri Century Gothic Lato Lucida Grande Wingdings 2 lectures Data Mashups Materials released under CC-BY License Outline Mashups Mashup Open Data in Use Tim Bernes-Lee, TED Talk 2010 London Fire Mashup London Fire Mashup by ODI Initial Display Sensor Data Sea-state Forecast Model SemSorGrid4Env Flight Radar 24 Real Estate Agents Visualising Aggregated Data City Mapper CDRC Maps UK - Ordnance Survey – Master Map Data OpenStreetMap – World Wide Map Data Data Brushing + Dashboards CoronaVirus 2019/20 PowerPoint Presentation Grafana - Dashboard Mashup Steps Discovering Data US Government Data is Open Open Government Data UAE – data examples Definitions Why Open Data? Privacy vs Availability Discovery: Datahub.io Datahub.io Data Wrangling Real Data is Dirty Data Completeness: n=All Data Assumptions Usability, Credibility, Useful Cleaning Approaches Cleaning Process Finding Open Data Findability - metadata The Future of Government Open Data in UK Summary Heriot-Watt University false false false 16.0000