financial mathematics dissertation

Executive Summary As identified by Her Majesty’s Chief Inspector of Education, Children’s Services and Skills,

there are a group of around 450 state-funded schools in England that have had poor inspection

results in every inspection they have had since 2005. These ‘stuck’ schools are receiving

increased levels of attention, since a poor inspection result is meant to instigate an improvement

in school performance.

This project aimed to investigate the data around stuck schools, to see if it was possible to

generate a machine learning model that could accurately predict whether a poorly performing

school would remain ‘stuck’ or would ‘escape’ the cycle of poor performance by itself.

To do this, a dataset needed to be created. This was done using input variables selected from a

number of publicly available sources. Many were provided by the Department for Education,

whilst details of every school inspection since 2005 were provided by Ofsted. The resulting

dataset had a row for each of the 21,900 open, state-funded schools in England and around 80

columns.

The definition of ‘stuck’ was updated slightly for this project, in consultation with the

Department for Education. The schools that met these new criteria were identified, along with

a different subset – those that had been performing poorly but had recently had one or more

good inspections and ‘escaped’ stuck. Schools that fitted in neither category were not used

further. The created dataset now had binary labels – ‘stuck’ and ‘escaped’ – from which a

binary classifier could be built and tested.

Six different binary classifiers were built in this project, using the Scikit-learn package in

Python: Random Forests, Support Vector Machines, Neural Networks, Gaussian Naïve Bayes,

Logistic Regression and K-Nearest Neighbours. For each model type, the optimum

combination of input features and model hyperparameters was found using an iterative process

involving random grid searches for model hyperparameters and sequential feature selection.

The best models were found to predict the future class of a school with 75% accuracy and area

under the ROC curve values of 75%. If a higher confidence in the prediction of a stuck school

is required, a Support Vector Machine model was correct in 88% of its predictions of stuck

schools (precision), although it only identified 40% of all stuck schools (recall). Logistic

Regression and Gaussian Naïve Bayes generally provided inferior results to the other four

model types, of which K-Nearest Neighbours was found to perform the best overall.

Accuracy values of 75% are useful, and show that the models can predict future inspection

results, but they are unlikely to be high enough to be of direct use to Ofsted and the Department

for Education. Given the selection of well known machine learning techniques used and the

large range of input features and hyperparameters tested, it is considered unlikely that

significant improvement upon these scores can be achieved without a step change in the quality

of the input data or the use of more advanced machine learning techniques that are beyond the

scope of this project.

An analysis of the most important features used by the models in making the classification was

also carried out, showing that different groups of features were being used by the different

models. Features concerning school financial balance were shown to be of high importance to

the Random Forest model in an assessment of its feature importance values.

This project was carried out with input from the two primary stakeholders: Ofsted and the

Department for Education. The results have been presented to them in person, with an

explanation of how they were achieved. They are now in a position to decide the future

direction of this work.

Acknowledgements My thanks in this project go to the EMU team at ONS who have been a pleasure to work with this summer – particularly to Joe for his help in all things from laptop setup to showing me how to get from the bike sheds to the showers and for his interest, support and guidance throughout; to Tim for giving me great flexibility, allowing me to concentrate solely on my project and being on hand if I needed anything; to Rebecca for keeping on finding errors in my list of stuck schools.

David Marshall, my supervisor in Cardiff University, has also been extremely helpful in this project, giving me useful practical advice on carrying out the work and for his assistance with writing this dissertation. I appreciate your willingness to meet on Skype on your day off.

I would also like to thank George and Louise at Ofsted and Pennie and Pippa at the Department for Education for their input during the project and their helpful comments and feedback in the presentation.

Contents 1. Introduction .................................................................................................................................... 1

1.1. School Inspections ....................................................................................................................... 1

1.2. Stuck Schools................................................................................................................................ 2

1.3. Ofsted and the Department for Education .................................................................................. 2

1.4. Tools Used .................................................................................................................................... 3

1.5. Project Aims ................................................................................................................................. 3

1.6. Process Plan ................................................................................................................................. 4

2. Literature Review ............................................................................................................................ 5

2.1. Previous Analysis of Schools Data ................................................................................................ 5

2.2. Machine Learning Models ............................................................................................................ 6

2.2.1. Logistic Regression ................................................................................................................ 6

2.2.2. Random Forests .................................................................................................................... 7

2.2.3. Support Vector Machines ..................................................................................................... 7

2.2.4. Neural Networks ................................................................................................................... 9

2.2.5. Naïve Gaussian Bayes Networks ......................................................................................... 11

2.2.6. K-Nearest Neighbours ......................................................................................................... 12

2.3. Methods for Refining the Models .............................................................................................. 13

2.3.1. Iterative imputation ............................................................................................................ 13

2.3.2. Sequential Forward and Backward Selection ..................................................................... 13

2.3.3. Recursive Feature Elimination ............................................................................................ 14

2.3.4. Oversampling ...................................................................................................................... 14

2.4. Measuring Model Effectiveness ................................................................................................. 15

2.4.1. Cross validation ................................................................................................................... 15

2.4.2. Accuracy .............................................................................................................................. 15

2.4.3. Precision and recall ............................................................................................................. 15

2.4.4. ROC curves .......................................................................................................................... 16

3. Input Data and Pre-Processing ...................................................................................................... 17

3.1. Datasets Available ...................................................................................................................... 17

3.1.1. Ofsted data.......................................................................................................................... 17

3.1.2. DfE data ............................................................................................................................... 17

3.1.3. Get Information About Schools data .................................................................................. 17

3.1.4. List of Academies ................................................................................................................ 18

3.2. Data Stitching ............................................................................................................................. 18

3.3. Data Cleaning ............................................................................................................................. 18

3.3.1. Missing data ........................................................................................................................ 18

3.3.2. Data types ........................................................................................................................... 19

3.3.3. Normalising ......................................................................................................................... 19

3.3.4. Imputation .......................................................................................................................... 19

4. Machine Learning Implementation............................................................................................... 21

4.1. Feature Generation .................................................................................................................... 21

4.1.1. Categorical data .................................................................................................................. 21

4.1.2. Changes over time .............................................................................................................. 21

4.1.3. Performance data ............................................................................................................... 21

4.2. Initial Feature Selection ............................................................................................................. 23

4.3. Stuck School Labelling ................................................................................................................ 23

4.3.1. Options for labels ................................................................................................................ 23

4.3.2. Updated definition of Stuck schools ................................................................................... 24

4.3.3. Data selection and labelling ................................................................................................ 24

4.3.4. Labelling method ................................................................................................................ 25

4.4. Data Summary ............................................................................................................................ 26

4.5. Principal Component Analysis .................................................................................................... 30

5. Results ........................................................................................................................................... 32

5.1. Assessment Methods Used ........................................................................................................ 32

5.2. Computing Set up for Modelling ................................................................................................ 33

5.3. Experiment 1: Finding the optimum machine learning classification model ............................ 37

5.3.1. Experiment 1.1: Finding the best model type ..................................................................... 37

5.3.2. Experiment 1.2: Finding the optimum model hyperparameters ........................................ 43

5.4. Experiment 2: Finding the optimum features to input to the model ........................................ 53

5.4.1. Experiment 2.1: Investigating whether Sequential Forward Selection or Sequential Backward Selection give better results ......................................................................................... 53

5.4.2. Experiment 2.2: Investigating whether Recursive Feature Elimination improves the set of features selected ........................................................................................................................... 55

5.4.3. Key features selected in most effective model ................................................................... 57

5.5. Experiment 3: Finding the optimum operations on the input data ........................................... 59

5.5.1. Experiment 3.1: Determining whether oversampling improves model performance ....... 59

5.6. Attributes of Most Effective Model ........................................................................................... 61

6. Discussion ...................................................................................................................................... 62

7. Conclusions ................................................................................................................................... 67

8. References .................................................................................................................................... 70

9. Appendices .................................................................................................................................... 74

9.1. Appendix 1: Input Variables Used .............................................................................................. 74

9.1.1. School Financial Balance ..................................................................................................... 74

9.1.2. School performance data .................................................................................................... 74

9.1.3. Pupil population and absence data..................................................................................... 75

9.1.4. Spine .................................................................................................................................... 75

9.1.5. Workforce data ................................................................................................................... 75

9.1.6. School finances data ........................................................................................................... 76

9.1.7. Generated variables ............................................................................................................ 77

9.2. Appendix 2: Variables selected for models ................................................................................ 77

9.2.1. Neural Network ................................................................................................................... 77

9.2.2. Support Vector Machine ..................................................................................................... 78

9.2.3. Random Forest .................................................................................................................... 78

9.2.4. Gaussian Naïve Bayes.......................................................................................................... 79

9.2.5. Logistic Regression .............................................................................................................. 79

9.2.6. K-Nearest Neighbours ......................................................................................................... 80

List of Figures Figure 1: A multi layer perceptron with 1 hidden layer containing k units (Scikit-learn, no date b) .... 10 Figure 2: Confusion Matrix .................................................................................................................... 15 Figure 3: Frequency of each overall inspection history ........................................................................ 27 Figure 4: Histograms showing distribution between 'Stuck' schools, 'Escaped' schools and schools that fall into neither category ............................................................................................................... 29 Figure 5: Principal Component Analysis - Cumulative explained variance versus number of principal components used .................................................................................................................................. 30 Figure 6: The results of Principal Component Analysis - the two classes and the ‘other’ remaining schools that fall in neither class are plotted on Principal Component 1 vs Principal Component 2 axes .............................................................................................................................................................. 31 Figure 7: Process for selecting optimum model for each model type .................................................. 38 Figure 8: Performance of the best version of each model type. Note that different measures of the same six models are plotted, sorted in order of decreasing area under the ROC curve score. ........... 39 Figure 9: ROC curve for model with highest area under the ROC curve value for each model type. The lines plotted are the mean scores over the 5 folds of cross validation. ............................................... 40 Figure 10: ROC curves for models in Table 3. Each of the five cross validation folds is plotted, along with a mean and the range of one standard deviation ........................................................................ 42 Figure 11: Precision for Stuck class - The best model from each model type in terms of precision for identifying stuck schools. The same six models are plotted in each chart, sorted in order of precision score. ..................................................................................................................................................... 42 Figure 12: Variation of AUC and Accuracy with different hyperparameters for Support Vector Machines using features listed in section 9.2 ....................................................................................... 45 Figure 13: Variation of AUC and Accuracy with different hyperparameters for Neural Networks using features listed in Section 9.2 ................................................................................................................. 47 Figure 14: Variation of AUC and Accuracy with different hyperparameters for Random Forests using features listed in Section 9.2 ................................................................................................................. 49

Figure 15: Variation of AUC and Accuracy with different hyperparameters for K-Nearest Neighbours using features listed in Section 9.2 ....................................................................................................... 50 Figure 16: Sequential Forward/Backward Selection results: Model accuracy for different numbers of features ................................................................................................................................................. 54 Figure 17: Recursive Feature Elimination - For each of the six different models, for each level of RFE, the highest score of Area Under the ROC curve and Accuracy are plotted.......................................... 56 Figure 18: Oversampling - For each model type, the run with the highest accuracy and area under the ROC curve are shown for the group where oversampling was used on the training data and the group which did not use oversampling. ................................................................................................ 60 Figure 19: ROC curves for the selected models, with randomness disabled in the cross validation process. Fold 1 is therefore the first 20% of the data points, Fold 2 is 20-40% etc. ............................ 64

List of Tables Table 1: Numbers of each category of school ...................................................................................... 27 Table 2: Minimum values of each metric to qualify. ............................................................................ 33 Table 3: Characteristics of the best model for each model type, measured by area under the ROC curve and subject to the minimum constraints of Table 2 ................................................................... 39 Table 4: Difference in training and test set accuracy when the kernel is changed. Each run is otherwise identical, using the parameters shown in Table 3 ............................................................... 46 Table 5: Features that appear in more than one of the selected 6 models ......................................... 57 Table 6: Importance of each feature to the selected Random Forest model....................................... 57 Table 7: Importance of the top 30 features when the parameters selected of the Random Forest model are applied to all features .......................................................................................................... 58 Table 8: Confusion matrix for best model ............................................................................................ 61 Table 9: Properties of best model ......................................................................................................... 61

1

1. Introduction This project is based on the inspection results of all state-funded schools in England and, in

particular, a small subset of these schools that are considered to be ‘stuck’ in a cycle of poor

performance. The aim of this project is to use machine learning to investigate whether it is

possible to predict the future performance of a school that has been repeatedly receiving poor

inspection results. It has been carried out at the Office for National Statistics in Newport, South

Wales.

1.1. School Inspections School inspections are one of the primary measures by which schools are judged. State-funded

schools in England can be inspected at any time, with a minimum of 15 minutes’ notice given

that an inspector is due to arrive at the premises (Office for Standards in Education, 2018). The

inspector is then required to be granted immediate access to everything they deem necessary

to investigate how the school is performing in a range of areas, such as how the school is being

managed and the effectiveness of the teaching in the classroom.

There are different types of inspection, such as ‘full’ Section 5 inspections that last for two

school days and ‘short’ Section 8 inspections that can last for one day. Section 5 ‘full’

inspections are what is investigated in this project and will be referred to simply as an

inspection from this point onwards.

An inspection will result in the school receiving ratings and feedback on a number of criteria.

The headline figure, which is what is considered in this project, is the ‘Overall Effectiveness’

rating. There are four available ratings:

- Category 1: Outstanding

- Category 2: Good

- Category 3: Requires improvement

- Category 4: Inadequate (subcategories: Serious Weakness and Special Measures)

Whilst it is clear that all schools would want to be ‘Outstanding’, the distinction applied in this

work is whether a school is ‘Good or better’ (Category 1 or 2), or ‘less than Good’ (Category

3 or 4).

The frequency with which a school is inspected can vary greatly. An average school could be

inspected once every four years whilst a poorly performing school may be inspected far more

2

frequently. In 2012, government policy stated that it was stopping routine inspections of

schools which have received an Outstanding rating (Fowler, 2012), although would monitor

their data and would inspect if it was deemed necessary. At present, this exemption remains in

place although it has been announced that routine inspections of Outstanding schools will be

reinstated (Department for Education, 2019).

There are different types of state-funded schools in England, such as grammar schools and

comprehensives. Recent government policy has led to the creation of academies. These are

schools where funding is received directly from the government, as opposed to non-academy

schools which receive funding from their Local Authority, who are in turn funded by the

government. The creation of these academies has either been voluntary or forced – any school

rated as Inadequate is legally obliged to become an academy.

From September 2019 onwards, the Education Inspection Framework (Ofsted, 2019) will be

in place, which will change how Ofsted carries out inspections.

1.2. Stuck Schools ‘Stuck’ is a phrase used to describe a school with a repeating pattern of poor inspections. As

used by Ofsted (Spielman, 2018), the criteria for a school to be labelled as stuck are:

- Having had at least four inspections since 2005.

- Every inspection rated ‘less than Good’ (Category 3 or 4).

When a school closes and reopens under a new name, whether to become an academy or not,

the closed school is said to be a predecessor of the opened school. The stuck school criteria

consider all inspections of predecessor schools, so a school can be labelled stuck if it has never

been inspected but its predecessor(s) have and meet the criteria.

Stuck schools are a point that have been focused on as an area to improve by Ofsted, who are

working with the Department for Education to look into them and see what they can do to

improve (Spielman, 2018).

1.3. Ofsted and the Department for Education These are two government organisations that are key stakeholders in this work.

The Office for Standards in Education, Children’s Services and Skills is better known as

Ofsted. They are responsible for carrying out inspections on services providing education and

skills, as well as those that care for children and young people. They then publish the reports

3

of their findings and inform policymakers of the effectiveness of the services. In this work, the

focus is on school inspections carried out and reported by Ofsted.

The Department for Education (DfE) is responsible for the schooling system as a whole. It is

capable of carrying out ‘interventions’ on schools which it believes would benefit from extra

support. These interventions can take many forms, such as increasing funding or providing a

management consultant.

1.4. Tools Used This work was carried out entirely using the Python programming language. The data

processing was carried out using the pandas (McKinney, 2010) and numpy (Walt, Colbert and

Varoquaux, 2011) libraries whilst the Scikit-learn library (Pedregosa FABIANPEDREGOSA

et al., 2011) was used for the machine learning models. Matplotlib (Hunter, 2007) was used to

generate the plots. Sequential Feature Selection was implemented using mlxtend (Raschka,

2018). Oversampling was implemented using imblearn (Lemaitre, Nogueira and Aridas, 2017).

A GitHub repository was used to store the code used throughout this project. It can be accessed

at github.com/crees00/SchoolsData.

1.5. Project Aims The primary aim of this work is to make a model that can accurately predict, for a school with

three consecutive ‘less than Good’ (Category 3 or 4) inspections, whether the next inspection

will be ‘Good or better’ (Category 1 or 2) or ‘less than Good’ (Category 3 or 4). This binary

classification is to be achieved with the greatest accuracy possible.

The second aim is to find the important features in making this prediction and assign an

importance value to them. This is useful in many ways – it allows insight to improve the

accuracy of the model. It also helps to back up the results of the model, providing more weight

to them. It makes the model more accessible and less ‘black box’, allowing the DfE to trust it

more and ensure that evidence is available if a school were to question the model’s output. As

a result, it is important that each variable has an understandable meaning and principal

components cannot be used for modelling.

In order to achieve the two project aims stated above, a dataset must first be created. This

dataset must contain data on every currently open, state-funded school in England, with enough

variables to allow modelling to be carried out. The data must be cleaned and formatted in a

way that is amenable to modelling, with binary class labels generated and added to the dataset.

4

1.6. Process Plan 1. Create dataset from multiple data sources.

2. Label data.

3. Preliminary analysis of data.

4. Iteratively run hyperparameter searches and Sequential Feature Selection to identify

best combination of hyperparameters and features for each model type.

5. Analyse results and present findings to Ofsted and DfE.

5

2. Literature Review 2.1. Previous Analysis of Schools Data Analysis of schools data has been completed in a number of different forms for different

purposes. The prediction of secondary school performance using machine learning has been

carried out in Tunisia (Rebai, Ben Yahia and Essid, 2019). Here, Random Forests and

regression trees were used to identify variables associated with strong exam performance for a

school. School size, the male/female split and class size are among the key variables, of which

it is noted that there is a ‘high non-linearity of the relationships between these key factors and

school performance’. Dummy variables are used for the different geographical regions, but this

is shown to provide few positive results as the information is contained in another variable

(urban/rural). The paper does, however, use relatively few variables which would not appear

to cover the range of inputs required.

The complexity of the link between school level data and exam performance is again noted in

a paper which uses regression trees to generate feature importance for predicting school exam

performance (Masci, Johnes and Agasisti, 2018). The percentage of disadvantaged students,

school funding and student truancy are listed as key variables. It is noted that, while these

variables are linked to exam results, exam results themselves are to be used as a variable in this

project. Student exam results have been predicted with relatively poor accuracy using deep

learning (Tanuar et al., 2018).

Student exam results have also been predicted using, among others, Naïve Bayes, Support

Vector Machines, Random Forests and Logistic Regression as it is recognised that no one

algorithm works best for every problem (Canagareddy, Subarayadu and Hurbungs, 2019). In

this paper, a subset of the variables, such as student age and gender, are removed prior to

modelling because they ‘do not have any impact on the predictions’. Their results appear to

show that their classifier performance decreases as a result, suggesting that variables should be

tested in the model before concluding that they are ineffective and removing them from the

analysis.

As noted above, the uses of machine learning prediction identified in the literature apply to

exam performance as opposed to school inspection performance. They also mainly use

individual student data and make predictions on an individual student level. Some analyses

have been undertaken to characterise stuck schools (Spielman, 2018; Thomson, 2019) but these

do little beyond providing summary statistics.

6

There therefore exists an opportunity to investigate whether machine learning techniques can

be used to predict future inspection performance of schools.

2.2. Machine Learning Models Each open state-funded school in England forms a data point in this analysis, with many

features and a single binary class: Stuck or Escaped. The aim of this work is to establish a

binary classifier that can accurately predict, for an unlabelled school, whether it will be Stuck

or Escaped.

There are many available machine learning models which can be used to build a binary

classifier. It is assumed that the reader is familiar with the relatively well-known techniques

used in this work and, as such, a brief description is provided. The Scikit-learn implementation

of these models (Pedregosa FABIANPEDREGOSA et al., 2011) have hyperparameters which

can be tuned to improve the model performance. The key hyperparameters for each model are

described under sub-headings, together with the options available for them in Scikit-learn. The

remainder of this chapter uses information from the Scikit-learn website heavily, in the

documentation related to the Scikit-learn tool named.

2.2.1. Logistic Regression Logistic Regression is a well known binary classifier, described based on the Machine Learning

Journal article (Lin, Yu and Huang, 2011). In Logistic Regression, the conditional probabilities

describing the possible outcomes for a single data point are modelled using a logistic function:

𝑃(𝑦 |𝒙) ≡ 1

1 + 𝑒−𝑦𝒘𝑇𝒙

where 𝒙 is the data point, 𝑦 is the class label and 𝒘 is the weight vector. Given binary (two

class) training data with 𝑙 points, Logistic Regression minimises the following cost function:

𝑃(𝒘) = 𝐶 ∑ log (1 + 𝑒𝑖 −𝑦𝑖𝒘𝑇𝒙𝑖 ) +

1 2

𝒘𝑇 𝒘 𝑙

𝑖=1

where 𝐶 > 0 is a penalty parameter. In this project, 𝑙2 regularisation as shown above is used,

with the Limited-memory BFGS (L-BFGS) solver selected for its stability.

Logistic Regression is a useful classifier because it is relatively simple, well known and can

provide insight into the relative importance of the input features.

The module Scikit-learn.linear_model.LogisticRegression was used for this analysis.

7

2.2.2. Random Forests Random Forests are a type of ensemble classifier based on the decision tree. A known issue

with decision trees is the possibility of overfitting the data due to the tree depth being too great

(Shalev-Shwartz and Ben-David, 2014), but Random Forests can remove this issue. The

Random Forest algorithm is a perturb-and-combine technique, where a diverse set of classifiers

is created by introducing randomness into the classifier construction. The prediction of the

ensemble is given as the averaged prediction of the individual classifiers.

Each tree in the ensemble is built from a random sample drawn with replacement (a bootstrap

sample) from the training set. When splitting each node during the construction of a tree, the

best split is found either from all input features or a random subset of specified size. The

purpose of these two sources of randomness is to decrease the variance of the model. Random

Forests achieve a reduced variance by combining a diverse selection of trees although this can

increase the bias. The Scikit-learn implementation of Random Forests used was Scikit-

learn.ensemble.RandomForestClassifier. This combines classifiers by averaging their

probabilistic prediction instead of letting each classifier vote for a single class.

2.2.2.1. Number of Estimators The number of trees making up the Random Forest. The accuracy of the model converges to a

limit as the number of trees in the forest becomes large. There is therefore a trade-off between

model accuracy and processing time as larger models will take longer to compute.

2.2.2.2. Maximum Number of Features to Consider The size of the random subsets of features to consider when splitting a node. The lower the

number, the greater the reduction of variance, but also the greater the increase in bias.

2.2.2.3. Criterion The function used to measure the quality of the split at each node in a tree. Either the Gini

impurity or the information gain (entropy) can be used in the model.

2.2.2.4. Bootstrap Whether a subset of the data points or all of the data points are used when adding a tree to the

Random Forest.

2.2.3. Support Vector Machines The following description is largely based on Chapter 5 of Pattern Classification (Duda, Hart

and Stork, 2001) and Chapter 16.5 of Numerical Recipes (Press et al., 2007). Support Vector

Machines are a form of linear discriminant function that specialise in separating data where the

pattern is in a higher dimension.

8

If the data is linearly separable, there exists a hyperplane in 𝑛 dimensions, an 𝑛 − 1

dimensional surface, given by

𝑓(𝒙) ≡ 𝒘. 𝒙 + 𝑏 = 0

that completely separates the training data 𝒙. All that remains is to find 𝒘 (a normal vector to

the hyperplane) and 𝑏 (an offset), then 𝑓(𝒙)will be the decision rule – if 𝑓(𝒙) > 0 then Class=1

and the point is on one side of the hyperplane, if 𝑓(𝒙) < 0 then Class=0 and the point is on the

other side of the hyperplane.

The margin is the perpendicular distance to points nearest to the hyperplane on both sides. The

goal in training a Support Vector Machine is to find the separating hyperplane with the largest

margin. The larger the margin, the better the classifier is expected to perform on unseen test

cases.

The support vectors are two vectors that are equally close to the hyperplane. They pass through

the training samples that define the optimal separating hyperplane and are the most difficult

points to classify as they are nearest to the hyperplane.

An important benefit of the Support Vector Machine is that the complexity of the resulting

classifier is based on the number of support vectors rather than the dimensionality of the

transformed space. Support Vector Machines therefore tend to be less prone to problems of

overfitting than some other methods. As they are based only on the support vectors, outliers

have less impact.

A disadvantage of the Support Vector Machine is that it does not lend itself to providing

probability estimates and it does not indicate the importance of different input features.

The Scikit-learn.svm.SVC module was used to implement Support Vector Machines.

2.2.3.1. C value The optimal decision boundary described above is that which maximises the distance between

the nearest training points and the decision boundary, assuming that the data is linearly

separable. If the data cannot be separated by a hyperplane, then training data points will be

misclassified and the model is said to have a ‘soft margin’. This misclassification is penalised

by the C value in the optimisation process, with a higher C value more strongly penalising

misclassification, with a penalty related to the distance between the misclassified point and the

hyperplane.

9

Setting C to a high value therefore favours achieving perfect separation of the data, at the risk

of generating an overly complex model. Setting a lower C value favours increasing the margin,

which will perform worse on the training data but could be more robust to variation in the test

data.

2.2.3.2. Kernel function Support Vector Machines can utilise the ‘kernel trick’. If an embedding function exists that

maps the n-dimensional feature vectors to a much higher N-dimensional space, it may be

possible that a very non-linear separating surface in the n-dimensional space maps into a linear

hyperplane in the N-dimensional space. This potentially highly complex mapping does not

need to be computed, instead a kernel is computed that could have come from the mapping.

The three kernel functions available in the model are linear, polynomial and the Gaussian radial

basis function (‘rbf’). The polynomial kernel seeks smoother, more global solutions whilst the

Gaussian radial basis function is more influenced by local nearest neighbour effects.

2.2.3.3. Degree of polynomial If a polynomial kernel is selected, this is the order of the polynomial.

2.2.3.4. Gamma value Gamma value is a parameter used in the polynomial and Gaussian radial basis function kernels.

2.2.4. Neural Networks The following is based largely on Chapter 4.4 of Machine Learning (Mitchell, 1997).

A single perceptron unit with a binary threshold takes a vector of input values 𝒙, calculates a

linear combination of these inputs and outputs a 1 if the result is greater than some threshold

and -1 otherwise. The weights 𝑎𝑖 and bias 𝑏 are real valued constants that are learned so that

the perceptron produces the correct (-1 or +1) output for each of the given training examples.

Output 𝑜(𝒙) = { 1 𝑖𝑓 𝑏 + 𝑎1𝑥1 + ⋯ + 𝑎𝑛𝑥𝑛 > 0 −1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

A single perceptron can only express a linear decision surface. Adding in layers of

perceptrons (Figure 1) allows the model to learn a more complex, non-linear decision surface.

Each layer of perceptrons contains a pre-defined number of perceptron units, each of which

can learn different weights (𝒂) and bias (𝑏). The layers of perceptron units are fully

connected – each unit in one layer is connected to all units in the previous layer and all units

in the next layer. The inputs to the second layer are the outputs from the first layer, and so on.

A multi-layer perceptron model may be classed as a feed-forward Neural Network.

10

Figure 1: A multi layer perceptron with 1 hidden layer containing k units (Scikit-learn, no date b)

The backpropagation algorithm learns the weights for a multilayer network. It employs gradient descent to attempt to minimise the squared error between the network output values and the target value for these outputs (the training data labels). For each training example, it applies the network to the example, calculates the error of the network output for this example, computes the gradient with respect to the error on this example, then updates all weights in the network.

The number of hidden units governs the complexity of the decision boundary (Duda, Hart and Stork, 2001) so, if the data classes are highly interspersed, more hidden units are required.

Well trained Neural Networks are capable of accurately classifying highly complex non-linear datasets. A disadvantage of the Scikit-learn implementation is that there is a non-convex loss function with more than one local minimum. This can lead to variation between runs with the same data. Neural Networks also require the tuning of a number of hyperparameters, such as the number of hidden units, layers and iterations.

Scikit-learn.neural_network.MLPClassifier was the implementation of Neural Networks used in this project.

2.2.4.1. Activation function For a perceptron unit, this is a binary threshold function. In this work, the rectified linear unit

(ReLU) was used due to its constant gradient (for positive values), ensuring that the learning

rate was not unnecessarily slow for large values.

11

2.2.4.2. Number of layers The number of hidden layers in the network. It is unlikely that more than three layers would be

required, as it would be necessary to have ‘special problem conditions or requirements to

recommend the use of more than three layers’ (Duda, Hart and Stork, 2001).

2.2.4.3. Number of nodes per layer In order to simplify the parameter search, the decision was taken to have, for a single Neural

Network, the same number of units in each layer. For example, the following setups were

possible: [2,2,2] or [5,5,5,5].

2.2.4.4. Solver Three solvers are available in the Scikit-learn implementation:

- Stochastic Gradient Descent – Updates parameters using the gradient of the loss

function with respect to a parameter that needs adaptation.

- Adam – This is also a stochastic optimiser but it can automatically adjust the amount to

update parameters based on adaptive estimates of lower-order moments.

- L-BFGS – This approximates the Hessian matrix which represents the second-order

partial derivative of a function. It then approximates the inverse of the Hessian matrix

to perform parameter updates.

2.2.4.5. Alpha Alpha is a regularisation term which helps avoid overfitting by penalising weights with large

magnitudes.

2.2.5. Naïve Gaussian Bayes Networks This description is based on ‘The Optimality of Naïve Bayes’ (Zhang, 2004).

Naïve Gaussian Bayes networks are based on applying Bayes’ theorem with the ‘naïve’

assumption of conditional independence between every pair of features given the value of the

class label 𝑦. Bayes’ theorem states the following relationship, given class label 𝑦 and

dependent feature vector 𝒙:

𝑃(𝑦 |𝑥1, … , 𝑥𝑛) = 𝑃(𝑦)𝑃(𝑥1, … , 𝑥𝑛 |𝑦)

𝑃(𝑥1, … , 𝑥𝑛)

Using the naïve conditional independence assumption for all features, this relationship is

simplified to:

12

𝑃(𝑦 |𝑥1, … , 𝑥𝑛 ) = 𝑃(𝑦) ∏ 𝑃(𝑥𝑖 |𝑦)

𝑛 𝑖=1

𝑃(𝑥1, … , 𝑥𝑛)

As 𝑃(𝑥1, … , 𝑥𝑛 ) is constant given the input, we can use the following classification rule:

𝑃(𝑦 |𝑥1, … , 𝑥𝑛 ) ∝ 𝑃(𝑦) ∏ 𝑃(𝑥𝑖 |𝑦) 𝑛

𝑖=1

⇒ 𝑦 ̂ = 𝑎𝑟𝑔𝑚𝑎𝑥 𝑦

𝑃(𝑦) ∏ 𝑃(𝑥𝑖 |𝑦) 𝑛

𝑖=1

Maximum A Posteriori estimation can then be used to estimate 𝑃(𝑦) and 𝑃(𝑥𝑖 |𝑦), with 𝑃(𝑦)

then being the relative frequency of class label 𝑦 in the training set.

In the Gaussian Naïve Bayes algorithm used, the likelihood of the features is assumed to be

Gaussian:

𝑃(𝑥𝑖 |𝑦) = 1

√2𝜋𝜎𝑦2 exp (−

(𝑥𝑖 − 𝜇𝑦 ) 2

2𝜎𝑦2 )

The parameters 𝜎𝑦 and 𝜇𝑦 are estimated using maximum likelihood. There are no

hyperparameters to tune in this model.

The conditional independence assumption is rarely true in most real-world applications.

Despite this, there are numerous scenarios where these models have been proven to be

effective. They require a small amount of training data to estimate the necessary parameters

and are extremely fast compared to more sophisticated methods. The decoupling of the class

conditional feature distributions means that each distribution can be independently estimated

as a one-dimensional distribution. This then helps avoid problems with the curse of

dimensionality.

The module used in this project was Scikit-learn.naive_bayes.GaussianNB.

2.2.6. K-Nearest Neighbours The K-Nearest Neighbours algorithm assumes that all data points correspond to points in n-

dimensional space (Mitchell, 1997). The nearest neighbours of an instance are defined in terms

of a distance measure as specified by the user. When k is 1, the algorithm assigns a label to the

test point corresponding to the nearest training point in the space. For larger values of k, the

algorithm assigns the most common class label of the k nearest training points.

Unless weightings are applied, all features are treated equally. As such, if there are large

numbers of unimportant features in the model, the classification can be dominated by these

13

unimportant features. Also, each test point must be compared to every training point so this

can be computationally intensive.

The implementation of K-Nearest Neighbours used was Scikit-

learn.neighbors.KNeighborsClassifier.

2.2.6.1. K value The larger the value of k, the larger the space being investigated and the less prone to error due

to noise.

2.2.6.2. P measure The order of Minkowski distance to measure the distance between points with. For example,

p=1 is the Manhattan distance and p=2 is the Euclidian distance.

2.3. Methods for Refining the Models Aside from selecting a model type and tuning its parameters (Section 2.2), there are many ways

in which a model can be further improved. The methods used in this project are described

below.

2.3.1. Iterative imputation The models required complete sets of data to function, yet a substantial quantity of the input

data was missing (Section 3.3.1). It was therefore required to use data imputation. The

imputation method selected was the IterativeImputer from Scikit-learn. This models each

feature with missing values as a function of other features, and uses that estimate for

imputation. This is achieved by fitting a regressor to the input data and using this to predict the

missing values in an iterative, round-robin fashion. This imputation method was selected to

provide a more ‘accurate’ prediction of the missing value than simply imputing the mean or

the mode, although this has not been verified. It also maintains a level of variance within the

data.

2.3.2. Sequential Forward and Backward Selection SFS and SBS are described based on the information on the mlxtend website (mlxtend, no

date). This technique is a greedy method of selecting the optimum combination of features to

use in the model. For forward selection, the selected feature set is initially empty and the

available features are input into a list. In each iteration, each feature in the input list is added

to the selected feature set. The model is trained and tested (using cross validation) on the new

variable set, the score is recorded and the feature is removed. When all features in the input list

14

have been tested, the feature that resulted in the best score is removed from the input list and

added to the selected feature set. This is continued until all features have been selected if an

early stopping criterion is not used. Backward selection works in the reverse fashion – the

selected feature set initially contains all features and one feature is removed in each iteration.

The scoring measure can be selected from standard measures such as model accuracy and the

area under the ROC curve. This is computationally intensive as the model must be trained and

tested a large number of times for each feature selection run.

In this work, the module mlxtend.feature_selection.SequentialFeatureSelector was used to

implement sequential feature selection in the SFS.py script.

2.3.3. Recursive Feature Elimination Recursive Feature Elimination is another method of selecting features and is described in the

Scikit-learn documentation. This is achieved by recursively removing the features which score

lowest in a Logistic Regression until the number of required features is reached. It was achieved

using the Scikit-learn package Scikit-learn.feature_selection.RFE.

2.3.4. Oversampling When using different subsets of the data, with different criteria for positive and negative labels,

it is possible that there can be a large difference between the sizes of the two classes for a binary

classifier. As a result, the classifier can be distorted by having few training points in the less

populous ‘minor’ class. This can be mitigated through the generation of synthetic cases of the

minor class in a technique called oversampling.

An oversampling technique called SMOTE (Chawla et al., 2002) is used in this work. The

following description is based on the documentation of imbalanced-learn (Lemaitre, Nogueira

and Aridas, 2017) which provided the implementation of SMOTE used. This module is

imblearn.over_sampling.SMOTE. The regular SMOTE algorithm takes a point of the minor

class, selects one of its nearest neighbours in the same class and generates a new point by linear

interpolation between the two existing points, giving the new point the label of the minor class.

The result of using this technique is that the training set will have the same number of each

class in it. The more populous ‘major’ class of the training set and all of the test set remain

unchanged.

15

2.4. Measuring Model Effectiveness A key consideration when building a machine learning model is how the results from the model

should be assessed to identify which are most effective and how they can be improved.

2.4.1. Cross validation In order to provide an indication of the accuracy of the model, it must be tested on previously

unseen data. As a result, for each run, a subset of the data should be removed and held as a test

set, the remainder being the training set. The model can then be trained solely on the training

set and its performance judged on how the model performs when applied to the test set. The

reported performance of the model is then dependent on the training/test split used – some

splits would result in better performance on the test set than others. To reduce this effect, 𝑘

fold cross validation can be implemented as described in section 9.6.2 of Pattern Recognition

(Duda, Hart and Stork, 2001):

The training set is randomly divided into 𝑘 disjoint sets of equal size 𝑛/𝑘, where 𝑛 is the total

number of data points. The classifier is trained 𝑘 times, each time with a different set held out

as a test set. The estimated performance is the mean of these 𝑘 scores. The number of folds

traditionally used is 𝑘 = 10.

The Scikit-learn.model_selection.KFold module was used to implement cross validation

because the full implementations of cross validation in Scikit-learn were not compatible with

the way the models and data had been set up. Setting up the different training and test sets is

demonstrated in the oneFullRun(..) function in genericModelClass.py as shown in Section 5.2.

This generates output for each of the 𝑘 folds separately, so separate post processing is required

to calculate the average output.

2.4.2. Accuracy Accuracy is the percentage of the test points that were correctly classified by the model.

2.4.3. Precision and recall The confusion matrix (Figure 2) summarises a classifier’s performance.

True

1 0

Predicted 1 True positives False positives

0 False negatives True negative Figure 2: Confusion Matrix

Precision shows what proportion of the points classified as class 1 are actually in class 1:

16

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠

𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠 + 𝐹𝑎𝑙𝑠𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠

Recall shows what proportion of the points that are labelled as class 1 are predicted to be in class 1:

𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠

𝑇𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒𝑠 + 𝐹𝑎𝑙𝑠𝑒 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝑠

These measures are particularly useful when there is an uneven split between the two values.

The F1 score is the harmonic mean of precision and recall, giving equal weight to each:

𝐹1 = 2 ∗ 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ∗ 𝑅𝑒𝑐𝑎𝑙𝑙 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙

The confusion matrices, precision and recall were calculated using the classification_report and confusion_matrix classes within Scikit-learn.metrics.

2.4.4. ROC curves An ROC curve (see examples in Figure 10 in Results section) is built upon a model’s false

positive rate and true positive rate – the proportion of points that the model has classed as class

1 that are actually in class 0 and 1 respectively. By calculating these rates for a range of

classifier threshold values, a Receiver Operating Characteristic (ROC) curve can be plotted. If

the threshold is set high enough, all points are predicted to be in class 0, so both rates are zero,

and the opposite holds for when the threshold is set to a minimum value. The method of

obtaining the values for ROC curves for the different model types is set out in Chapter 3 of

‘ROC Analysis of Classifier in Machine Learning: A Survey’ (Majnik and Bosnic, 2011).

The area under an ROC curve can be calculated, for which a perfect classifier would have a

value of 1. An Introduction to ROC Analysis (Fawcett, 2006) states that “The AUC has an

important statistical property: the AUC of a classifier is equivalent to the probability that the

classifier will rank a randomly chosen positive instance higher than a randomly chosen negative

instance.” A classifier which assigns classes at random would expect to achieve an AUC of

0.5. Therefore, for a model to be useful it must have an AUC score exceeding 0.5.

The datapoints and AUC results for the ROC curves plotted in this report were generated using

Scikit-learn.metrics.roc_curve and Scikit-learn.metrics.roc_auc_score respectively.

17

3. Input Data and Pre-Processing A key part of this project was assembling the data into a useable format compatible with the

machine learning models.

3.1. Datasets Available This project used publicly available data provided by Ofsted and the DfE.

3.1.1. Ofsted data Ofsted provided a record of every inspection that had occurred between 1st September 2005

and 31st August 2018, of which there are approximately 100,000. This data came in the form

of .csv files. The period until 31st August 2015 was covered by a single .csv file provided by

Ofsted, with the remaining data available in a separate file per term from the government

website. As stated in Section 1.1, these 100,000 inspections are neither evenly distributed

between schools nor by time.

Each inspection had around 50 variables, the key ones are shown below.

• URN – Each school has a Unique Reference Number which is the primary identification

of the school. If a school reopens under a different name, or becomes an academy, it is

given a new URN.

• LA/ESTAB Number – The Local Authority / Establishment number is another unique

identifier that was used in the cases where URN was not available.

• Inspection Number – This is the unique reference for the inspection.

• Overall Effectiveness rating – This is the category (1/2/3/4) that is provided.

• Inspection Start Date – This is used for providing the year of the inspection.

3.1.2. DfE data The data provided by the Department for Education had been acquired by the ONS before the

start of this project.

A list of the variables used is provided in the appendix (Section 9.1).

3.1.3. Get Information About Schools data This service, formerly known as EduBase, provides a list of all state-funded schools which are

currently open in England. This analysis focuses solely on these schools, although any

18

inspection that a current school’s predecessors had are considered in the current school’s

inspection history.

3.1.4. List of Academies This list of schools made a link between a current school and any predecessor school(s) it had.

This could be used in combining the inspection histories of schools.

3.2. Data Stitching An initial aim of the project was to create a single large dataset containing all relevant data for

each open school in England. The list of all currently open schools was used to ensure that

there was one row for each school, then columns were added in batches. The

combineInspData.py, creatingAMonster.py, GDIhelper.py and genericDataIn.py scripts were

used to carry out the following work.

A series of helper functions were created to perform the tasks. For each data source, a dictionary

was filled with the .csv file paths, the column names containing each data type and a selection

of other required inputs. The dictionaries were then passed sequentially to the helper functions

to add columns to the existing dataframe. During this process, the data types were corrected.

Some data was in separate .csv files for different subsets of the data for a given year, with

inconsistent names for equivalent columns. Here, the corresponding column names of the

required columns were identified and placed in lists. The lists of the two input dataframes were

then used to merge the individual columns to provide a combined dataframe that could be

joined to the primary dataframe. A similar operation was carried out on the examination

performance data which is described in section 4.1.3.

The final dataset contained one row for each open, state funded school in England. The URN

(Section 3.1.1) was the unique identifier, followed by columns for each selected variable. A

‘Class’ column, indicating the binary classification assigned to the school (Section 4.5) was

added as a final column.

3.3. Data Cleaning Data retrieved from different sources needed to be cleaned to ensure that information was not

lost whilst ensuring that all data input to the models would be compatible with the models.

3.3.1. Missing data Between the data sources there was a range of levels of data completeness. The completeness

of each variable was assessed to ascertain which had sufficient data to be useful in the analysis.

This was partly carried out using the makePickColsToUse(..) function within

19

pickColsToUse.py. Some variables were replicated between different data sources, with

different levels of completeness. In this case, the more complete source was used.

The data stitching technique also meant that if a school had a predecessor, the predecessor URN

was not used to find data from when the predecessor was open. The cleaning process also

removed some values that were incorrect, leading to more missing data.

3.3.2. Data types All data was required to be in numeric form, either as a float or an integer, to be compatible

with the models. The input data contained multiple types and formats, often with multiple types

within a single column. A selection of helper functions were required to deal with specific

formats such as percentages and currency. Where data was in an inappropriate format, such as

text in a numerical variable, this was replaced with the blank value np.nan. This was largely

carried out using the GDIhelper.py and genericDataIn.py scripts.

3.3.3. Normalising The data values used range in magnitude from large school budgets to proportions smaller than

one. They also are spread over different distributions. Some of the models used in this analysis,

such as Support Vector Machines, are not scale invariant, so a feature with larger values would

have a disproportionate impact on the results given. Not normalising the data could also lead

to numerical difficulties during the calculation as kernel values usually depend on the inner

products of feature vectors (Hsu, Chang and Lin, 2008).

In order to allow the models to work effectively, the data in all variables used in the analysis

was normalised using the normalise(..) and normaliseSDcol(..) functions in pickColsToUse.py.

This resulted in each variable having a mean value of 0 and a standard deviation of 1.

3.3.4. Imputation The models used in this analysis required that there were no missing values. As discussed in

section 3.3.1, the input data contained many missing values, with no single variable being

complete. In order to run the models using this data, the missing data had to be imputed.

The imputation method described in Section 2.3.1 was implemented as shown in the extract

from pickColsToUse.py below.

20

Imputing data will clearly introduce inaccuracy into the dataset because it has been generated

by the imputer rather than being a measured value. It is expected that the imputation would

cause the results to be worse than those obtained if the complete, original data were available.

This risk, however, is accepted due to the necessity of passing complete input data to the

models.

21

4. Machine Learning Implementation 4.1. Feature Generation As well as the features immediately available in the input data, a selection of extra features

were created based on the existing features.

4.1.1. Categorical data Categorical features must be converted to numerical values for use in the models, which was

achieved using fixCategoricalcols(..) in pickColsToUse.py. For some variables, such as

‘Boarding’ and ‘HasBoys’, this was achieved by grouping categories together to give a binary

output. When there were more than two outcomes for a variable, ‘one-hot encoding’ was

implemented using the pandas get_dummies() function. This ensured that all values were

treated equally by the model and would not introduce any unwanted effects.

4.1.2. Changes over time Many of the datasets used in this work are published annually, with data available for around

5 to 10 years. This was accounted for by generating variables that were a difference between

the current value and the corresponding value a specified number of years ago. If this was done,

it was added as a separate variable to the original.

4.1.3. Performance data School examination performance in England is published for pupils sitting examinations at the

end of Key Stage 2 (age 11), Key Stage 4 (age 16) and Key Stage 5 (age 18). Schools are

generally either for pupils aged up to age 11 (primary) or for older children (secondary).

Conducting separate primary and secondary analyses was beyond the scope of this project, so

a means of comparing the examination performance of primary and secondary schools was

implemented using the fixPerfCol(..) function in genericDataIn.py as described below.

22

The Key Stage 2 data selected shows the percentage of children that reach the ‘expected

standard’ in reading, writing and maths. Two main Key Stage 4 metrics were available for

assessing school examination performance: ‘Attainment 8’ and ‘Progress 8’. Attainment 8 is

an integer measure of a pupil’s performance across 8 core subjects and Progress 8 is this same

measure but relative to a pupil’s previous performance (Department for Education (DfE),

23

2016). Whilst the Progress 8 measure arguably provides a better indication of the school’s

performance, it was not selected because there is not a comparable Key Stage 2 measure which

measures improvement. The Key Stage 4 measure selected shows the average Attainment 8

score per pupil.

For each set of data, the applicable schools were ranked by their performance. The performance

rankings were then converted to percentages, so the median school would receive a score of

50% and the best would receive 100%. The new column added to the main dataset was then

the percentage ranking score for that school, whether it was for Key Stage 2 or Key Stage 4. If

a school had results for both Key Stage 2 and Key Stage 4, a mean was taken of their two

scores.

4.2. Initial Feature Selection The initial data sources contained over one thousand variables. Many of these were duplicates,

variants of others or not useful. When these were removed, there were still more features than

were necessary for a model with close to 22,000 data points. The initial feature selection was

carried out after consultation with colleagues at the ONS who have worked with these datasets

previously. The strategy used was to keep more features than were expected to be optimal, then

to narrow down the feature space once the initial models had been run.

4.3. Stuck School Labelling The dataset created was unlabelled – it did not contain the school classification labels that

would be needed for modelling. This section describes the process of adding the labels to the

dataset. Selecting the best definition of the classes to be classified by the model would be

important to the success of the project. Two different labelling methods were used, both of

which were binary.

4.3.1. Options for labels The initial basis of the project was to investigate schools classed as ‘Stuck’ by Ofsted (Section

1.2). This group of schools had already received extra focus from the authorities and analysis

had been carried out on them, such as the Ofsted Annual Report (Spielman, 2018). This

definition, however, had some clear flaws.

The first flaw in the existing stuck school definition is that the start date of the time window in

which inspections count towards a school being stuck is fixed at 1st September 2005. As time

passes, this window of time will increase and the effective meaning will change.

24

A further issue is that a stuck school can only be termed stuck if it has had zero ‘Good or better’

inspections since the starting date. This means that if a school had one ‘Good’ inspection in

September 2005 then it could never be classed as stuck, even if it subsequently received ten

consecutive ‘Inadequate’ (category 4) ratings. Given the expected variation within inspection

results, a single ‘Good’ inspection in 13 years is not unlikely if a school is inspected regularly,

even if it generally receives poor ratings. This would prevent a school from being labelled as

stuck even if it would fit into the thinking behind the stuck definition.

4.3.2. Updated definition of Stuck schools Given the flaws in the existing stuck schools definition, a new definition was agreed based on

input from the Department for Education. This new definition states that, to be stuck, a school

must have:

- At least four previous inspections (including those of predecessors).

- An Overall Effectiveness rating of ‘less than Good’ (category 3 or 4) in each of its four

most recent inspections.

4.3.3. Data selection and labelling With this new definition of stuck schools, it was decided that the modelling should proceed by

dividing the data into three groups:

- Currently Stuck schools (Class=1): As defined in Section 4.3.2. Examples of ratings of

schools that would fall into this category (from first inspection to most recent):

o 4,4,3,4,3

o 2,2,4,4,4,3

- ‘Escaped Stuck’ schools (Class=0): Schools that have had at least 3 consecutive ‘less

than Good’ inspections, with all inspections following this (minimum of one) rated as

‘Good or better’. For example:

o 4,3,4,2

o 2,2,3,3,3,2

o 2,2,4,3,4,4,1,1,2

- All other schools: These schools do not fall into either category and so were removed

from the dataset.

25

These labels allowed a specific question to be asked: If a school has three consecutive ‘less

than Good’ inspections, can we predict whether its next inspection will also be ‘less than

Good’?

The new class definitions clearly defined the training data labels required and the subset of

schools that the model could be used to predict in future. They also ensured that there was no

overlap between the classes and resulted in a relatively even split between the two classes. A

significant drawback of this approach is that a large proportion of the input data was being

removed and not used in the modelling.

4.3.4. Labelling method The Stuck school definitions of Section 1.2 and 4.3.2 needed to be applied to the dataset. The

selected definition could then be appended onto the dataset as a binary ‘Class’ column which

could be used as the data labels for training and testing the models.

The list of schools which fit into either definition of stuck was not available, so the

schoolClass.py script was written to identify which schools met the different criteria. This was

achieved by first defining an Inspection class in Python and generating an instance for each of

the ~100,000 inspections in the Ofsted data with the attributes listed in Section 3.1.1. Next, a

School class was defined and an instance generated for each school (open or closed) whose

URN matched the URN attribute of an Inspection, populating the School instance with

information from the dataset.

In order to incorporate inspections from predecessor schools, the predecessor URNs were first

identified and added to the relevant School instance using addPredecessorURNsFromDF(..).

For each School, the addPredecessorInspections(..) function was called recursively to work

through the predecessor schools identified and read in all of their Inspection instances. This

ensured that, even if a school’s predecessor itself had a predecessor, then the inspections would

be counted correctly. This then made it easy to work out which schools were stuck (old

definition) using calcStuck(..).

26

Further functions could then identify which schools were open, sort the inspections by year and

carry out further analysis, such as plotting the total inspection history of each open school as

shown in the following section (Figure 3).

4.4. Data Summary A summary of the number of schools in each category is shown in Table 1.

27

Table 1: Numbers of each category of school

All inspected

schools

All open schools Class=1

‘Stuck’

Class=0

‘Escaped’

32131 21943 805 908

The overall inspection histories of all schools are shown in Figure 3. The figure shows, for

example, that approximately 3,500 schools have an inspection history of 3 ‘Good or better’

inspections and 0 ‘less than Good’ inspections.

Figure 3: Frequency of each overall inspection history

A series of histograms are plotted in Figure 4 from classComparisonForEachCol.py. Each plot

shows three overlaid histograms which represent the distribution of values between the classes,

with the orange 'Escaped' bars being partially transparent to see the blue 'Stuck' bars behind.

Variable descriptions are provided in Section 9.1. The histogram y-axes are normalised to show

the relative frequencies of each class as opposed to absolute values. They are plotted before the

dataset was normalised so that the x-axis is on the original scale of the variable. For binary

values, 1 represents True and 0 represents False.

Some basic, generalised observations from the plots are as follows:

28

- Both stuck and escaped schools have much greater rates of pupils eligible for free

school meals, low achievement in key stage 1 and English as an additional language

than the average English state-funded school.

- A much higher proportion of stuck schools are secondary than for escaped schools,

which are both above the overall average. The same applies for academies.

- Stuck schools tend to have generally worse exam performance than escaped schools,

with both classes having generally inferior performance to all other schools.

- Stuck schools tend to have a lower Pupil:Teacher ratio than escaped schools.

- Stuck schools tend to have a greater pupil absence rate than escaped schools. Both

groups have frequencies of absence of 0.05% or greater that are considerably higher

than average.

- Stuck schools tend to have more pupils than escaped schools.

- Stuck schools tend to have had a greater increase in supply staff spend since 4 years

ago than escaped schools.

- There is not a clear link between teacher pay and a school being stuck or escaped.

29

Figure 4: Histograms showing distribution between 'Stuck' schools, 'Escaped' schools and schools that fall into neither category

30

4.5. Principal Component Analysis A principal component analysis was run with all input features using Scikit-

learn.decomposition.PCA and the results shown in Figure 5. The plot shows that 25 principal

components are required to capture 80% of the variance and 39 principal components are

required to capture 90% of the variance.

Figure 5: Principal Component Analysis - Cumulative explained variance versus number of principal components used

Stuck, escaped and all other schools are plotted on Principal Component 1 vs Principal

Component 2 axes in Figure 6 using plotClassesOnPCA(..) in PCAallFeatures.py. Note that

points are plotted in the order shown, with the grey points ‘beneath’ the other points if they are

coincident, and the orange points for escaped schools are on top.

The scatter plot shows that the classes are heavily intermingled in terms of principal

components, suggesting that building a classifier to separate them could be challenging. The

first two principal components do, however, only account for 18% of the variance in the data

so there is a large quantity of information that is not contained in the plot.

31

Figure 6: The results of Principal Component Analysis - the two classes and the ‘other’ remaining schools that fall in neither class are plotted on Principal Component 1 vs Principal Component 2 axes

32

5. Results During this work, a series of experiments and sub-experiments were undertaken, with the

overall aim of finding the optimum predictive model for this dataset. The model components

investigated in this chapter are all interlinked. For example, the best hyperparameters for a

Support Vector Machine model will depend on the features input to the model whilst the best

choice of features for a Support Vector Machine model will depend on the hyperparameters

set. The approach taken was therefore iterative.

5.1. Assessment Methods Used The number of cross validation folds (Section 2.4.1), 𝑘, used in this project is 5. This provides

a balance between providing enough folds to smooth out the results sufficiently and the

processing time required to effectively run each model five times. All results in this report have

used 5-fold cross validation, where the result provided is the average of the individual scores

of the 5 folds.

Many different metrics were considered when assessing the performance of a single model

(Section 2.4). The simplest measure, accuracy, was used heavily as this is the basic requirement

of a classifier. It was noted that the split of the data between the two classes was 47:53, so a

reported accuracy value would not be misleading due to the class sizes being imbalanced.

When accuracy values were not at a sufficiently high level, the area under the ROC curve was

considered more strongly. This is because, for models which allow tuning such as Random

Forests and Logistic Regression, the operating point of the model could be changed. For a

model with low accuracy, the class boundary decision threshold could be increased. This would

reduce the number of schools predicted to be in the positive class (and therefore reduce recall),

but precision would increase as a higher proportion of these positive predictions would be

correct. It was noted that there is a clear correlation between model accuracy and area under

the ROC curve, so choosing between the two measures had a relatively small impact on the

results.

For some of the measures of performance, the results could be misleading. For example, in the

case with the original definition of stuck schools, a model would achieve over 98% accuracy

by simply predicting every school not to be stuck. In the following results, therefore, minimum

values for a selection of measures are set (Table 2) to reduce the chances of an unhelpful

classifier being selected.

33

Table 2: Minimum values of each metric to qualify.

Accuracy AUC F1 F0 Recall 1 Recall 0 Precision 1

0.6 0.6 0.25 0.25 0.1 0.1 0.1

5.2. Computing Set up for Modelling The models were set up and run using genericModelClass.py. First, an instance of ModelData

was generated for a run. This contained the data, carried out the train/test split and ran

oversampling and/or recursive feature elimination if specified. Next, an instance of the

specified model type, for example RandomForest, was generated. This inherits from the Model

parent class which has numerous get..(self) and set..(self) methods as well as plotROC(self).

Each instance of a child class of Model is initialised with the ‘dataName’, a ModelData instance

and a dictionary of run parameters.

ModelData instances were stored in modelDataDict and Model instances in modelDict. The

naming of the ModelData and Model instances with generated, descriptive names ensured that

each ModelData instance was only generated once and could be reused, avoiding wasting

computing resource. The fitModel(self, [params]) method for each Model instance was only

called once it had been checked that it had not previously been run.

The runAGroup(..) function was set up to work through each combination of the data inputs

(RFE options and whether or not to use oversampling) for each model type and pass inputs to

runsForModels(..). In the extract of runAGroup(..) below, the model postprocesses and clears

the modelDict if it has greater than 400 entries to save memory.

34

Every combination of the parameter dictionary of lists was generated using itertools in the two

lines of code indicated below (Rees, 2017) in runsForModels(..), and a random subset (using

random.sample) of the selected size was passed sequentially to oneFullRun(..).

35

The oneFullRun(..) function then generates the data name, the train/test sets for cross

validation, instances of ModelData and Model and fits the model to the data. The regular

expression re module was used here.

36

Once the runs were completed, the individual .csv files generated were recombined into one

file in analyseAvgCSVs.py. Columns for the different parameters and run settings (all identified

37

from the run names using regular expressions) were added to allow analysis. The plots were

then generated using plotBestModel.py.

5.3. Experiment 1: Finding the optimum machine learning classification model The optimum machine learning classification model used depended on many aspects, each of

which were investigated in the following sub-experiments.

5.3.1. Experiment 1.1: Finding the best model type There are a range of well known models available for binary classification, of which six were

selected for consideration at the outset (Section 2.2). In order to compare the effectiveness of

the models, each model type was first optimised as described below and illustrated in Figure

7.

An initial hyperparameter search was carried out for each model type, based on an initial

selection of approximately half of the available variables selected to be expected to be of high

importance to the model. The set of hyperparameters for each model that resulted in the highest

average accuracy score was selected. These six models were then run through Sequential

Forward and Backward Selection. For each of the six models, the run (forwards or backwards)

that led to the highest accuracy score was used. A further hyperparameter search was then

carried out for each model. This used the set of features that had resulted in the highest accuracy

from Sequential Forward or Backward Selection for that model and searched a narrower

hyperparameter search space based on the results of the initial hyperparameter search.

38

Figure 7: Process for selecting optimum model for each model type

The performance of the six optimised models was then compared across different measures to

ascertain which was the most effective.

5.3.1.1. Results The best performing model for each of the six model types was selected. The models selected

were the models with the highest area under the ROC curve, subject to minimum threshold

scores as specified in Table 2.

Figure 8 shows that the model with the highest area under the ROC curve was the K-Nearest

Neighbours model, which also had the highest accuracy. Neural Networks, Support Vector

Machines and Random Forests all have a similar accuracy whilst the Logistic Regression model

has a higher recall for class 1 and a lower recall for class 0. The K-Nearest Neighbours model

has the highest precision for class 1, whilst the Neural Network and Logistic Regression have

the highest precision for class 0. Gaussian Naïve Bayes has the lowest score for five of the six

measures.

Final model features and hyperparameters

Model used for analysis

Feature set selected from second round of SFS or SBS. Narrower range of hyperparameters

Final hyperparameter search

All features. Best hyperparameters from refined search

Sequential Forward/Backward selection

Feature set selected from SFS or SBS. Narrower range of hyperparameters

Refined hyperparameter search

All features. Best hyperparameters from initial search

Sequential Forward/Backward selection

Initial feature set. Wide hyperparameter range

Initial hyperparameter search

39

Figure 8: Performance of the best version of each model type. Note that different measures of the same six models are plotted, sorted in order of decreasing area under the ROC curve score.

The parameters used in each model type are shown in Table 3.

Table 3: Characteristics of the best model for each model type, measured by area under the ROC curve and subject to the minimum constraints of Table 2

K-Nearest

Neighbours

Neural

Network

Support

Vector

Machine

Random

Forest

Logistic

Regression

Gaussian

Naïve Bayes

No RFE or

oversampling

No RFE or

oversampling

No RFE or

oversampling

No RFE or

oversampling

Oversampling

used

Oversampling

and RFE with

5 best features

selected

28 Features 19 Features 7 Features 8 Features 9 Features 10 Features

(before RFE)

K = 41 5 layers C = 1.455 12 estimators

P = 1 12 nodes per

layer

RBF kernel 6 features

Adam solver Gamma =

0.244

Entropy

criterion

Alpha =

0.0667

Bootstrapping

used

The ROC curves for the six models in Table 3 are plotted in Figure 9. The K-Nearest

Neighbours model’s ROC curve is above the other models’ curves for almost all values of

40

threshold. All curves exhibit a similar shape, although the Logistic Regression and Gaussian

Naïve Bayes models have a much greater false positive rate than the other models for lower

values of true positive rate.

Figure 9: ROC curve for model with highest area under the ROC curve value for each model type. The lines plotted are the mean scores over the 5 folds of cross validation.

For each model in Table 3, separate ROC curves are plotted in Figure 10. The mean values

correspond to those plotted in Figure 9. The K-Nearest Neighbours model has the greatest range

in performance between folds, with a wider band of one standard deviation from the mean. The

K-Nearest Neighbours, Neural Network, Random Forest and Support Vector Machine models

are all relatively consistent up to a true positive rate of around 0.4, although there is a wider

41

spread for the Neural Network. The curves for K-Nearest Neighbours are smoother because

there are fewer (k) threshold values from which to plot the curve.

42

Figure 10: ROC curves for models in Table 3. Each of the five cross validation folds is plotted, along with a mean and the range of one standard deviation

An alternative measure of the best classifier is to select the classifier, subject to the minimum

thresholds in Table 2, with the best precision for data points in class 1: Stuck schools. The

results in Figure 11 show that the Support Vector Machine model has a significantly higher

precision than the other models. It has the lowest recall for class 1 of any of the models, with

a high recall for class 0. It is therefore predicting that a much higher proportion of schools will

be in class 0 than in class 1.

Figure 11: Precision for Stuck class - The best model from each model type in terms of precision for identifying stuck schools. The same six models are plotted in each chart, sorted in order of precision score.

5.3.1.2. Conclusion The performance of the best model of five of the six model types is similar, with Gaussian

Naïve Bayes having an appreciably lower performance. K-Nearest Neighbours produces the

model with the highest area under the ROC curve and the best accuracy. If a high precision for

predicting stuck schools is required, then the Support Vector Machine model plotted in Figure

11 is best. Gaussian Naïve Bayes is again the poorest performer by this measure.

The ROC curve of the K-Nearest Neighbours model in Figure 9 shows that it is the best

performing classifier because its ROC curve is above the other curves throughout. For K-

Nearest Neighbours, the Neural Network, Support Vector Machine and the Random Forest,

there is little to choose between the classifiers if the thresholds are set to achieve a true positive

rate of 0.4.

If the alternative measure of precision for stuck schools is used, a Support Vector Machine

model is superior to the other model types. It only identifies 40% of the schools that will

43

become stuck but around 88% of those that it selects will become stuck. If a smaller, higher

confidence selection of schools is to be identified as likely to become stuck then this model is

superior to the others.

5.3.2. Experiment 1.2: Finding the optimum model hyperparameters Within many of the machine learning models available there are many parameters that can be

adjusted to finetune the working of the model. When running these models, it is important to

consider that the default settings may not provide the best results for the input data supplied.

The parameters for consideration in this experiment, selected based on their perceived

likelihood of improving the model and availability in the Scikit-learn model implementation,

are described for each model in Section 2.2. Logistic Regression and Gaussian Naïve Bayes

did not have any tuneable parameters so do not feature in this experiment.

In order to find the optimum parameters for each model type, a series of parameter searches

were conducted, as described in the following paragraphs. Figure 7 shows how these steps

fitted into the overall process used.

Round 1

For categorical parameters, each value of the parameter was added to the parameter search

space. For each numeric parameter, an initial maximum and minimum value were selected

based on judgement from past experience and literature. This range was made wide enough

that the optimum value was considered highly likely to fall in the range. Between these two

values were then added a series of intermediate values on either a linear or logarithmic scale,

depending on the values involved.

For each model type, all possible combinations of the run parameter values were generated.

Due to the finite computer processing time available, not all combinations could be

investigated. As a result, for each run, the number of parameter combinations to investigate,

e.g. 50, was input. The program would then run the model with 50 different randomly selected

combinations of the parameters. This effectively implemented a subset of the runs available

through a full grid search. It was decided that this would allow a wider investigation of the

parameter space than a grid search, increasing the probability of finding optimal parameters

than a full search through a narrower search space. It did, however, lead to the possibility of

missing the optimum combination of parameters from within the ranges selected.

44

For each combination of parameter values selected, the model was run and its cross validation

results were recorded.

Rounds 2 and 3

For each parameter, the accuracy and AUC scores for the most effective models from the

previous round were plotted against the parameter values. From inspection of these plots, the

range of values which was most likely to contain the optimum was identified. The parameter

values in the search space were then adjusted, reducing the range and increasing the resolution

within the range. The parameter search was then run again on the narrower search space.

5.3.2.1. Results The results of the final hyperparameter search for each of the four models with tuneable

parameters are shown in the following figures. They are not subject to the minimum threshold

values specified in Table 2. Recursive Feature Elimination and Oversampling are not used in

this section. The final selected parameters for each model type are shown in Table 3.

45

Figure 12: Variation of AUC and Accuracy with different hyperparameters for Support Vector Machines using features listed in section 9.2

46

The Support Vector Machine results in Figure 12 show that accuracy and AUC appear to peak

at around a C value of 10. The gamma value has been heavily investigated at values less than

0.1 but there appears to be a peak in AUC between 0.1 and 1. This is not reflected in the values

of accuracy, where the peak is between 0.01 and 0.1. The higher AUC values with gamma

greater than 0.1 largely coincide with C values between 1 and 10. The radial basis function

kernel outperformed the polynomial kernel of degree 1, 2 and 3. Of the polynomial kernels, the

best performance was with a polynomial kernel of degree 2. This was investigated further, as

shown in Table 4. Higher order polynomials were trained and tested in the same way as the

other models, and using the same C and gamma values as in the Support Vector Machine in

Table 3. High order polynomial kernels were not used throughout the experiment due to the

lack of sufficient available computing power.

Table 4: Difference in training and test set accuracy when the kernel is changed. Each run is otherwise identical, using the parameters shown in Table 3

Kernel Training

accuracy

Test

accuracy

RBF

Gamma=0.2442

85% 72%

Polynomial

Degree 1

74% 72%

Polynomial

Degree 2

79% 72%

Polynomial

Degree 3

86% 71%

Polynomial

Degree 4

87% 70%

Polynomial

Degree 5

87% 70%

Polynomial

Degree 6

87% 69%

Polynomial

Degree 7

86% 68%

Polynomial

Degree 8

85% 66%

47

Figure 13: Variation of AUC and Accuracy with different hyperparameters for Neural Networks using features listed in Section 9.2

48

The Neural Network results in Figure 13 show that the Adam solver scored best on both

measures. There is small variation in the peak results when the number of layers and nodes per

layer are varied, with a clear decrease in accuracy and AUC when there are less than 3 nodes

per layer. Varying alpha over a large logarithmic scale appears to have a negligible effect on

the model performance.

49

Figure 14: Variation of AUC and Accuracy with different hyperparameters for Random Forests using features listed in Section 9.2

50

The accuracy and AUC of a Random Forest model are shown to be generally higher when

using entropy as the scoring criterion. The best scores do not appear to be attained when large

numbers of estimators are used, with higher scores achieved when fewer than 250 estimators

were used. It is noted, however, that the highest scores do coincide with the most densely tested

area of the parameter search space. A peak in both AUC and accuracy appears to exist when

the maximum number of features to consider is set between 5 and 10. The use of bootstrapping

tends to result in higher scores of AUC and average, but the small number of highest AUC

scores occurred when bootstrapping was not used.

Figure 15: Variation of AUC and Accuracy with different hyperparameters for K-Nearest Neighbours using features listed in Section 9.2

The value of k used in K-Nearest Neighbours (Figure 15) has a clear influence on both AUC

and accuracy, with peaks in the range from 25 to 45. The best scores are achieved with a p

parameter value of 1.

5.3.2.2. Conclusion For the K-Nearest Neighbours model, the two parameters trained both have a clear impact on

model performance. The model works best when it takes the most common class label amongst

the nearest 25-45 points. For values of k below 4, there is a noticeable drop in performance

51

which is much less than if k is 50. This suggests that there is noise in the model, as just taking

the single nearest neighbour is much worse than averaging over a larger number. Increasing

the value of k smooths the decision boundary. The best value of p to use from the results is 1.

The Adam solver proved to be the most effective for the Neural Network models which was

expected due to its widespread popularity. There appears to be little variation in performance

when the number of layers is varied between 2 and 7, and the number of units per layer varied

between 3 and 19. If this behaviour is representative of the reality, a smaller network would be

preferred to reduce processing times and complexity. Alpha does not appear to affect

performance in the range tested.

The Support Vector Machine optimum model has a C value of just greater than 1. This implies

that a balance between fitting the data points accurately, but without overfitting, is required as

the penalty for misclassifying a point is not too great. The kernel that gave the best results was

the radial basis function kernel. This agrees with the expectation that it is a useful default

kernel. It is perhaps more surprising that the polynomial kernel of degree 2 performs better

than the kernel of degree 3. This indicates that the best decision boundary found in these

experiments is not overly complex and is discussed further.

Given the highly interspersed nature of the data (Figure 6), the most complex kernel function

was expected to have the most success. This result, and the accuracy rates of around 70%,

suggests that the best Support Vector Machine investigated does not have a highly complex

boundary (in the original, non-transformed feature space) and simply finds areas of higher

density of each class and separates them. It also suggests that, if high levels of complexity do

not lead to better results, that there is a tendency for the model not to generalise well i.e. to

overfit the training data.

On further investigation, Table 4 shows that increasing model complexity does result in an

increased accuracy score on the training set. This suggests that, to achieve higher training

scores, a more complex model would be required than those that have been used in this

experiment. Higher order polynomials were found to increase the training accuracy slightly up

to a maximum of 87% before diminishing. The table shows, however, that increasing the degree

of the polynomial decreases the test set accuracy, meaning that the model is becoming more

overfitted to the training data. For the same accuracy on the training data, the radial basis

function kernel performs better on the test data.

52

For Random Forests, the more estimators used, the better the model is expected to perform,

with the performance reaching an asymptotic limit. It is therefore required to find the smallest

number which gives an appropriate performance level. Small numbers of estimators are not

shown because they were ruled out in earlier rounds of the hyperparameter search as being

ineffective. The results show that the model performance varies little with the number of

estimators, so using large numbers of estimators appears to be a waste of computing power.

The highest scores, however, are for models with fewer than 250 estimators, which defies

expectations. This can be explained by the random nature of the Random Forest.

The highest scores are located where the parameter has been tested the most. Random Forests

use randomness in two ways to build the trees (Section 2.2.2). Each Random Forest, even if

generated using exactly the same hyperparameters, will therefore likely generate different

results. By repeatedly testing in a confined search space, it is to be expected that a greater range

of outcomes is achieved. As this experiment is concentrating on the highest scoring models,

this distorts the results, as results which are unusually high are reported as the optimal solutions.

If the same density of testing was carried out over the range from 250 to 2000 estimators, it is

expected that results similar to (or slightly better than) those for below 250 estimators would

be attained.

The maximum number of features to consider peaks between 5 and 10, so a value is selected

from that range. Bootstrapping and the decision criterion selected do not appear to make too

much difference to the results.

Overall, there can be reasonable confidence that, for the parameters that were available, the

models are near their optimum. A more thorough search could be completed with more

computing power, which would have three effects. The first is that the parameter values could

be further refined, for small potential gains in performance. This could be achieved by using a

finer and finer resolution grid search, reducing the range of values tested each time. This would

also be helped by completing the full searches as opposed to random subsets.

More computing power could also be used to investigate the random nature of the results more

thoroughly, to characterise the noise and identify which results are due to noise and which are

due to the effect itself. This could most easily be done by simply repeating runs multiple times.

The final way these hyperparameters could be optimised if more computing power were

available would be to investigate a wider range of parameters. Many parameters could not be

investigated as thoroughly as desired because of the computing power required. For example,

53

neither high order polynomial kernels nor high C values for Support Vector Machines could

be used as they took too long to compute. Deeper Neural Networks with more combinations of

hidden layers could also be investigated.

5.4. Experiment 2: Finding the optimum features to input to the model The second aim of the project was to identify which features had the greatest importance in

predicting the future performance of a poorly performing school. For some of the models used,

such as Logistic Regression, the model can easily generate a measure of the importance of each

feature in the model. For other models, such as Support Vector Machines and Neural Networks,

there is no direct method of determining the importance of an individual feature to a model. As

shown in Section 5.3.1, Support Vector Machines and Neural Networks have proven to be

some of the more successful models and, as a result, the selection of input features for these

models is an important consideration.

One way to infer a ranking of feature importance to the models is by using Sequential Forward

or Backward Selection (Section 2.3.2). This can be used on any classification model as it

simply trains and tests the model with different combinations of features and records which

features result in the best scores. This is, however, extremely computationally intensive and so

can only be used sparingly.

5.4.1. Experiment 2.1: Investigating whether Sequential Forward Selection or Sequential Backward Selection give better results

Sequential Selection is a greedy algorithm for selecting the optimum combination of features

for a given model (see Section 2.3.2) and was implemented in SFS.py. As a single measure of

model effectiveness must be used, and input prior to running the algorithm, the accuracy was

selected. On each iteration of the algorithm, the feature that resulted in the highest accuracy

would be added or removed from the selected feature set.

Because the algorithm is greedy and does not necessarily reach an optimal solution, the results

of forward and backward selection can be different.

5.4.1.1. Results Figure 16 show the results of the experiment which was the second round of sequential feature

selection (see Figure 7). In each graph, a single set of model hyperparameters were used. The

model accuracy is plotted for each number of features, for both forward and backward

selection. The forward selection was limited to the first 50 features and both forward and

backward selection had a minimum of three features. A dotted line indicates the highest

accuracy achieved during the process.

54

Figure 16: Sequential Forward/Backward Selection results: Model accuracy for different numbers of features

In each of the plots, the maximum accuracy is achieved with forward selection. The difference

between the maximum accuracy for forward and backward selection ranges from 1% to 5%.

For most sizes of feature set, forward selection selects features which lead to a higher accuracy.

The only exceptions to this are for feature sets in the range 30-50 features for the Neural

Network and Support Vector Machine. In these cases, backward selection gave an accuracy

that was similar to or better than that from forward selection.

K-Nearest Neighbours shows that there is not much change in performance from features sets

of size 15 to 45 features for forward selection. For the other models, a peak in the range of 10

to 20 features is followed by a clear decrease in accuracy when further features are added.

5.4.1.2. Conclusion As shown by the results of this experiment, for the models selected, sequential forward

selection is more effective than sequential backward selection. The maximum accuracy

55

achieved is consistently higher for forward selection and, for a given size of feature set, the

model accuracy is almost always higher when forward selection is used.

For most models, a clear peak exists which indicates a possible feature set to select for use.

With K-Nearest Neighbours, however, the accuracy stays within a range of around 0.5% whilst

a further 30 features are added.

The plots show a difference in performance between forward and backward selection. This is

due to the algorithms being greedy and heading for local maxima. It therefore cannot be

concluded with confidence that the optimum set of features is that which results in the highest

accuracy score in this sequential feature selection process.

5.4.2. Experiment 2.2: Investigating whether Recursive Feature Elimination improves the set of features selected

Recursive Feature Elimination (Section 2.3.3) is another technique for feature selection. A

large quantity of runs were carried out in which RFE was incorporated at different levels. Each

run had a specified number of features to use – 5, 10, 15, 20, 25, All (no RFE). The features

selected by RFE would be used, the others removed from the input data to the model prior to

model fitting. The input feature sets used were those selected as optimum from the second

Sequential Forward Selection run in Section 5.4.1. Where the number of features for selection

in RFE is greater than the number of features in the input feature set, the RFE process has no

effect and the full input feature set is used. The hyperparameters used were the ranges of

hyperparameters tested in the final hyperparameter search.

The aim of this experiment was to find out if the set of features input to the model would prove

to be the best, or whether a subset of the features would result in an improved model.

56

5.4.2.1. Results Figure 17 shows the results of the experiments for Recursive Feature Elimination.

Figure 17: Recursive Feature Elimination - For each of the six different models, for each level of RFE, the highest score of Area Under the ROC curve and Accuracy are plotted

The results show that applying recursive feature elimination to a set of features which has been

selected through sequential feature selection does not result in an increase to the area under the

ROC curve or the model accuracy. The K-Nearest Neighbours plots show that the more features

that are used, the higher the accuracy of the model. The Gaussian Naïve Bayes plot does not

have a value for where the RFE value is 5 because no run met the minimum thresholds set (see

Section 5.1)

57

5.4.2.2. Conclusion The plots show that applying recursive feature elimination to take a subset of the input features,

which were themselves a subset of the initial features, does not improve the performance of the

model. This adds confidence that the features selected using SFS are a good selection. As the

RFE process and SFS process are different, the fact that they do not contradict each other

increases confidence in the results obtained.

5.4.3. Key features selected in most effective model The features selected for each model are shown in the appendices (Section 9.2).

Of the six feature sets selected, the features that are common to at least two sets are shown in

Table 5. As the most frequently occurring features only occur in half of the models, there is

clearly variation in which features work best for different models.

Table 5: Features that appear in more than one of the selected 6 models

Feature name Frequency Energy_2yrDiff 3 Supply.Staff_2yrDiff 3 TotalRevBalance Change 7yr 3 ISSECONDARY 2 Other_2yrDiff 2 Energy_4yrDiff 2 Total revenue balance (1) 2017-18 2 HasBoys 2 Learning.Resources.2018 2 Special 2 HasGirls 2 Back.Office_4yrDiff 2 Energy.2018 2 TotalRevBalance Change 4yr 2 Premises_4yrDiff 2

The importance of each feature can be calculated from a Random Forest as the decrease in node

impurity weighted by the probability of reaching that node (Ronaghan, 2018). This was first

calculated for the features in the Random Forest model used, with the results shown in Table

6.

Table 6: Importance of each feature to the selected Random Forest model

Feature name Feature

importance TotalRevBalance Change 4yr 0.31334

58

TotalRevBalance Change 7yr 0.253867 Catering.2018 0.126941 Teaching.Staff.2018 0.107894 Self.Income_2yrDiff 0.095085 AGEL 0.041989 ISSECONDARY 0.026097 ISPRIMARY 0.016324 GOR_North West 0.00784 GOR_East Midlands 0.005651 Special 0.003589 HasGirls 0.001383

The selected Random Forest model was then trained and tested using all of the original input

features. The feature importance was then calculated and shown in Table 7.

Table 7: Importance of the top 30 features when the parameters selected of the Random Forest model are applied to all features

Feature name Feature

importance Total revenue balance (1) 2017-18 0.126383 TotalRevBalance Change 7yr 0.118881 TotalRevBalance Change 4yr 0.099824 TotalRevBalance Change 2yr 0.078365 Total.Spend.pp_4yrDiff 0.023139 Supply.Staff.2018 0.01918 Supply.Staff_2yrDiff 0.019165 Total revenue balance (1) as a % of total revenue income (6) 2017-18 0.018994 PERCTOT 0.016633 Supply.Staff_4yrDiff 0.0146 PerformancePctRank 0.014318 Total.Income.pp_4yrDiff 0.011931 AcademyNew 0.011606 Energy.2018 0.011453 Catering.2018 0.011407 Catering_4yrDiff 0.011356 Learning.Resources_2yrDiff 0.011065 Back.Office_2yrDiff 0.011061 Teaching.Staff_2yrDiff 0.010531 Premises_4yrDiff 0.010522 ICT.2018 0.010342 Back.Office.2018 0.010228 Consultancy.2018 0.009657 PTFSM6CLA1A__18 0.009642 Mean Gross FTE Salary of All Teachers (£s) 0.009461

59

Total.Income.pp.2018 0.009431 ICT_2yrDiff 0.009284 Energy_2yrDiff 0.009065 AGEH 0.008857 Consultancy_2yrDiff 0.008537

Table 7 shows that the Random Forest classifier places a high value on financial data, with the

top 8 features being of this type. The level of pupil absence is also an important feature, as are

school examination results. The spend on supply teachers also appears to be an important

factor.

5.5. Experiment 3: Finding the optimum operations on the input data The performance of any model is reliant on the data that is input. A series of experiments were

set up to determine which operations on the input data would result in the most effective model.

5.5.1. Experiment 3.1: Determining whether oversampling improves model performance Each run in the parameter search was run both with and without oversampling and the results

recorded. This allows direct comparison between the sets of runs as each group (oversampled

or not oversampled) contains runs with identical parameters aside from whether oversampling

is used. The SMOTE oversampling algorithm was implemented as described in Section 2.3.4.

This experiment tests whether the use of this oversampling technique caused an improvement

in the model results.

5.5.1.1. Results The results of the experiment are shown in Figure 18.

60

Figure 18: Oversampling - For each model type, the run with the highest accuracy and area under the ROC curve are shown for the group where oversampling was used on the training data and the group which did not use oversampling.

The plots show that the best runs with and without oversampling have very similar scores for

both accuracy and area under the ROC curve. Logistic Regression shows a larger best accuracy

when oversampling is used.

5.5.1.2. Conclusion The results show that applying oversampling makes little difference to the best model found

for each model type. In the case of Logistic Regression, an increase in accuracy is observed

when oversampling is applied. The minimal effect of oversampling on this dataset can largely

be explained by the fact that the dataset has a relatively even split between the two classes

61

(stuck and escaped stuck). As a result, there are few oversampled data points required to make

the split exactly 50:50.

This technique is expected to be far more effective when there is a more uneven split between

the classes, such as when using data with the initial definition of stuck school (see Section 4.5).

5.6. Attributes of Most Effective Model The most effective model overall is the K-Nearest Neighbours model with k=41 and p=1. The

confusion matrix is shown in Table 8, its scores across different measures are shown in Table

9 and its parameters were specified in Table 3.

Table 8: Confusion matrix for best model

True

1 0

Predicted 1 474 119

0 331 789

Table 9: Properties of best model

Accuracy 0.737

AUC 0.759

Precision for class 0

0.694

Precision for class 1

0.746

Recall for class 0

0.889

Recall for class 1

0.514

62

6. Discussion The results show that the best models found have AUC and accuracy scores of between 70%

and 75%. The results are relatively self-consistent in that the results appear to be heading

towards a limit of around 75%.

The majority of the work has been carried out using the newer definition of stuck schools,

which has led to a more challenging classification task. As is seen in Figure 6, the two classes

are strongly overlapped and have proved challenging to separate. Judging by the consistency

of the results achieved in this project, it appears that increasing the accuracy and AUC scores

significantly would require a major change in the techniques used, since those used in this work

have been used thoroughly. There are, however, many areas in which the existing techniques

used could be optimised further.

School examination results were suspected to be an indicator of likely school inspection

success, yet they only feature in one of the models. One reason why they do not feature could

be the fact that primary and secondary schools have completely different scoring systems which

were resolved in Section 4.1.3. This leads to a more general issue with the data: that it is

measured and recorded in different ways over the years. A large amount of the data used is

solely for the most recent academic year, so is relatively self-consistent. Many other data

points, though, are time based, showing a change over the years.

Time is an important aspect of this analysis which is difficult to bring in. Firstly, school

inspections take place at irregular intervals (Section 1.1). When labelling schools based on their

histories, the two simple ways of doing this are to arrange them in order, most recent first, or

to arrange them by the date that they took place. Each has its advantages, but also causes a loss

of information. Given that inspection order was selected, a possible downside is that the data

used takes into account some differences from 7 years previously, whilst all of the relevant

inspections could have taken place in the last 3 years.

The second issue with time is the use of current data to predict the future performance of

schools. The data used is based on the current year, or a change from a number of years

previous. The class labels, as noted, are not being considered against time.

A possible area of weakness in the modelling is that schools are linked to their predecessors.

Generally, it is assumed that this is a sensible decision as schools do not change completely if

they close and reopen – it is usually still the same building with the same teachers. Sometimes,

though, schools have a list of many predecessors, where many smaller schools have been

63

merged into a larger school. All of the predecessor schools’ inspection results count towards

the new school’s results, so if four small schools with a recent poor inspection each merge into

a large new school, that school is immediately labelled as stuck.

The way that the data has been merged also means that data for the current school is used, but

for predecessors it is not. Therefore, if the school opened in 2016 then it will not have any data

in the combined dataset for dates before 2016. This is likely a source of significant missing

data and could be remedied by writing a program to look up predecessor school data if there is

no data available for the currently open school. This would need to be done carefully, however,

given that the predecessor is not necessarily a direct representation of the current school.

Missing data is an important area of uncertainty. Given that none of the features were complete

and the models required fully complete data, there was little option but to impute the missing

data (Section 3.3.4). The selected imputer uses other values in the row and column of the

missing data point to infer what value to insert. As a result, the imputed value depends on the

other features in the dataset. If the missing data in a new column is imputed as soon as the

column is added to the dataset, the missing values will be imputed with different values to those

if all of the columns were imputed at the same time, once they had all been added. Whichever

imputation technique is selected, it will add error to the model which has not been

characterised.

The data also showed that there is great variation throughout it, with a clear difference in

performance between different sections of the data. If the data is sorted by URN (i.e. in

approximate order of when the school was opened) and the random selection option of the cross

validation is disabled then the plot in Figure 19 can be generated. Moving from Fold 1 through

to Fold 5 is then moving approximately from the oldest 20% of schools to the newest 20%. The

performance of the model appears to improve drastically from one fold to the next. For

example, the area under the curve in some of the models varies from 55% in the first fold to

over 85% in the fourth fold. These results appear counterintuitive because, as explained above,

the most complete data is expected to be for the oldest schools because their data would not be

lost due to being assigned to a predecessor. Given that the effect is so pronounced, there appears

to be good reason to investigate this further, to see if a significant improvement could be made

to the model.

64

Figure 19: ROC curves for the selected models, with randomness disabled in the cross validation process. Fold 1 is therefore the first 20% of the data points, Fold 2 is 20-40% etc.

65

Cross validation was used throughout this work, with all results reported being averages over 5 fold cross validation. Given the clear variation in the data, it would likely make sense to increase to 10 folds. This would minimise the distortion of the results from the unevenness of the data. Re-running the same experiment generates different results due to the differing cross validation splits, introducing noise into the results.

Using accuracy and AUC as the measures for determining the performance of the model assumes that false positives and false negatives have equal cost (Adams and Hand, 2000). In this work, these measures have been used heavily as the costs are assumed equal. Further work and discussions with the relevant stakeholders may lead to adjusted measures being used.

The main focus of this work has been on the updated definition of stuck schools, and modelling them against schools that have ‘escaped’ being stuck (Section 4.3). Using the original definition of stuck schools, and modelling against all of the rest of the dataset, the results are much improved. One reason for this is that there are more data points. By modelling just the two subsets of the data, 90% of the data is removed and not used. On top of this, a school in the original stuck category is likely very different from a randomly selected other school. Using the new definition and subsets, however, the two classes that are to be distinguished are likely to be very similar. They have both had a run of three consecutive poor inspections, with potentially a single inspection different between them. There are approximately 430 schools in class 1 under the original stuck definition, with approximately 800 schools in class 1 under the new definition. The old class 1 is a subset of the new class 1.

The technique used for selecting the hyperparameters and the features were both computationally intensive, requiring the training and testing of large quantities of models. The nature of the searches meant that sub-optimal combinations would be tried, taking far longer than other combinations of hyperparameters and features. Throughout this work, computing power was at a premium. The code was set up to work best in these conditions, for example routinely dumping results to .csv files and python ‘pickles’ to free up memory, limiting hyperparameter ranges to those that processed faster and designing the code to run unattended for hours/days. Further information on this topic is provided in Section 5.2. This project was also undertaken on four different laptops so file locations, version control and module versions were watched closely.

The use of sequential feature selection is a useful technique because it allows feature selection for models which do not generate a feature importance value. It is a greedy algorithm which will not necessarily find the optimal combination of parameters. It does, however, provide confidence that adding or removing a single feature from the feature set would not improve performance because if it did, the feature would have been selected by the algorithm. The only manual input into the feature selection was done at the start, in selecting and preparing the features to add to the dataset. Once they had been added to the dataset, the features were selected by the algorithms in the process.

Some of the more complex models, such as the Neural Network, are unlikely to be optimal. There are many parameters and architectures available which can be altered to potentially improve the model, which would take a large quantity of time and computing power. For example, the number of iterations and using layers with varying numbers of units have not been investigated. Neural Networks were not originally considered for this work due to the small quantity of training data available, so shallow networks are expected to work best. The models could also be optimised by making a more thorough investigation into performance results when the model is applied to the

66

training data. This would give more insight into the overfitting/underfitting nature of the model and how it could be improved.

67

7. Conclusions The primary aim of this project was to create a dataset and a resulting model that could

accurately predict the future performance of a school in terms of inspection results. These aims

have been achieved to a good level, given the time and resources available, and the complexity

of the problem.

Firstly, a literature review was carried out in Section 2 to identify and assess previous work in

the field and to investigate techniques to be used in this work. It does not appear that work to

use machine learning to predict school inspection results has been published in the literature,

although a number of studies have been done to predict examination results. Although related

to inspection results, the two are clearly distinct. The literature provided information on six

machine learning models to implement in this project and how they may be optimised.

The dataset creation has been achieved by generating a dataset of every currently open state-

funded school in England, and is explained in Sections 3 and 4. This dataset has over 80

variables which have come from a range of input sources that are of multiple formats. Some of

the variables have been generated through feature engineering. The variables selected have

been narrowed down from over 1000 in the input data. The data have been cleaned, normalised

and the missing values imputed. The data have also been assigned labels using an updated

definition of the binary classes, and data not relevant have been removed from the dataset.

Many models have been trained and tested (Section 5), to find the best performing model for

this task. For each of the six model types identified in the literature survey, the optimum

combination of hyperparameters and features have been found. This was achieved iteratively

using random hyperparameter grid searches and recursive feature selection. The feature sets

found are shown to differ for each model type, with the feature importance shown for Random

Forest models.

The classification accuracy and the area under the ROC curve of the best performing models

overall are around 75%, which is considered a good result when the complexity of the task, the

time available and the quality of the input data are taken into account. The two classes to be

distinguished are very similar, and the available training set is only a small subset of the

complete input data. If a higher confidence in the prediction of which schools will become

stuck is required, an 88% precision has been achieved for a Support Vector Machine with 40%

recall.

68

This work has been presented to both the Department for Education and Ofsted to explain what

has been done and the results achieved. It has been agreed that it is unlikely that further work

to model this data will provide a significant gain in predictive power beyond that of the

classifiers described in this report. At the time of writing, it is not yet known if this work will

be continued within the ONS beyond the project period.

An initial step in terms of further work would be to determine whether an increase in

classification accuracy was possible and, if so, by how much. If there is room for improvement,

an investigation would be required to find out why the models are not producing optimum

results. This would start with an analysis of training and test data to see the overfitting

tendencies of the models.

Should an improvement in classifier performance be sought, there are many possibilities for

work that would provide this. Firstly, there are likely to be techniques to extract high quality

classifications from the classifiers presented. For example, a more thorough investigation of

varying the thresholds could provide a classifier with an excellent precision, even if the recall

is low. The classifiers may also be effective for subsets of the data. For example, the classifier

has not been run on just primary schools. There is known to be a difference in the data between

primary and secondary schools, and this work has considered the two together.

The techniques described in this report could simply be extended with more computing power

available. A more exhaustive hyperparameter search, investigating more hyperparameters over

a larger range would be trivial to implement using the code written for this project. More

sequential feature selection runs, with more different models could also be carried out, with

randomly generated starting feature sets to try to find better results than those generated by the

greedy algorithm.

The input data is clearly an area that can be improved, that could lead to improvements in

model performance. Adding in more features could be done with more time available, but the

key thing would be to reduce the missing data. One part of this would be to incorporate data

for predecessor schools. Another would be to investigate whether more complete data sources

are available. The technique for imputation used, whilst likely to be the best available, also

could be investigated as it has a large bearing on the results and imputes different values

depending on what other features are in the dataset.

69

If a classifier of high quality is not attained, there is likely a large quantity of information to

gain from carrying out a more traditional statistical analysis. This would also tie in with the

more qualitative work that has been completed by Ofsted and the Department for Education.

Working on this project has been an excellent opportunity to carry out a machine learning

project in the ‘real world’, testing out a large variety of the skills acquired this year and in

previous years. It has been interesting to try out many different machine learning techniques

and find out what works best for myself. Working with the ONS, Ofsted and the Department

for Education has also given me some experience of how carrying out a machine learning

project on data can lead to insight and the change of government policy.

Given that there have been three months from first arriving at the ONS to completing this

report, there is only a finite quantity of work that can be achieved. Many steps of the project

could easily have been allocated more than three months themselves so, as a result, there are

areas where more thorough work could have been done. This would, however, have directly

resulted in not completing some of the later work.

The aim of the project was to carry out a full, from start to finish, data science project, with a

real risk that the end was not reached in time. For example, creating the dataset proved to be a

real challenge that put the project behind schedule. The techniques used had not been attempted

before by the team, who were interested to know if they would be of any use. There was

therefore uncertainty in how long the models would take to implement and whether they would

generate anything useful. Given that meaningful results have been generated and the results

have been communicated to the stakeholders within the allotted time, the goals of this project

are considered to have been achieved.

70

8. References Adams, N. M. and Hand, D. J. (2000) ‘Improving the practice of classifier performance

assessment’, Neural Computation, 12(2), pp. 305–311. doi: 10.1162/089976600300015808.

Canagareddy, D., Subarayadu, K. and Hurbungs, V. (2019) ‘A Machine Learning Model to

Predict the Performance of University Students BT - Smart and Sustainable Engineering for

Next Generation Applications’, in Fleming, P. et al. (eds). Cham: Springer International

Publishing, pp. 313–322.

Chawla, N. et al. (2002) ‘SMOTE: Synthetic Minority Over-sampling Technique’, J. Artif.

Intell. Res. (JAIR), 16, pp. 321–357. doi: 10.1613/jair.953.

Department for Education (2019) New drive to continue boosting standards in schools -

GOV.UK. Available at: https://www.gov.uk/government/news/new-drive-to-continue-

boosting-standards-in-schools (Accessed: 19 September 2019).

Department for Education (DfE) (2016) ‘Progress 8: How Progress 8 and Attainment 8

measures are calculated’, pp. 1–5. Available at:

https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data

/file/561021/Progress_8_and_Attainment_8_how_measures_are_calculated.pdf.

Duda, R. O., Hart, P. E. and Stork, D. G. (2001) Pattern classification. 2nd ed. New York ;

Chichester: Wiley (A Wiley-Interscience publication).

Fawcett, T. (2006) ‘An introduction to ROC analysis’, Pattern Recognition Letters, 27(8), pp.

861–874. doi: 10.1016/j.patrec.2005.10.010.

Fowler, J. (2012) ‘Ofsted Inspection of Outstanding schools - The Education ( Exemption

from School Inspection ) ( England ) Regulations 2012’, (January), pp. 0–2. Available at:

https://www.lgiu.org.uk/wp-content/uploads/2012/05/Ofsted-Inspection-of-Outstanding-

schools-The-Education-Exemption-from-School-Inspection-England-Regulations-2012.pdf.

Hsu, C.-W., Chang, C.-C. and Lin, C.-J. (2008) ‘A Practical Guide to Support Vector

Classification’, BJU international, 101(1), pp. 1396–400. Available at:

http://www.csie.ntu.edu.tw/~cjlin%0Ahttp://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.p

df.

Hunter, J. D. (2007) ‘Matplotlib: A 2D Graphics Environment’, Computing in Science &

Engineering, 9(3), pp. 90–95. doi: 10.1109/MCSE.2007.55.

71

Lemaitre, G., Nogueira, F. and Aridas, C. K. (2017) ‘Imbalanced-learn: A Python Toolbox to

Tackle the Curse of Imbalanced Datasets in Machine Learning’, Journal of Machine

Learning Research2, 18(17), pp. 1–5. Available at: http://jmlr.org/papers/v18/16-365.html.

Lin, C., Yu, H. and Huang, F. (2011) ‘Dual Coordinate Descent Methods for Logistic

Regression and Maximum Entropy Models’, Machine Learning, 2(85), pp. 41–75.

Majnik, M. and Bosnic, Z. (2011) ROC Analysis of Classifiers in Machine Learning : A

Survey Technical report MM-1 / 2011.

Masci, C., Johnes, G. and Agasisti, T. (2018) ‘Student and school performance across

countries: A machine learning approach’, European Journal of Operational Research.

Elsevier B.V., 269(3), pp. 1072–1085.

McKinney, W. (2010) ‘Data Structures for Statistical Computing in Python’, in van der Walt,

S. and Millman, J. (eds) Proceedings of the 9th Python in Science Conference, pp. 51–56.

Mitchell, T. M. (Tom M. (1997) Machine learning. New York: McGraw-Hill.

mlxtend (no date) Sequential Feature Selector - mlxtend. Available at:

http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/

(Accessed: 12 September 2019).

Office for Standards in Education (2018) ‘School inspection handbook’, Ofsted School

Inspection Handbook, (September). doi: 10.4324/9780203416242_chapter_2.

Ofsted (2019) ‘Education inspection framework for September 2019’, (May), pp. 1–14.

Available at: www.legislation.gov.uk/uksi/2014/3283/contents/made;

Pedregosa FABIANPEDREGOSA, F. et al. (2011) ‘Scikit-learn: Machine Learning in

Python’, Journal of Machine Learning Research, 12, pp. 2825–2830. Available at:

http://Scikit-learn.sourceforge.net.

Press, W. H. et al. (2007) Numerical Recipes : The Art of Scientific Computing. 3rd ed.

Cambridge ; New York: Cambridge University Press.

Raschka, S. (2018) ‘MLxtend: Providing machine learning and data science utilities and

extensions to Python’s scientific computing stack’, Journal of Open Source Software, 3(24),

p. 638. doi: 10.21105/joss.00638.

Rebai, S., Ben Yahia, F. and Essid, H. (2019) ‘A graphically based machine learning

72

approach to predict secondary schools performance in Tunisia’, Socio-Economic Planning

Sciences. Elsevier, (June), p. 100724. doi: 10.1016/j.seps.2019.06.009.

Rees, G. (2017) List all possible permutations from a python dictionary of lists - Code

Review Stack Exchange. Available at:

https://codereview.stackexchange.com/questions/171173/list-all-possible-permutations-from-

a-python-dictionary-of-lists (Accessed: 19 September 2019).

Ronaghan, S. (2018) The Mathematics of Decision Trees, Random Forest and Feature

Importance in Scikit-learn and Spark. Available at: https://medium.com/@srnghn/the-

mathematics-of-decision-trees-random-forest-and-feature-importance-in-Scikit-learn-and-

spark-f2861df67e3 (Accessed: 15 September 2019).

Scikit-learn (no date a) 1.1. Generalized Linear Models — Scikit-learn 0.21.3 documentation.

Available at: https://Scikit-learn.org/stable/modules/linear_model.html#linear-model

(Accessed: 11 September 2019).

Scikit-learn (no date b) Neural Network models (supervised) - 1.17.1 Multi-layer Perceptron.

Available at: https://Scikit-learn.org/stable/modules/neural_networks_supervised.html

(Accessed: 11 September 2019).

Shalev-Shwartz, S. and Ben-David, S. (2014) Understanding Machine Learning - From

Theory to Algorithms. 1st Ed. Cambridge University Press.

Spielman, A. (2018) The Annual Report of Her Majesty’s Chief Inspector of Education,

Children’s Services and Skills 2017/18 - GOV.UK. Available at:

https://www.gov.uk/government/publications/ofsted-annual-report-201718-education-

childrens-services-and-skills/the-annual-report-of-her-majestys-chief-inspector-of-education-

childrens-services-and-skills-201718 (Accessed: 12 September 2019).

Tanuar, E. et al. (2018) ‘Using Machine Learning Techniques to Earlier Predict Student’s

Performance’, in 2018 Indonesian Association for Pattern Recognition International

Conference (INAPR), pp. 85–89. doi: 10.1109/INAPR.2018.8626856.

Thomson, D. (2019) A look at Ofsted’s ‘stuck’ schools - FFT Education Datalab. Available

at: https://ffteducationdatalab.org.uk/2019/01/a-look-at-ofsteds-stuck-schools/ (Accessed: 18

September 2019).

Walt, S. van der, Colbert, S. C. and Varoquaux, G. (2011) ‘The NumPy Array: A Structure

73

for Efficient Numerical Computation’, Computing in Science & Engineering, 13(2), pp. 22–

30. doi: 10.1109/MCSE.2011.37.

Zhang, H. (2004) ‘The Optimality of Naive Bayes Naive Bayes and Augmented Naive

Bayes’, Aa, 1(2), p. 3.

74

9. Appendices 9.1. Appendix 1: Input Variables Used The variables used in this project are shown below, along with a brief description of their

meaning. Variable names with a year suffix mean that the value was recorded in that year. For

example, VariableName_16 would correspond to the value VariableName had in the academic

year 2015-16.

9.1.1. School Financial Balance Total revenue balance (1) 2017-18 – Float

Total revenue balance (1) as a % of total revenue income (6) 2017-18 – Float

TotalRevBalance Change 7yr – Float (calculated) – Difference between Total revenue

balance in 2010-11 and 2017-18.

TotalRevBalance Change 4yr – Float (calculated) – Difference between Total revenue

balance in 2013-14 and 2017-18.

TotalRevBalance Change 2yr – Float (calculated) – Difference between Total revenue

balance in 2015-16 and 2017-18.

9.1.2. School performance data TOTPUPS__18 – Integer – Total number of pupils in the school.

PTKS1GROUP_L__18 – Float – Values taken from columns with the following names:

“Percentage of pupils at the end of key stage 4 with low prior attainment at the end of key stage

2”, “Percentage of pupils in cohort with low KS1 attainment”, “% pupils in cohort with low

KS1 attainment”

PTKS1GROUP_M__18 – Float – Values taken from columns with the following names:

“Percentage of pupils at the end of key stage 4 with medium prior attainment at the end of key

stage 2”, “Percentage of pupils in cohort with medium KS1 attainment”, “% pupils in cohort

with medium KS1 attainment”.

PTKS1GROUP_H__18 – Float – Values taken from columns with the following names:

“Percentage of pupils at the end of key stage 4 with high prior attainment at the end of key

stage 2”, “Percentage of pupils in cohort with high KS1 attainment”, “% pupils in cohort with

high KS1 attainment”.

75

PTFSM6CLA1A__18 – Float – Percentage of pupils in school eligible for free school meals.

Values taken from columns with the following name: “Percentage of pupils at the end of key

stage 4 who are disadvantaged”, “Percentage of pupils who are disadvantaged”.

PTMOBN__18 – Float – Values taken from columns with the following names: “Percentage

of pupils at the end of Key Stage 4 who are non-mobile”, “Percentage of eligible pupils

classified as non-mobile”.

PSENELSE__18 – Float – Percentage of pupils with Special Educational Needs. Values taken

from columns with the following names: “Percentage of eligible pupils with SEN with

Statement or EHC plan”, “Percentage of Pupils with statements or supported at school action

plus”, “Percentage of pupils at the end of key stage 4 with special educational needs (SEN)

with a statement or Education, health and care (EHC) plan”, “Percentage of key stage 4 pupils

with statements of SEN (Special Educational Need) or on School Action Plus”.

PerformancePctRank – Float (calculated) – Percentage ranking of schools based on exam

performance. A full explanation is provided in 4.1.3. Values taken from columns with the

following names: “Average Attainment 8 score per pupil”, “Percentage of pupils reaching the

expected standard in reading, writing and maths”.

9.1.3. Pupil population and absence data PNUMEAL – Float – Percentage of pupils with English not as first language.

PNUMFSM – Float – Percentage of pupils eligible for free school meals.

PERCTOT – Float – Percentage of overall absence (authorised and unauthorised) for the full

2017/18 academic year.

9.1.4. Spine ISPRIMARY – Binary – Whether the school is primary.

ISSECONDARY – Binary – Whether the school is secondary.

ISPOST16 – Binary – Whether the school is for pupils aged over 16.

AGEL – Integer – The lowest age of pupils.

AGEH – Integer – The highest age of pupils.

9.1.5. Workforce data Pupil : Teacher Ratio – Float.

76

Mean Gross FTE Salary of All Teachers (£s) – Integer – Mean Full Time Equivalent salary

of teachers in school.

9.1.6. School finances data These variables are an annual total amount of money. Each of the titles in this section represents

three variables:

- The variable value in 2018

- The difference between the variable values in 2016 and 2018

- The difference between the variable values in 2014 and 2018.

Self.Income - Integer – Annual self generated income

Total.Income.pp – Integer – Annual total income per pupil

Teaching.Staff – Integer – Annual spend on teaching staff

Supply.Staff – Integer – Annual spend on supply staff

Ed.Support.Staff – Integer – Annual spend on educational support staff

Premises – Integer – Annual spend on premises

Back.Office – Integer – Annual spend on back office staff

Catering – Integer – Annual spend on catering

Other.Staff – Integer – Annual spend on other staff

Energy – Integer – Annual spend on energy

Learning.Resources – Integer – Annual spend on learning resources

ICT – Integer – Annual spend on IT equipment

Consultancy – Integer – Annual spend on external consultants

Other – Integer – Other annual spend

Total.Spend.pp – Integer – Total spend per pupil

77

9.1.7. Generated variables Boarding – Binary (converted from text) - Whether the school has pupils who stay at the school

overnight. Positive cases: “Boarding school”, “Children’s home (Boarding school)”, “College

/ FE residential accommodation”. All other cases negative.

SixthForm – Binary (converted from text) – Whether the school has a sixth form.

HasBoys – Binary (converted from text) – School has male students. Positive cases: “Mixed”,

“Boys”.

HasGirls – Binary (converted from text) – School has female students. Positive cases:

“Mixed”, “Girls”.

Maintained – Binary – School is a local authority maintained school

Academy – Binary – School is an academy

Special – Binary – School is a special school

GOR_East Midlands / GOR_East of England / GOR_London / GOR_North East /

GOR_North West / GOR_South East / GOR_South West / GOR_West Midlands /

GOR_Yorkshire and the Humber – Binary variables – Whether school is in the Government

Office Region.

9.2. Appendix 2: Variables selected for models Each optimised model used a different subset of the available variables. The variables used for

each model are shown in this appendix.

9.2.1. Neural Network 'Total revenue balance (1) 2017-18',

'Other.Staff_2yrDiff',

'Premises_4yrDiff',

'Mean Gross FTE Salary of All Teachers (£s)',

'Supply.Staff_2yrDiff',

'BoardingNew',

'TotalRevBalance Change 7yr',

78

'TotalRevBalance Change 2yr',

'Supply.Staff_4yrDiff',

'PNUMEAL',

'GOR_West Midlands',

'MaintainedNew',

'GOR_North East',

'ISPOST16',

'Supply.Staff.2018',

'Other.2018',

'Learning.Resources_2yrDiff',

'TotalRevBalance Change 4yr',

'ICT.2018',

9.2.2. Support Vector Machine 'TotalRevBalance Change 7yr',

'Total revenue balance (1) as a % of total revenue income (6) 2017-18',

'Supply.Staff_2yrDiff',

'Mean Gross FTE Salary of All Teachers (£s)',

'Consultancy_4yrDiff',

'Ed.Support.Staff_4yrDiff',

'Consultancy.2018',

9.2.3. Random Forest 'TotalRevBalance Change 4yr',

'PerformancePctRank',

79

'Supply.Staff_4yrDiff',

'ISSECONDARY',

'TotalRevBalance Change 7yr',

'PTKS1GROUP_H__18',

'Total revenue balance (1) as a % of total revenue income (6) 2017-18',

'AGEL',

9.2.4. Gaussian Naïve Bayes 'TotalRevBalance Change 4yr',

'Total.Income.pp.2018',

'Supply.Staff.2018',

'Total revenue balance (1) 2017-18',

'Self.Income.2018',

'Mean Gross FTE Salary of All Teachers (£s)',

'PSENELSE__18',

'TotalRevBalance Change 7yr',

'Ed.Support.Staff.2018',

'Consultancy_2yrDiff'

9.2.5. Logistic Regression 'Total revenue balance (1) 2017-18',

'Supply.Staff_2yrDiff',

'TotalRevBalance Change 7yr',

'Other_4yrDiff',

'Teaching.Staff_4yrDiff',

80

'Other.Staff_4yrDiff',

'Other.Staff_2yrDiff',

'Supply.Staff.2018',

'Ed.Support.Staff.2018'

9.2.6. K-Nearest Neighbours 'Total revenue balance (1) 2017-18',

'TotalRevBalance Change 4yr',

'HasGirlsNew',

'HasBoysNew',

'ISPRIMARY',

'Catering.2018',

'GOR_North East',

'Ed.Support.Staff_4yrDiff',

'BoardingNew',

'Back.Office_2yrDiff',

'Consultancy.2018',

'Consultancy_2yrDiff',

'SpecialNew',

'Ed.Support.Staff_2yrDiff',

'PERCTOT',

'Other.Staff_4yrDiff',

'Learning.Resources.2018',

'Other.2018',

'Other_4yrDiff',

81

'Energy_4yrDiff',

'TotalRevBalance Change 7yr',

'Teaching.Staff.2018',

'Total revenue balance (1) as a % of total revenue income (6) 2017-18',

'Learning.Resources_4yrDiff',

'ISSECONDARY',

'SixthFormNew',

'ISPOST16',

'Catering_2yrDiff',

Tata Steel Optimisation Dissertation(1).pdf