{"id":692,"date":"2017-02-15T09:25:55","date_gmt":"2017-02-15T09:25:55","guid":{"rendered":"http:\/\/anna.ps\/blog\/?p=692"},"modified":"2017-02-22T17:55:58","modified_gmt":"2017-02-22T17:55:58","slug":"my-year-and-a-bit-working-on-tech-for-good-projects","status":"publish","type":"post","link":"https:\/\/anna.ps\/blog\/my-year-and-a-bit-working-on-tech-for-good-projects","title":{"rendered":"My year-and-a-bit working on tech-for-good projects"},"content":{"rendered":"

In the past year or so I did a lot of work on public-interest tech and data projects. I was so busy writing code, designing systems and hiring people that I failed to write anything at all about why these projects were worthwhile, and the sort of design and engineering challenges I had to overcome.<\/p>\n

If you\u2019re even slightly into projects that use data and coding for public good, I hope you\u2019ll find this write-up at least mildly interesting!<\/p>\n

Work for Private Eye<\/h4>\n

Once in a while, a dream project comes along. This was the case when a Private Eye journalist called Christian Eriksson<\/a> wrote to say that he\u2019d obtained details of all the UK properties owned by overseas companies via FOI, and wanted help with\u00a0the data. This is how I came to build the Overseas Property map<\/a> for Private Eye, which lets you see which of your neighbours own their property through an overseas company. I\u2019ll write more the tech side of this separately at some point, but essentially the map shows 70,000 properties, two-thirds of which are owned in tax havens.<\/p>\n

\"Detail<\/a>
A detail from the map showing streets in Mayfair – whole blocks are owned by overseas companies.<\/figcaption><\/figure>\n

Christian and fellow Eye hack Richard Brooks wrote more than 30 stories about the arms dealers, money launderers and tax avoiders hiding property via these companies – the stories eventually became a Private Eye Special Report<\/a>. The map was discussed<\/a> in Parliament<\/a>, written up in the FT<\/a>, and the government eventually released the same data publicly<\/a>.<\/p>\n

This December, the investigation and map were nominated for the British Journalism Awards, in the ‘digital innovation’ and ‘investigation of the year’ categories, so I got to go to a fancy awards party. (Not too<\/i> fancy – the goodie\u00a0bag consisted of some nuts and a bottle of Heineken.) We were highly commended in the ‘digital innovation’ category, which was nice.<\/p>\n

I also worked on another project for Private Eye<\/a>. Freelance journalist Dale Hinton spotted that some local councillors (amazingly!) choose not to pay their council tax, and dug out the numbers across the country. Then the Eye\u2019s Rotten Boroughs editor, Tim Minogue, suggested mapping the data. The resulting map<\/a> just shows the number of rogues in each council. There were some creative excuses from the rogues, but my favourite was the councillor who admitted simply: \u201cI ballsed up\u201d.<\/p>\n

Tech lead at Evidence-Based Medicine DataLab<\/h4>\n

My day job for most of 2016 was as tech lead at the Evidence-Based Medicine DataLab<\/a> at the University of Oxford. This is a new institution set up by the brilliant Dr Ben Goldacre<\/a> (of Bad Science<\/a> fame). Evidence-based medicine uses evidence to inform medical practice, and the Lab aims to extend that by helping doctors use data better. I was the first hire.<\/p>\n

As you might expect, this was a fascinating and rewarding job. I led on all the technology projects, collaborated on research, and helped build the team from 2 to 9 full-time staff, so a big chunk of my year was spent recruiting. In many ways 2016 was the year when I stopped being \u2018just a coder\u2019, and started to learn what it means to be a CTO. Here are some of the projects I worked on.<\/p>\n

OpenPrescribing<\/h4>\n

I got the job at EBM DataLab on the strength of having been the sole developer on OpenPrescribing<\/a>, collaborating with Ben and funded by Dr Peter Brindle at West of England Academic Health Sciences Network<\/a>. This site provides a rapid search interface, dashboards and API to all the prescribing decisions made by GPs in England & Wales since 2010. Basically, it makes it easier to see which medicines were prescribed where.<\/p>\n

The big challenge on this project was design and UX. I interviewed doctors, prescribing managers and researchers, and we ended up with dashboards to show each organisation where it\u2019s an outlier on various measures – so each GP or group of GPs can quickly see where it could save money or improve patient care.<\/p>\n

The charts use percentiles to allow users to compare themselves with similar organisations, e.g. here\u2019s how the group of GPs in central Birmingham<\/a> used to prescribe many more expensive branded contraceptive pills than similar groups elsewhere, but improved things recently:<\/p>\n

\"Cerazette<\/a>
If this group of GPs prescribed branded contraceptives in the same proportion as the median (blue dashed line), they would have spent about \u00a330,000 less in the past six months alone. This is the exact same drug\u00a0– the only difference is the brand name.<\/figcaption><\/figure>\n

There\u2019s also a fast search form<\/a> for users who know what they\u2019re looking for, and an API<\/a> that lets researchers query for raw data files. Technically, it\u2019s a Postgres\/Django\/DRF back-end, and JavaScript front-end with Highcharts to render the graphs (code here<\/a>).<\/p>\n

The raw data files<\/a> are so unwieldy that previously (we were told) they were only really used by pharma companies, to check where their drugs were being under-prescribed and target marketing accordingly. In fact, we heard that lobbying from pharma was what got the NHS to release the open data in the first place!<\/p>\n

OpenPrescribing was also an interesting technical challenge, because the dataset was reasonably large (80GB, 500 million rows), and users need to run fast queries across all of it. Since I didn\u2019t have millions to give to Oracle, which is what the NHS does internally, I used Postgres for our version. With a bit of love and optimisation, it was all performant and scaled well.<\/p>\n

Writing papers with BigQuery<\/h4>\n

As well as building services, EBM DataLab writes original research. Over the year I co-authored three papers, and wrote numerous analyses now in the paper pipeline. I can\u2019t go into detail since these are all pre-publication, but they\u2019re mostly based on the prescribing dataset, about how the NHS manages (or doesn\u2019t) its \u00a310 billion annual prescribing budget.<\/p>\n

Probably the most enjoyable technical aspect of last year was setting up the data analysis tools for this – well, I\u2019m not going to call it ‘big data’ because it\u2019s not terabytes, but let\u2019s say it\u2019s reasonably sized data. I set up a BigQuery<\/a> dataset, which makes querying this huge dataset fast, and as simple as writing SQL. Then I connected the BigQuery dataset to Jupyter<\/a> notebooks, writing analyses in pandas<\/a> and visualising data in matplotlib<\/a> – I highly recommend this setup if you\u2019ve got big<\/span> reasonably sized data to analyse.<\/p>\n

Tracking clinical trials<\/h4>\n

Another project was TrialsTracker<\/a>, which tracks which universities, hospitals and drug companies aren\u2019t reporting their clinical trial results. This matters because clinical trials are the best way we have to test whether a new medicine is safe and effective, but many trials never report their results – especially trials that find the medicine isn\u2019t effective. In fact, trials with negative results are twice as likely to remain unreported<\/a> as those with positive results.<\/p>\n

The TrialsTracker<\/a> project tries to fix this by naming and shaming the organisations that aren\u2019t reporting their clinical trials. This was Ben\u2019s idea, and I wrote the code<\/a> to make it work. It gets details of all trials registered on clinicaltrials.gov<\/a> that are listed as ‘completed’, and then checks whether their results are published either there or on PubMed<\/a> using a linked identifier (so a researcher can find them easily). Then it aggregates the results by trial sponsor, showing the organisations with the worst publication record:<\/p>\n

\"Screenshot<\/a><\/p>\n

My approach to this was minimum viable product: it\u2019s a simple, responsive site that clearly lays out the numbers for each organisation, and provides an incentive to publish their unpublished trials (since the data is updated regularly, if they publish past trials, their position in the table will improve over time). We wrote a paper on it in F1000 Research<\/a>, the open science journal, and the project was covered in the Economist<\/a>.<\/p>\n

The best part of this project was getting numerous mails from researchers saying \u201cthis will help me lobby my organisation to publish more\u201d. Yay!<\/p>\n

Other projects<\/h4>\n

I also worked on the alpha of Retractobot<\/a> (coming soon), a new service to notify researchers when a paper they\u2019ve cited gets retracted. This matters because more and more papers are being retracted<\/a>, yet they continue to get cited frequently<\/a>, so bad results go on polluting science long after they\u2019ve been withdrawn. And I built the front-end website for the COMPare project<\/a> – this is a valiant group of five medical students, led by Ben, who checked for switched outcomes<\/a> in every trial published in the top five medical journals for six weeks, then tried to get them fixed. (Spoiler: the journals were NOT happy.) Here’s\u00a0more about COMPare<\/a>.<\/p>\n

Onwards!<\/h4>\n

After just over a year at EBM DataLab, I decided to move on to pastures new at the end of 2016. I\u2019d had a lot of fun, but the organisation was now stable and mature, and I was keen to explore other interests outside healthcare. I\u2019ve left the tech side of things in the highly capable hands of our developer Seb Bacon, previously CTO at OpenCorporates<\/a>.<\/p>\n

Since then, I\u2019ve been having fun working through a list of about 25 one-day coding and data analysis side projects (of the kind you always want to do, but never have time). These side projects include: several around housing and land, including with Inside AirBNB\u2019s data; statistical methods for conversion funnels; building an Alexa skill; setting up a deep learning project with Keras and Tensorflow to classify fashion images; more work on dress sizing data; and a few data journalism projects.<\/p>\n

Longer-term, I’m thinking of joining an early-stage venture as tech lead. If you\u2019d like to chat about the above, or just about anything related to coding, stats or maps, I\u2019m always keen to have coffee with interesting people: drop me a line<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

In the past year or so I did a lot of work on public-interest tech and data projects. I was so busy writing code, designing systems and hiring people that I failed to write anything at all about why these projects were worthwhile, and the sort of design and engineering challenges I had to overcome. […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[15,14],"tags":[],"class_list":["post-692","post","type-post","status-publish","format-standard","hentry","category-personal","category-tech-for-good"],"blocksy_meta":[],"_links":{"self":[{"href":"https:\/\/anna.ps\/blog\/wp-json\/wp\/v2\/posts\/692","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/anna.ps\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/anna.ps\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/anna.ps\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/anna.ps\/blog\/wp-json\/wp\/v2\/comments?post=692"}],"version-history":[{"count":29,"href":"https:\/\/anna.ps\/blog\/wp-json\/wp\/v2\/posts\/692\/revisions"}],"predecessor-version":[{"id":725,"href":"https:\/\/anna.ps\/blog\/wp-json\/wp\/v2\/posts\/692\/revisions\/725"}],"wp:attachment":[{"href":"https:\/\/anna.ps\/blog\/wp-json\/wp\/v2\/media?parent=692"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/anna.ps\/blog\/wp-json\/wp\/v2\/categories?post=692"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/anna.ps\/blog\/wp-json\/wp\/v2\/tags?post=692"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}