David Whiting


Experienced Java and Scala data developer with a mathematical background and a passion for simple well-designed code, constant refactoring and challenging the status quo.

Previous experience includes Big Data pipelines and infrastructure development with Java, Hadoop, Crunch, Spark and Kafka; front end development with HTML, Javascript and jQuery; web application prototyping using Python and Flask; and assorted other data mining, aggregation and visualisation tasks.

Employment history

SoundCloud (Oct 2015 - Present), Berlin

Lead Software Engineer (Data Platform Team)

  • Performant distributed data transport, storage and processing (Java 8, Kafka, HDFS, Hadoop, Crunch, Spark).
  • High-throughput (>100k msg/s) backend services for data collection
  • Developer tooling and communications.

Spotify (Feb 2013 - Feb 2015), Stockholm

Senior Data Engineer

  • Creating infrastructure to improve the developer experience for engineers and analysts who use data to drive new products and make informed decisions, including migrating the data development platform from Python over Hadoop streaming to native JVM code based on the Apache Crunch framework (Java, Scala, Hadoop, Python, Hive, Pig)

Last.fm (Aug 2011 - Feb 2013), London

Java Developer, Data Team

  • Ground-up development of new scalable royalty reporting system using Hadoop for data filtering and aggregation, complete with a web front-end (Java, Hadoop, Cascading, Spring MVC, HTML, Javascript, jQuery, SQL, Postgres)
  • Maintainance and enhancement of existing company-wide statistics gathering and presentation software (Java, SQL, shell scripting, PHP).
  • Data extraction, interpretation and visualisation (Hive, R, GraphViz, Flot, Gephi)
  • Prototyping new data services for user statistics and radio playlisting (Python + Flask)

IPL Information Processing Limited (Oct 2010 - Jul 2011), Bath

Graduate Programmer

  • Improving test coverage on large public-sector resource management system (C#, NUnit)
  • Development of internal staff management tools (C#, ASP.NET, SQL)
  • Error logging and input validation implementation for online banking platform (C#, ASP.NET)
  • Software design verification for aviation engineering
  • Development of new training materials for future graduate engineers (Java, JUnit, JMock)

Morgan Stanley (Jul 2008 - Jun 2009), London

Software Developer (Industrial Placement)

Assorted small projects using Java, C#, ASP.NET and SQL, with lots of SOAP messaging going between them all (JSON over HTTP wasn't yet fashionable at the time)


University of Bath (Sept 2006 - Jun 2010)

BSc Mathematics, upper second class honours

A mix of pure and applied modules including Algebraic Structures, Mathematical Biology, Complex Analysis, Digital Signal Processing, Category Theory and Vector Calculus.


Open source projects

Active contributor and committer to the Apache Crunch project


See my separate Music Portfolio


Cycling, long-distance running, swimming, triathlon, hiking, skiing