Predicting the future popularity of programming languages
Sep 14, 2019 00:00 · 1069 words · 6 minute read
Insert data and ETL
Stack overflow data were used for this analysis. The dataset was downloaded from the Stack Exchange Data Explorer. The processed file is also downloadable here. It was used by David Robinson in a datacamp project.
Each Stack Overflow question has a tag, which marks a question to describe its topic or technology. This data has one observation for each pair of a tag and a year, showing the number of questions asked in that tag in that year and the total number of questions asked in that year. For instance, there were 54 questions asked about the .htaccess tag in 2008, out of a total of 58390 questions in that year.
| year | tag | number | year_total |
|---|---|---|---|
| 2008 | .htaccess | 54 | 58390 |
| 2008 | .net | 5910 | 58390 |
| 2008 | .net-2.0 | 289 | 58390 |
| 2008 | .net-3.5 | 319 | 58390 |
| 2008 | .net-4.0 | 6 | 58390 |
| 2008 | .net-assembly | 3 | 58390 |
Instead of using the counts of each tag questions in a year we’ll calculate & use the fraction of each tag questions and the overall questions in that year. It is more convenient to perform the comparison now.
| year | tag | number | year_total | fraction |
|---|---|---|---|---|
| 2008 | .htaccess | 54 | 58390 | 0.0009 |
| 2008 | .net | 5910 | 58390 | 0.1012 |
| 2008 | .net-2.0 | 289 | 58390 | 0.0049 |
| 2008 | .net-3.5 | 319 | 58390 | 0.0055 |
| 2008 | .net-4.0 | 6 | 58390 | 0.0001 |
| 2008 | .net-assembly | 3 | 58390 | 0.0001 |
How have popular programming languages changed over time?
It would be interesting to look at the popularity of the top programming languages. In particular, the following programming languages are included:
- java
- JavaScript
- c#
- c++
- python
- php
- ruby
- r
The fraction of each tag questions (on the overall questions in the year) used for this comparison.
We can identify that some languages are rising & others that falling in popularity. In particular JavaScript, java, c#, c++, ruby are falling and python with R (languages used in analytics) are rising. Although this is true we should check if the total number of questions is growing significantly.
## `summarise()` ungrouping output (override with `.groups` argument)
It is clear that since 2013 the total number of questions is not significantly growing. So we can say that the trends of the programming languages could be significant, especially from 2013 onward.
Predicting the future popularity of programming languages
It would be interesting to predict the future popularity of the programming
languages. I’ll use the forecast package
to generate predictions. In particular, I’m combining the power of the main forecasting
methodologies, ARIMA & Exponential smoothing. In particular for each time-series
(each programming language) 2 separate models are created, using ARIMA* & Exponential smoothing methods,
and the best one is selected for prediction. MAPE (mean absolute percentage error)**
is chosen to evaluate the forecasting models.
* ARIMA (Auto-regressive integrating moving average) is a very popular technique for time
series modelling. It describes the correlation between data points and takes into account
the difference of the values.
** Exponential Smoothing methods include simple exponential smoothing (larger weights are
assigned to more recent observations than to observations from the distant past),
double exponential smoothing or Holt linear trend model (also takes account the
trend of the series) and triple exponential smoothing or Host’s Winters method
(also takes account both the trend and the seasonality of the time series)
| tag | mape_arima | mape_ets |
|---|---|---|
| c# | 5.14 | 9.15 |
| c++ | 5.12 | 4.36 |
| java | 3.62 | 6.32 |
| php | 4.54 | 9.81 |
| python | 5.80 | 10.65 |
| r | 15.26 | 8.91 |
| ruby | 6.63 | 11.44 |
As you can see above, R & C++ predictions are better when applying exponential smoothing method than ARIMA. For the rest of the programming languages, ARIMA seems to be the best methodology.
Below there is a table with the future predictions, using the best performing model
| tag | index | key | fraction | lo.80 | lo.95 | hi.80 | hi.95 |
|---|---|---|---|---|---|---|---|
| c# | 2019 | forecast | 0.0473700 | 0.0369888 | 0.0314934 | 0.0577512 | 0.0632466 |
| c# | 2020 | forecast | 0.0400400 | 0.0253588 | 0.0175870 | 0.0547212 | 0.0624930 |
| c# | 2021 | forecast | 0.0327100 | 0.0147293 | 0.0052109 | 0.0506907 | 0.0602091 |
| c# | 2022 | forecast | 0.0253800 | 0.0046176 | -0.0063733 | 0.0461424 | 0.0571333 |
| c# | 2023 | forecast | 0.0180500 | -0.0051630 | -0.0174512 | 0.0412630 | 0.0535512 |
| c++ | 2019 | forecast | 0.0245024 | 0.0225875 | 0.0215739 | 0.0264173 | 0.0274309 |
| c++ | 2020 | forecast | 0.0221187 | 0.0203901 | 0.0194750 | 0.0238472 | 0.0247623 |
| c++ | 2021 | forecast | 0.0197349 | 0.0181926 | 0.0173762 | 0.0212772 | 0.0220936 |
| c++ | 2022 | forecast | 0.0173512 | 0.0159952 | 0.0152773 | 0.0187071 | 0.0194250 |
| c++ | 2023 | forecast | 0.0149674 | 0.0137977 | 0.0131785 | 0.0161371 | 0.0167563 |
| java | 2019 | forecast | 0.0777976 | 0.0720138 | 0.0689520 | 0.0835814 | 0.0866432 |
| java | 2020 | forecast | 0.0815882 | 0.0719591 | 0.0668618 | 0.0912173 | 0.0963146 |
| java | 2021 | forecast | 0.0862540 | 0.0749955 | 0.0690356 | 0.0975125 | 0.1034724 |
| java | 2022 | forecast | 0.0895722 | 0.0781559 | 0.0721125 | 0.1009884 | 0.1070319 |
| java | 2023 | forecast | 0.0904288 | 0.0788499 | 0.0727203 | 0.1020077 | 0.1081372 |
| php | 2019 | forecast | 0.0449024 | 0.0398318 | 0.0371475 | 0.0499731 | 0.0526573 |
| php | 2020 | forecast | 0.0318811 | 0.0209028 | 0.0150912 | 0.0428594 | 0.0486710 |
| php | 2021 | forecast | 0.0179549 | 0.0003111 | -0.0090290 | 0.0355988 | 0.0449388 |
| php | 2022 | forecast | 0.0037594 | -0.0208401 | -0.0338623 | 0.0283589 | 0.0413811 |
| php | 2023 | forecast | 0.0000000 | -0.0415394 | -0.0581922 | 0.0213763 | 0.0380291 |
| python | 2019 | forecast | 0.1095000 | 0.1035134 | 0.1003444 | 0.1154866 | 0.1186556 |
| python | 2020 | forecast | 0.1201000 | 0.1067137 | 0.0996274 | 0.1334863 | 0.1405726 |
| python | 2021 | forecast | 0.1307000 | 0.1083004 | 0.0964427 | 0.1530996 | 0.1649573 |
| python | 2022 | forecast | 0.1413000 | 0.1085103 | 0.0911525 | 0.1740897 | 0.1914475 |
| python | 2023 | forecast | 0.1519000 | 0.1075025 | 0.0839999 | 0.1962975 | 0.2198001 |
| r | 2019 | forecast | 0.0281568 | 0.0223167 | 0.0192252 | 0.0339968 | 0.0370883 |
| r | 2020 | forecast | 0.0295822 | 0.0210127 | 0.0164763 | 0.0381516 | 0.0426880 |
| r | 2021 | forecast | 0.0309768 | 0.0200994 | 0.0143412 | 0.0418542 | 0.0476123 |
| r | 2022 | forecast | 0.0323412 | 0.0193380 | 0.0124546 | 0.0453444 | 0.0522279 |
| r | 2023 | forecast | 0.0336761 | 0.0186408 | 0.0106816 | 0.0487115 | 0.0566707 |
| ruby | 2019 | forecast | 0.0040000 | 0.0026986 | 0.0020097 | 0.0053014 | 0.0059903 |
| ruby | 2020 | forecast | 0.0021000 | -0.0008100 | -0.0023504 | 0.0050100 | 0.0065504 |
| ruby | 2021 | forecast | 0.0002000 | -0.0046693 | -0.0072470 | 0.0050693 | 0.0076470 |
| ruby | 2022 | forecast | 0.0000000 | -0.0088279 | -0.0126012 | 0.0054279 | 0.0092012 |
| ruby | 2023 | forecast | 0.0000000 | -0.0132513 | -0.0183603 | 0.0060513 | 0.0111603 |
Below there is a plot with the future predictions
Summary
To compare programming languages popularity we used the fraction of total questions (on Stack Overflow) that concern each language for the last 10 years. Stack overflow is, by far, the most popular platform for questions and answers on a wide range of topics in computer programming.
Of course, there are a lot of other metrics that could be used to measure popularity. Furthermore, forecasting models, like all models, are not perfect and could have higher deviations than predicted. But, given the relatively small MAPE for all time series (less than 9 % for all), the predictions should be a good indication for the future popularity of programming languages.
Overall, the results are the following:
- Analytics programming languages (Python & R) will continue gaining popularity
- Java will gain a little and then keep a constant popularity
- JavaScript, C# & C++ will loose significant popularity
- PHP & Ruby could lose almost all their popularity and become obsolete in the next 5 years