/contrib/famzah

Enthusiasm never stops

C++ vs. Python vs. Perl vs. PHP performance benchmark (2016)

35 Comments

There are newer benchmarks: C++ vs. Python vs. PHP vs. Java vs. Others performance benchmark (2016 Q3)

The benchmarks here do not try to be complete, as they are showing the performance of the languages in one aspect, and mainly: loops, dynamic arrays with numbers, basic math operations.

This is a redo of the tests done in previous years. You are strongly encouraged to read the additional information about the tests in the article.

Here are the benchmark results:

Language CPU time Slower than Language
version
Source
code
User System Total C++ previous
C++ (optimized with -O2) 0.952 0.172 1.124 g++ 5.3.1 link
Java 8 (non-std lib) 1.332 0.096 1.428 27% 27% 1.8.0_72 link
Python 2.7 + PyPy 1.560 0.160 1.720 53% 20% PyPy 4.0.1 link
Javascript (nodejs) 1.524 0.516 2.040 81% 19% 4.2.6 link
C++ (not optimized) 2.988 0.168 3.156 181% 55% g++ 5.3.1 link
PHP 7.0 6.524 0.184 6.708 497% 113% 7.0.2 link
Java 8 14.616 0.908 15.524 1281% 131% 1.8.0_72 link
Python 3.5 18.656 0.348 19.004 1591% 22% 3.5.1 link
Python 2.7 20.776 0.336 21.112 1778% 11% 2.7.11 link
Perl 25.044 0.236 25.280 2149% 20% 5.22.1 link
PHP 5.6 66.444 2.340 68.784 6020% 172% 5.6.17 link

The clear winner among the script languages is… PHP 7. 🙂

Yes, that’s not a mistake. Apparently the PHP team did a great job! The rumor that PHP 7 is really fast confirmed for this particular benchmark test. You can also review the PHP 7 infographic by the Zend Performance Team.

Brief analysis of the results:

  • NodeJS got almost 2x faster.
  • Java 8 seems almost 2x slower.
  • Python has no significant change in the performance. Every new release is a little bit faster but overall Python is steadily 15x slower than C++.
  • Perl has the same trend as Python and is steadily 22x slower than C++.
  • PHP 5.x is the slowest with results between 47x to 60x behind C++.
  • PHP 7 made the big surprise. It is about 10x faster than PHP 5.x, and about 3x faster than Python which is the next fastest script language.

The tests were run on a Debian Linux 64-bit machine.

You can download the source codes, an Excel results sheet, and the benchmark batch script at:
https://github.com/famzah/langs-performance

 

Advertisements

Author: Ivan Zahariev

An experienced Linux & IT enthusiast, Engineer by heart, Systems architect & developer.

35 thoughts on “C++ vs. Python vs. Perl vs. PHP performance benchmark (2016)

  1. I wonder how PyPy stacks against CPython in those tests. Would you be interested in giving it a shot?

    • It’s lightning fast:

      user 0m1.560s
      sys 0m0.160s

      Total CPU time of 1.660 seconds which makes it faster than NodeJS and very close to C++. Also 12x faster than the standard Python.

      Python 2.7.10 (4.0.1+dfsg-1, Nov 20 2015, 19:46:58)
      [PyPy 4.0.1 with GCC 5.2.1 20151028]

  2. Great! JIT for the win! Maybe add the results to the table so that we can brag about it ? 😛

  3. One should bare in mind the way this benchmark works slants in statically typed languages favour. These benchmarks should be run as system services that respond to standard in and reply to standard out. Node must perform type inference before optimisation and execution. In reality systems built with these languages are long running services responding to requests. I would like to see the performance in a “real world” scenario. The most basic would be responding to standard input. This would provide a clearer picture of actual performance instead of compiler performance.

    • The scripts are executing in a “long running” mode. The benchmark times include the interpretation/parsing phase for each language, but it’s so small that its significance is negligible. The math function is called 10 times inside each long running process, in order to have more reliable results.

      • I would like to see the bench done using this in node.js

        var start = process.hrtime();//put this at the top of the js file
        var total = process.hrtime(start);//put this after the loop

        ‘total’ variable will contain a tuple array with position 0 being seconds and position 1 being nano seconds sum these values and you have the actual run time of the benchmark. Also node is now on version 6.2 not 4.2.
        Dump sum of tuple to file or whatever

        In fairness C++ bench should not include compile time

      • In fact
        var start = process.hrtime();
        should probably be placed just before the bench loop not before the function definition as this would bring it as close as possible to ‘real life’ usage. Also the bench function should be called once or twice before running the loop to warm up the VM.

        Similar steps should be taken for each language, taking in to account typical ‘real world’ configurations and each languages distinct strengths.

  4. Feel free to redo the tests using the current and your proposed setup, and let us know the outcome. This seems interesting.

  5. Interesting comparison. I have modified the php code to make it a bit faster in https://github.com/paul4156/langs-performance/blob/master/primes.php.

    Also, a slower, but easier-to-understand (no more array key mapping) hash map solution is provided for comparison. c++ using unordered_map is slower than php (20 sec vs 14 sec).

    Next to do is to compare the memory usage of each language.

  6. I got a contribution for the Go language: https://github.com/famzah/langs-performance/pull/1

    Go performs as fast as Nodejs:

    == JavaScript (nodejs) ==

    real 0m4.365s
    user 0m4.180s
    sys 0m0.236s

    real 0m4.301s
    user 0m4.132s
    sys 0m0.260s

    v0.10.25

    == Go (not optimized, default compiler) ==

    real 0m3.819s
    user 0m4.192s
    sys 0m0.128s

    real 0m3.222s
    user 0m4.332s
    sys 0m0.100s

  7. For real Java testing, you should test time of execution of calculation method, not whole JVM start/stop + time of calculation. See: http://stackoverflow.com/a/3382969 . Probably related to other languages as well (unless you want to test runtime start time which probably doesn’t depend at all on a tested algorithm).

    • Hi Piotr,

      That’s correct and already explained in the original article — “The times include the interpretation/parsing phase for each language, but it’s so small that its significance is negligible.” Furthermore, we run the calculation 10 times once the JVM has been started, and I believe that the time it takes is significantly bigger than the initial JVM start.

      Unless Java 8 JVM start/stop time is incredibly huge, the test results should be valid.

      I don’t have Java 8 to test right now, but here are the result for Java version “1.7.0_111”:
      – JVM start/stop time (a program which only prints “Hello world”): 0.120 seconds total user + system
      – The calculation benchmark (includes the JVM start/stop time): 10.824 seconds total user + system

      So the absolute pure calculation benchmark time is 10.824 – 0.120 = 10.704 seconds CPU time. Therefore, the error is 1.1%. I consider this fair.

      • In my test (Java 7):

        user+system: 12,504 [s]
        calculation time: 3,476 [s]

        Code:

        public static void main(String[] args) {
        long startTime = System.currentTimeMillis();
        ArrayList res;
        for (int i = 1; i <= 10; ++i) {
        res = (new PrimeNumbersGenerator1()).get_primes7(10000000);
        System.out.format("Found %d prime numbers.\n", res.size());
        }
        long stopTime = System.currentTimeMillis();
        long elapsedTime = stopTime – startTime;
        System.out.println("Elapsed time: "+elapsedTime+" [ms]");
        }

      • I’ve done some tests using Java 7 measuring the total CPU time it takes for the whole program to run:
        – 10 loops take 7.5 CPU seconds (0.75 sec/loop)
        – 50 loops take 30 CPU seconds (0.60 sec/loop)
        – 100 loops take 54 CPU seconds (0.54 sec/loop)
        – 300 loops take 156 CPU seconds (0.52 sec/loop) — 44% faster!

        You are right to suspect that the JVM start/stop + optimization passes take a long time. My tests show that I must probably run the Java programs much longer (at least for 60 CPU seconds), in order to be able to benchmark the pure program running time.

        Long story short — Java is 44% faster than what it is shown now in the summary table of this blog article.

        More research should be done for the other languages as well. I mean that most (if not all) of them must be tried with more loops, in order to verify if their interpreter/initial-start time is negligible. Or maybe it will be better to simply adjust the loops count for each language to a number where the total test runs for at least 90 CPU seconds.

    • Hi Piotr,

      I modified the benchmark method. Now all programs run for 90 wall-clock seconds. Even though the interpreter start/stop is included, it really is a very tiny part of the whole running time now.

      Here are the new results: https://blog.famzah.net/2016/09/10/cpp-vs-python-vs-php-vs-java-vs-others-performance-benchmark-2016-q3/

      To my surprise, Java 8 didn’t perform much faster as we expected. Maybe it gets slower as more and more loops are done, which also allocates more RAM.

      • JVM has a “-server” option which defaults many options to performance settings (if I’m not wrong). So, it may preallocate heap or collect garbage efficiently to perform better. The “Server” Hotspot VM may perform better, too. Many different measurements might be encountered with the JVM’s XX options, I assume. Might worth to try.

      • MT San, you are right and I didn’t know that. Luckily, “java” is already running in “-server” mode according to its help message on my Linux machines:

        $ java -h
        
        Usage: java [-options] class [args...]
                   (to execute a class)
           or  java [-options] -jar jarfile [args...]
                   (to execute a jar file)
        
        where options include:
            -d32          use a 32-bit data model if available
            -d64          use a 64-bit data model if available
            -client       to select the "client" VM
            -server       to select the "server" VM
        ...
            -hotspot      is a synonym for the "client" VM  [deprecated]
        
                          The default VM is server,
                          because you are running on a server-class machine.
        
  8. You chart is confusing. Your headers show C++ is slower than/previous but the values don’t match up with your conclusion. Nodejs shows it is 81% *slower*. And PHP 7 is not the best on the chart compared to C++…

    • Maverick, you need to elaborate.

      Nodejs (2.040 seconds) *is* 81% slower than C++ (1.124 seconds). If you divide both times you get a coefficient of 1.81, which means that Nodejs needs 1.81 times more CPU time to complete the task. Expressed in percentage, 1.81 is 81% slower.

      “PHP 7 is not the best on the chart compared to C++” — I don’t get it what you mean by this. It is the best *script* language. And by script language I mean a language which doesn’t need to be compiled before you execute it, like you do for C++, Java and Nodejs. Python + PyPy is a script language but it’s in gray and doesn’t count because PyPy does not fully support all Python code — it’s a optimization solution with some limitations.

      • For instance you say nodejs got almost 2x faster. However Nodejs is 81% slower than C++ when **previously** it was only 19%. From those numbers it seems to me that nodejs became significantly slower!

      • “Slower than previous” here means slower than the previous language in the current table. Not of the previous (last year’s) tests.

  9. That makes much more sense.

  10. Thank you for doing these benchmarks!You did a great job there 🙂

    Could you do the same with swift? I am currently learning Swift for mobile rather than Objective C or Java and I wonder if it is as performant as says apple…

    • Hi, thanks. Unfortunately, I can’t code in all programming languages 🙂 It would be best if you make a fork of the GitHub repo, code your version in Swift, and then I can try to benchmark it.

  11. Lamer article. Why did you use ArrayList for java and Vector for C++ ?? Doesn`t you know how use LinkedList ??

  12. There is two big problem.

    In your article, `The clear winner among the script languages is… PHP 7.`, so JavaScript is not a script language?

    Also, the stable version of Nodejs is v6, so do you think it’s reasonable that use the old nodejs version to test with the newest version of PHP?

    • Hi. You are looking at the older results. There are newer one, as explained in the beginning of this page.

      The used Node.js version there is “4.3.1” released on 2016-02-16. At the time of the last tests (Sep/2016), Node.js v6 was still not in LTS release (https://nodejs.org/en/download/releases/) and thus not yet “recommended for most users” (see what the Download page says regarding “Current” vs. “LTS” versions — https://nodejs.org/en/download/).

      Last but not least in regards to language versions — I don’t posses unlimited free personal time, in order to constantly update everything 🙂 The whole project is open-source, so feel free to prove by your own test results that there is a performance benefit to update the Node.js interpreter version. And then when I can, I will do it and publish the results.

      Regarding whether JavaScript is a script language or not. Of course, it is. The question is a bit open if it fits into the category “general purpose script language”. This is what I meant when I declared PHP7 as the winner. Somehow I imagine Node.js mostly as a script language used for server applications. It isn’t much used nor designed to be used synchronously and in command-line tool programs. Or was it?

  13. why are you tampering with the PHP7 results you have 2 instructions extra:

    error_reporting(E_ALL);
    ini_set(‘display_errors’, ‘1’);

    These should be set in the php.ini file and not in the code execution time as your code is currently set to.

    • Hi Martin, I don’t think that these two instructions make any difference in regards to the final benchmark results. They probably take something like microseconds to be executed.

  14. chart of this test, “total” values

  15. I miss Ruby here :-).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s