{"id":17716,"date":"2025-06-30T14:37:27","date_gmt":"2025-06-30T12:37:27","guid":{"rendered":"https:\/\/www.centigrade.de\/blog\/open-source-llms-im-test-wie-gut-eignen-sie-sich-fuer-als-entwicklungsunterstuetzung\/"},"modified":"2025-07-17T15:47:26","modified_gmt":"2025-07-17T13:47:26","slug":"open-source-llms-put-to-the-test-how-well-suited-are-they-for-development-support","status":"publish","type":"blog","link":"https:\/\/www.centigrade.de\/en\/blog\/open-source-llms-put-to-the-test-how-well-suited-are-they-for-development-support\/","title":{"rendered":"Open-source LLMs put to the test: How well suited are they for development support?"},"content":{"rendered":"

\"Zwei<\/h2>\n

1. Intro<\/strong><\/h2>\n

In this blog article, we address the question of whether open source models provided locally within companies are mature enough to compete with the \u201cbig\u201d commercial models such as Claude or Chat-GPT. The hype surrounding AI \u2013 regardless of the industry \u2013 could hardly be greater. Everywhere, automation and efficiency gains are promised, because machines are supposed to achieve more in the same amount of time than dozens of people ever could. We are investigating the extent to which this also applies to free open source models. To do this, we took a realistic Angular example project<\/a> and set the LLMs Code Llama 3.1, Deep Seek Coder 2, Phind Code Llama, and Wizard Coder<\/strong> a few tasks…<\/p>\n

<\/p>\n

The results are then compared with Claude Sonnet 4’s answers to provide a solid reference value. Who wins? And by how much? Read on to find out.<\/p>\n

The results of this test include unavoidable subjective assessments and therefore inevitably reflect our perspective. Some readers will certainly have had different experiences and therefore not share our opinion on the test results. And that’s perfectly fine!<\/p>\n

1.1 Motivation<\/strong><\/h3>\n

Large language models (LLMs) are no longer just a hype\u2014they have established themselves as practical aids in software development. They can offer valuable support, especially in web development, where frameworks, libraries, and best practices are constantly evolving. But how good are local open-source models with up to 60B really? And is it worth using them in a professional environment?<\/p>\n

That’s exactly what we want to find out in our test. Because if open-source LLMs deliver what they promise, they could not only optimize our workflows, but also represent a more independent and, above all, more privacy-friendly alternative to proprietary models.<\/p>\n

1.2 Testeted LLMs<\/strong><\/h3>\n

In our test, we selected the best-known and most successful open source models. These include models developed and trained by large corporations such as Meta and DeepSeek AI, as well as those that are highly ranked on the Big Code Models Leaderboard<\/a>.<\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t
Name <\/th>Number of parameters<\/th>Open-Source<\/th>Description<\/th>\n<\/tr>\n<\/thead>\n
Code Llama 3.1<\/td>8B<\/td>?<\/td>Meta's powerful model for code generation and understanding.<\/td>\n<\/tr>\n
Deep Seek Coder 2<\/td>15.7B<\/td>?<\/td>Specialized in code analysis and generation with extended context.<\/td>\n<\/tr>\n
Phind Code Llama<\/td>3.8B<\/td>?<\/td>Optimized for accurate answers and fast code assistance.<\/td>\n<\/tr>\n
Wizard Coder<\/td>33B<\/td>?<\/td>Finely tuned for complex programming tasks and refactorings.<\/td>\n<\/tr>\n
Claude Sonnet 4<\/td>not public<\/td>?<\/td>Anthropic's proprietary model with strong code comprehension capabilities.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n

2. Test setup<\/strong><\/h2>\n

The test takes place in a realistic Angular 18<\/strong> project, which serves as a realistic basis for our evaluation. We use VS Code<\/a> as our IDE with the Continue extension<\/a>, which allows us to integrate locally hosted models. The open-source LLMs are deployed on our infrastructure using Ollama and integrated into Continue to provide convenient access to the models from within the IDE.<\/p>\n

Hardware Limitations<\/h3>\n

By hosting on our own infrastructure, we are naturally subject to hardware limitations that restrict the maximum number of parameters in the models, for example. The largest models that still run smoothly for us have 60 billion parameters. With larger models, the test would certainly turn out differently here and there.<\/p>\n

2.1 Procedure<\/h3>\n

All models receive the same eight tasks with identical prompts and context data. The tests cover various aspects of web development:<\/p>\n

    \n
  • Explain app and routing structure <\/strong>(code comprehension): The LLM should explain the routing structure and shed light on the app’s architecture.<\/li>\n
  • Analyze a service’s API <\/strong>(code analysis): Here, the responsibility of a service and how it is used in the app should be explained.<\/li>\n
  • Add a simple feature <\/strong>(code modification): The LLMs should add the information “word count” and “reading time” to an article preview.<\/li>\n
  • Fix a bug <\/strong>(critical thinking): A bug with very subtle effects was built in here. The LLMs should find and fix it.<\/li>\n
  • Refactoring initialization observables <\/strong>(refactoring): Here, the models were presented with several component initialization logics based on RxJS that share a common pattern. This pattern should be found and cleanly extracted.<\/li>\n
  • Accessibility check <\/strong>(A11y knowledge): This task tests the recognition of accessibility issues in components.<\/li>\n
  • Writing unit tests <\/strong>(test automation): The LLMs are presented with a component for which they are to write unit tests.<\/li>\n
  • Anti-pattern recognition <\/strong>(framework-specific knowledge): This task tests knowledge of best practices, whether Angular-specific or general to software development.<\/li>\n<\/ul>\n

    2.2 Evaluation<\/h3>\n

    The answers are evaluated according to the following criteria:<\/p>\n

      \n
    • Accuracy<\/strong>: Are the statements correct or misleading?<\/li>\n
    • Completeness<\/strong>: Are any essential aspects missing?<\/li>\n
    • Comprehensibility<\/strong>: Is the answer clear and structured?<\/li>\n
    • Relevance<\/strong>: Does the solution address the core of the question?<\/li>\n<\/ul>\n

      Each aspect is given a score of 1\u201310, from which the average is calculated as the overall rating.<\/p>\n

      Results …<\/h2>\n

      For the sake of brevity, we have not included the exact tasks and their answers from the LLMs here; the article only shows the grading and evaluation of the results.<\/p>\n

      Classification of the evaluation<\/h3>\n

      Although we have made every effort to keep the ratings as objective as possible, a certain degree of bias cannot be ruled out. Not everyone will agree with all the scores awarded, and that is perfectly legitimate. Work environments and requirements often vary greatly, so a “one-size-fits-all” rating is not possible.<\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t
      Task<\/th>Claude Sonnet 4 <\/th>Code Llama <\/th>DeepSeek Coder 2 <\/th>Phind Code Llama<\/th>Wizard Coder <\/th><\/td><\/td>\n<\/tr>\n<\/thead>\n
      1 \u2013 Understanding the code<\/td>9,75<\/td>7,75<\/td>8,75<\/td>8,25<\/td>7,25<\/td><\/td><\/td>\n<\/tr>\n
      2 \u2013 Code Analyse<\/td>10,0<\/td>8,25<\/td>9,25<\/td>7,0<\/td>9,0<\/td><\/td><\/td>\n<\/tr>\n
      3 \u2013 Feature-expansion<\/td>10,0<\/td>6,25<\/td>7,75<\/td>4,0<\/td>8,75<\/td><\/td><\/td>\n<\/tr>\n
      4 \u2013 Bug-Fixing<\/td>10,0<\/td>5,25<\/td>5,75<\/td>5,0<\/td>3,5<\/td><\/td><\/td>\n<\/tr>\n
      5 \u2013 Refactoring of Observables<\/td>5,5<\/td>4,25<\/td>1,0<\/td>4,75<\/td>2,5<\/td><\/td><\/td>\n<\/tr>\n
      6 \u2013 Accessibility-Check<\/td>9,75<\/td>5,0<\/td>5,25<\/td>6,25<\/td>5,25<\/td><\/td><\/td>\n<\/tr>\n
      7 \u2013 Writing unit-tests<\/td>8,75<\/td>7,0<\/td>4,25<\/td>8,75<\/td>3,0<\/td><\/td><\/td>\n<\/tr>\n
      8 \u2013 Angular Best Practices<\/td>10,0<\/td>1,25<\/td>8,25<\/td>2,0<\/td>2,75<\/td><\/td><\/td>\n<\/tr>\n
      <\/td><\/td><\/td><\/td><\/td><\/td><\/td><\/td>\n<\/tr>\n
      \u00d8 \/ 10<\/td>9,22 \/ 10<\/td>5,63 \/ 10<\/td>6,28 \/ 10<\/td>5,75 \/ 10<\/td>5,25 \/ 10<\/td><\/td><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n

      Explanations of the extremes<\/h3>\n

      In order to better classify the highest and lowest scores, we have briefly described the extremes here:<\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t
      Model<\/th>Best Task<\/th>Worst Task
      \n<\/th>\n<\/tr>\n<\/thead>\n
      Code Llama\t<\/td>8.25 points for code analysis<\/td>1.25 points for Angular best practices<\/td>\n<\/tr>\n
      <\/td>- Recognized methods and HTTP endpoints are correct
      \n- Clear functional description of the service
      \n- However: filter parameters are unclear, error handling is missing<\/td>      		- Almost all suggestions were incorrect or irrelevant
      \n- Did not understand takeUntilDestroyed
      \n- Only recognized single responsibility principle (SRP) violations
      \n- However, since Angular is used less frequently than other front-end frameworks such as React, we consider this result to be understandable, as the LLM training data will contain correspondingly little knowledge of Angular.<\/td>\n<\/tr>\n
      Deep Seek Coder 2\t<\/td>9.25 points for code analysis<\/td>1.0 points for refactoring an observable<\/td>\n<\/tr>\n
      <\/td>- Recognized methods and HTTP endpoints are correct
      \n- Clear functional description of the service
      \n- However: Missing parameters\/return types, outdated DI recommendation<\/td>      		- The task could not be completed
      \n- After repeated attempts, the output always stops at the same point<\/td>\n<\/tr>\n
      Phind Llama<\/td>Write 8.75 points in unit tests<\/td>2.0 points for Angular best practices<\/td>\n<\/tr>\n
      <\/td>- Tests compiled directly
      \n- Best practices such as dependency doubles and maintaining smoke tests were followed
      \n- Template is also tested
      \n- However: Incorrectly added a standalone component to the declarations array.<\/td>      		- Outdated unsubscribe patterns recommended
      \n- Unnecessary component extractions suggested
      \n- No helpful architecture recommendations
      \n- However: Since Angular is used less frequently than other frontend frameworks such as React, we consider this result understandable, as the LLM training data will contain correspondingly little Angular knowledge.<\/td>\n<\/tr>\n
      Wizard Coder<\/td>9.0 points for code analysis<\/td>2.5 points for refactoring an observable<\/td>\n<\/tr>\n
      <\/td>- Good, concise, and precise listing of the class API
      \n- Uses technical terms and concepts
      \n- Shows that it is a REST API
      \n- Addresses the topic of error handling<\/td>      		- Code that does not compile is suggested (ignores encapsulation)
      \n- Generated code does not match the original in terms of content
      \n- Introduces a bug because the wrong service is called
      \n- Poor naming of functions<\/td>\n<\/tr>\n
      Claude\t<\/td>10.0 points for code analysis, feature enhancement, and bug fixing<\/td>5.5 points for refactoring an observable<\/td>\n<\/tr>\n
      <\/td>- Good and concise description of APIs and their benefits
      \n- Proposed code is simple, well named, and compiles
      \n- Cause and scenario of error are explained coherently and bug is found<\/td>      		- Code compiles, but the abstraction created offers no added value
      \n- A code snippet was reproduced semantically 1:1<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n

      … And evaluation<\/h2>\n

      The proprietary Claude model clearly dominates with an average score of 9.2\/10<\/strong>, outperforming open-source models (average score of 5.6\u20136.3) in almost all tasks, especially complex debugging, refactoring, and framework knowledge<\/strong>. Here is a summary of the results table:<\/p>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t\n\t
      Model<\/th>Strengths<\/th>Weaknesses<\/th>\n<\/tr>\n<\/thead>\n
      Code Llama<\/td>High scores in routine tasks (Task 1, Task 2)
      \n<\/td>      		Weaknesses in complex implementation with semantic details and outdated knowledge (Task 5, Task 8)<\/td>\n<\/tr>\n
      Deep Seek Coder<\/td>Robust in descriptive tasks (Task 1, Task 2)
      \n<\/td>      		\"Fails at architecture\/pattern tasks (Task 5, Task 8).
      \nRefactoring task could not be completed.
      \nAccessibility rather superficial\"<\/td>\n<\/tr>\n
      Phind<\/td>Very good in structured language and tests (Task 1, Task 7)
      \n<\/td>      		Weak in Angular details, refactoring, anti-pattern recognition
      \n\"Complete failure in anti-pattern recognition (Task 5, Task 8).<\/td>\n<\/tr>\n
      Wizard Coder<\/td>Very strong in explicit tasks with a clear structure (Task 2, Task 3)<\/td>\"Complete failures in anti-pattern detection (Task 5, Task 8).
      \nWeaknesses in identifying the exact cause of errors and testable code.<\/td>\n<\/tr>\n
      Claude<\/td>Convincing in almost all disciplines, has good and up-to-date knowledge of Angular and best practices<\/td>Weak when it comes to semantic details that need to be refactored in a meaningful way.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n

      Strengths of all LLMs:<\/strong><\/p>\n

        \n
      • Code understanding:<\/li>\n
      • Summarizing obvious app architectures<\/li>\n
      • Summarizing class APIs<\/li>\n
      • Use of classes and components<\/li>\n<\/ul>\n

        Mixed performance<\/strong><\/p>\n

          \n
        • Unit test generation<\/li>\n
        • From “too complicated to get up and running” to “compiled out-of-the-box”<\/li>\n
        • From “relevant things omitted” to “too many details tested”<\/li>\n
        • Feature extension<\/li>\n
        • From “contains absolute beginner-level errors” to “flawlessly implemented”<\/li>\n<\/ul>\n

          Weaknesses of open-source <\/em>LLMs:<\/strong><\/p>\n

            \n
          • Finding subtle bugs<\/li>\n
          • The bug is not limited to one or a few specific locations, but rather many possibilities are “suggested.”<\/li>\n
          • A kind of checklist is provided that does not necessarily have anything to do with the error.<\/li>\n
          • The explanations for the bug sometimes do not make much sense in the actual error context.<\/li>\n
          • Poor critical thinking <\/strong>(bug fixing, best practices)<\/li>\n
          • Superficial knowledge of accessibility and Angular<\/strong><\/li>\n<\/ul>\n

            Weaknesses of all <\/em>LLMs:<\/strong><\/p>\n

              \n
            • RxJS \/ Observables refactoring<\/li>\n
            • Meaningless abstractions are created very frequently<\/li>\n
            • This also results in poor and incomprehensible naming<\/li>\n
            • In most cases, the code no longer compiles after refactoring<\/li>\n
            • Code becomes more complicated, not simpler<\/li>\n<\/ul>\n

              Conclusion<\/h2>\n

              Open-source LLMs are sufficient for simple code generation, but proprietary models are (still) superior for professional development. While experienced developers can gain valuable insights from the models, beginners run the risk of adopting flawed concepts due to incorrect or outdated suggestions from the LLMs, thereby acquiring unfavorable habits. The models have very different strengths and weaknesses, which leads to a wide range of applications but also limitations.<\/p>\n

              At the end of the day, it’s the results that count. Those who use AI models effectively can turn them into valuable tools for completing a wide variety of tasks quickly and conveniently. For our part, we remain excited to see what future models will bring us and are happy to continue experimenting with them.<\/p>\n","protected":false},"author":82,"featured_media":0,"template":"","tags":[840,403,1031,983,656,654],"class_list":["post-17716","blog","type-blog","status-publish","hentry","tag-ai","tag-development-en","tag-development-2","tag-ki-2","tag-web-engineering-en-2","tag-web-engineering-en"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.centigrade.de\/en\/wp-json\/wp\/v2\/blog\/17716","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.centigrade.de\/en\/wp-json\/wp\/v2\/blog"}],"about":[{"href":"https:\/\/www.centigrade.de\/en\/wp-json\/wp\/v2\/types\/blog"}],"author":[{"embeddable":true,"href":"https:\/\/www.centigrade.de\/en\/wp-json\/wp\/v2\/users\/82"}],"version-history":[{"count":4,"href":"https:\/\/www.centigrade.de\/en\/wp-json\/wp\/v2\/blog\/17716\/revisions"}],"predecessor-version":[{"id":17718,"href":"https:\/\/www.centigrade.de\/en\/wp-json\/wp\/v2\/blog\/17716\/revisions\/17718"}],"wp:attachment":[{"href":"https:\/\/www.centigrade.de\/en\/wp-json\/wp\/v2\/media?parent=17716"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.centigrade.de\/en\/wp-json\/wp\/v2\/tags?post=17716"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}