Navigating the AI Development Landscape: It's time for GitHub Copilot to meet SAP

GitHub Copilot, one of the most talked-about AI code-generation tools of the past year, is my latest stop in the journey of exploring AI code development tools in the SAP domain. In case you missed it, in the first part of this series, I explored SAP Build Code and Joule. SAP Build Code performed admirably in accelerating development and automating tedious tasks, while Joule’s integration was a bit scripted but effective at generating code and understanding the problem at hand. Although, as always, there’s room for improvement. You can find the full post here.

In this post, I’ll evaluate GitHub Copilot using the same task I assigned to SAP Build Code. My goal is to compare how these tools assist in the development of an SAP CAP (Cloud Application Programming) project, focusing on their strengths, weaknesses, and overall usability.

GitHub Copilot: Your AI Pair Programmer

To conduct this evaluation, I followed a similar process to what I did with SAP Build Code: starting with data modelling, then moving on to test data, creating a basic UI, and finally diving into the action code. GitHub Copilot is not an SAP-specific tool, so it doesn’t provide boilerplate code or guide the entire project development process. Instead, I coded everything manually and observed how Copilot interacted with my code.

As development IDE, I used Visual Studio Code with GitHub copilot extension activated.

Step 1: Let’s get the project barebones

As with any SAP CAP project I started on terminal and cds init <name> to get the basic folder structure going. I then opened this new folder in vs.code and here we see Copilot enabled immediately on the left .

My first task was to create a data model. I provided Copilot with the same prompt to what I used with Joule and just added that this was an SAP CAP framework project. Copilot managed to provide code for a basic CAP data model, which I could easily move into the appropriate file. Now it doesn’t necessarily know where to place the db file so you have to create this manually in your db folder but I expect any user with basic CAP knowledge to know at least where to place files.

This highlights a key difference: Joule generated end-to-end code with a database model, service, and annotations, while Copilot only handled the data model. Now depending on your preference, you can argue the one is doing too much while the other is doing too little. We’ll let the community decide. 😊

Step 2: Tweak and adjust!

With no graphical editor, just plain CDS editing, Copilot immediately started suggesting or helping with definitions and annotations. While some suggestions were on point and helpful, others were random or incorrect. I appreciated having Copilot on the left panel as a resource, allowing me to ask questions about specific annotations and receive responses and examples, which saved time compared to searching through documentation. Some suggestions introduced me to new annotations I hadn’t used before—thank you, Copilot.

Overall, I found it beneficial to have Copilot assist with the tedious aspects of CDS editing, even if its SAP-specific knowledge wasn’t always up to par.

CDS model finalised, now let’s get some test data. After a few attempts, I finally got Copilot to provide CSV data. As to be expected, the data was very ‘robotic’ and changes were needed to make it fit in with our use case. Overall, SAP Build Code and Joule was better in test-data generation with a graphical CSV editor (with prefilled columns) and AI to generate and modify data as needed.

Step 3: Connect to S4

Connecting to an external API in CAP is well-documented at this point. The process involves downloading the API’s EDMX file, running a cds import to bring it into the project, and configuring the necessary settings. I also configured the project for HANA, UAA authentication, an approuter, and MTA deployment—tasks that were primarily executed through CLI commands, with no involvement from Copilot.

In contrast, Build Code handled a lot of the configuration upfront. For instance, it automatically set up the HANA connection and generated the approuter configuration. While this wasn’t a huge inconvenience, it did highlight the difference in approach between the two tools: Build Code aims to simplify the setup process, while Copilot provides more freedom (and responsibility) to the developer.

Step 4: Action to post-to-s4

When it was time to define a new function on the service and respective hook code, GitHub Copilot performed reasonably well, trying to autocomplete sentences and definitions. When I created the JavaScript hooks file, I asked it to generate a template for CAP framework hooks with an on event for sentToSAP. Surprisingly, it provided the basic structure.

From there, Copilot assisted me line-by-line code to implement the logic. The suggestions were often generally on the right track but needed tweaking to fit my specific use case. In the end, I wrote most of the logic myself, but Copilot’s assistance with autocompleting and anticipating my next steps was helpful.

Step 5: Let’s run and see what we got!

Time for our first test run. Open terminal and cds watch. Standard CAP page here allowing us to see our oData and preview with the generated Fiori Elements screens.

And our Fiori Elements generate screen looks pretty bare without any annotations so let’s go ahead and add some. Joule had already pre-generated the annotations file in Build Code, but in Visual Studio Code, this had to be done from scratch. Copilot made some attempts to suggest annotations, but these were often either incorrect or incomplete, requiring manual intervention.

After some trial-and-error, the annotations were done. Did copilot help? A little. Was Joule better? Marginally! Apart from initial annotation generation, Joule or Build Code could not assist with further editing and tweaking.

Step 6: Let’s build the UI

For the UI, I used the Fiori template wizard available in Visual Studio Code. The process was similar to what I experienced with Build Code, but with more options, especially for freestyle applications and Fiori elements. Select ‘List Report Page’ and off we go. The process is rudimentary the same so won’t go into details.

Once the UI code was generated in the app folder, I set about adding the custom action to the object page. I attempted to use Copilot to help with this task, specifically to add the custom action in the manifest.json file. While Copilot understood the task and provided some guidance, the generated snippet was ultimately incorrect. Thankfully, having just done this on Build Code, I recognized the mistake and added the correct code manually.

This experience underscored a general issue: overly trusting AI can lead to wasted time. Debugging or trying to figure out why a generated code snippet doesn’t work can be frustrating. In this case, I don’t blame Copilot entirely—the SAPUI5 documentation is complex, with snippets for multiple configurations depending on whether you’re using OData v2 or v4. Nonetheless, Copilot’s inaccuracies with SAP-specific configurations were a reminder that AI tools still have a way to go in understanding domain-specific contexts.
Moving on to the actual code in the CustomAction class, Copilot again tried to assist by generating code suggestions. The results were mixed: while it did a decent job with general JavaScript, the SAPUI5-specific code was often incorrect.

Final testing

With the button code written, I proceeded to the final phase: end-to-end testing. At this point, I realized I hadn’t configured the S4/HANA API to point to my actual S4 system. After completing the configuration, I ran the test again, only to encounter an error: a missing dependency for the CAP framework to make HTTP calls. Copilot tried to assist by suggesting a solution, but it didn’t quite hit the mark. Luckily, I recognized the issue from a previous project and quickly resolved it by adding the missing dependency.

Once the dependency was in place, the test was successful, and the custom action was able to communicate with S4/HANA as expected. Project Done!

Final comments

My experience with GitHub Copilot for an SAP CAP project was generally positive, though not without its challenges. Copilot integrates seamlessly into Visual Studio Code, suggesting useful additions without being too intrusive. The Copilot chat window was a particularly valuable resource, allowing me to ask questions about SAP-specific tasks and get immediate feedback. However, Copilot’s SAP domain knowledge felt outdated or random, particularly with UI annotations and SAP-specific configurations. This limited its usefulness in certain scenarios.

For SAP-specific tasks like CAP, UI5, or ABAP, Copilot currently doesn’t add much value compared to SAP Build Code. Especially if you are new to the SAP way-of-working and prefer cloud IDEs, SAP Build Code is the tool for you. Personally, I use Visual Studio Code as my main IDE (for SAP and non-SAP development) so having copilot by my side, is a nice touch. To that end, I’ll continue to keep GitHub Copilot in my toolbox and look forward to seeing how it evolves over time especially in the SAP domain.