GitHub Copilot

What we appreciate most about GitHub Copilot revolves around its significant contributions todeveloper productivityandcode quality. Key aspects we value include: Speed of Delivery: Copilot dramatically accelerates our development process, which was a primary driver for its adoption within our product-based company. Enhanced Code Quality: It effectively aids in checking and improving the quality of our code. This is especially beneficial for developers from diverse backgrounds, such as data scientists and Python developers. Furthermore, it helps less experienced developers produce higher-quality code and serves as an effective learning tool, as seen with our interns. Intelligent Assistance: The tool possesses built-in intelligence that understands the development context, providingautomatic code writingand helpful suggestions for documentation and code structure. This integrated help minimizes distractions by reducing the need to search external resources. Seamless Integration: Its strong capability to integrate with various common development environments, including Visual Studio and VS Code, is highly valued. This integration ensures it becomes a natural and efficient part of our development ecosystem. Specialized Language Support: We've found it exceptionally valuable for Python-based projects, particularly when working with machine learning libraries such as scikit-learn, TensorFlow, and PyTorch, and in developing generative models. Ease of Use & Rapid Adoption: Starting with Copilot is straightforward and user-friendly, supporting self-onboarding, which facilitated its quick adoption across our organization. Support for less experienced developers: One benefit is its ability to improve the code quality of interns and less-experienced developers. It provides real-time assistance and suggestions, which not only elevates their output but also serves as an effective learning tool, helping them adopt better coding standards, a key factor in our ability to develop a new product.

I like how it can make tedious work way faster, and while inconsistent it can be useful sometimes to explain some things in the code base but you still may want to cross check with your team or have some context yourself and always take the advice with some skepticism.

Easy to set up, easy to get into. Like most Saas companies, the tech stack is quite common: MVC framework in Rails/Django/Spring/Node. Co-Pilot does well at setting up the boilerplate structure to get from 0 to 1 and does a decent job of helping one understand code paths as well.

While GitHub Copilot is a fantastic tool, there are a few areas that could be improved, particularly concerning its underlying AI models and broader operational considerations. Firstly, we occasionally encounter accuracy issues. However, it's important to clarify that this isn't necessarily a flaw of the tool itself, but rather a limitation of the underlying generative AI models that power it. These models are still evolving, especially in their reasoning ability, which is a critical aspect for AI success. As the core AI technology advances, I anticipate the accuracy of Copilot's suggestions will naturally improve. Secondly, I have limited knowledge regarding their handling of privacy and security issues. I haven't had the opportunity to conduct a deep dive into Copilot's specific privacy and security protocols. This area is particularly critical when considering legal implications and geographic-specific regulatory compliance, such as the differences between DPDP and GDPR. For industries like banking and finance, where I have significant experience, addressing stringent regulatory requirements, compliance, privacy, and security is paramount. These are areas where I believe improvement is necessary to ensure the tool fully meets the diverse and evolving compliance landscapes across different regions and industries. Therefore, while the core functionality and benefits of the tool are outstanding, the key areas for improvement lie in the ongoing advancement of the AI models' reasoning capabilities and a clearer, more robust demonstration of its privacy, security, and regulatory compliance measures, especially for sensitive industries and global deployments.

There is a lot of inconsistency when it comes to the willingness of the AI agent to process a request, when asked the same prompt even if it did it before. It might refuse to do it after or give an entirely different answer. The auto complete feature is intrusive and a lot of times the suggestions do not make sense at all, and it can be super useful for straightfoward tedious work but for everything else we found more productivity in just turning that feature off. Additionally, sometimes advice doesn't really answer the question if it's an elaborate one. Lastly, it can break the compilers in certain IDE's when it's enabled, and the only way to really fix this is to disable, restart the IDE and enable copilot again.

unlike other tools out there, co-pilot did not do a great job at detecting potential security issues and exploits. For example, it failed to call our a SQL injection attack vector when we allowed arbitrary search parameters for a search REST endpoint. It also didn't know about supply chain attacks and known CVE threats with some experimental open source libraries we were testing. although it's good at boilerplate suggestions, we dont have great confidence that it can actually tell the difference between good code and common code. We noticed in our monolithic repo, it was repeating anti-patterns in the way we set up large unit tests rather than suggesting more maintainable, DRY-er ways to approach it altogether.