LangSmith

Langsmith has been a great extension to our LLM observability and offline evals. The traces, curated datasets, and evals have been fairly straight forward to set up and we're seeing a lot of depth in the details from them. At the same time, there's a noticeable learning curve in how concepts are organized (projects, runs, datasets, evals, and tags) and the UX can feel dense for a small team like ours that doesn't live in Langsmith every day. Once we settled on our golden path for our first chat interface, (ie. setting up log traces, structured evals), the product has become a core part of our workflow. Namely, planning the hypothesis at the beginning of the sprint, seeing the outputs during the sprint, then running evals at the end of the sprint to do more holistic evidence gathering.