Today I pushed an update to RecipeSnap to utilize the latest Claude 3.5 Sonnet model. Previously I was using the OpenAI gpt-4-vision-preview that was released in November 2023. Recently I've anecdotally noticed the model was making more mistakes. In the past six months models have become more capable and less expensive, so it was time to make an update to a newer model.
Why change to Claude? One reason was the release of Claude 3.5 Sonnet. Previous Claude models performed poorly extracting information from images. Claude 3.5 Sonnet was an enormous improvement making Claude a viable option for the task.
Additionally, the OpenAI API has been unreliable recently. The same suite of test images are failing at increased rates with no clear explanation why. The image encoding hasn't changed. The images themselves are the same. Yet the API returns errors about the image being too large or being in the wrong format. I couldn't find a clear resolution for these issues, so I moved to Claude, which worked rightaway.
I'm still working through what the issue is with these images. Ultimately I'd like to do a head-to-head evaluation between Claude 3.5 Sonnet and the latest OpenAI gpt-4o and gpt-4o-mini models to empirically determine what is the best model for the application.
In the meantime you can go checkout the Claude-powered version of RecipeSnap!